[BAHIR-183] [WIP] HDFS based MQTT client persistence

[lukasz-antoniak]

As discussed in the JIRA ticket (https://issues.apache.org/jira/browse/BAHIR-183), revisited version of HDFS persistence of MQTT messages, which should ease recovery in case of device failure. The design discussion may be continued in JIRA, but I decided to open WIP PR for @yanlin-Lynn to review and comment.

[yanlin-Lynn]

Sorry to keep you wait, I will review this PR in a few days.

[Brahim13brahim]

yes

[lresende]

@lukasz-antoniak what's the status of this pr, is this ready for final review/merge?

[lukasz-antoniak]

@lresende, I would answer positive. This PR changes the way we implement HDFS persistence due to issues pointed out in JIRA commant.
@yanlin-Lynn, can you also have a look since I changed mostly the code you have introduced and worked on previously?

[yanlin-Lynn]

@lresende, I would answer positive. This PR changes the way we implement HDFS persistence due to issues pointed out in JIRA commant.
@yanlin-Lynn, can you also have a look since I changed mostly the code you have introduced and worked on previously?

Very sorry to keep you wait. I will review it in weekends.

[yanlin-Lynn]

I just worry about the performance of creating a file for each coming message.

[yanlin-Lynn]

And, I think we should better keep the ability to do flow-control, eg, control max number of message in each batch.