On Fri, Jun 2, 2017 at 10:53 PM, Ralph Goers <rgoers@apache.org> wrote:
It uses BerkelyDB and writes the event to it before passing the event to another thread where it uses the Flume RPC client to send the event off. You could copy just the BerkelyDB logic and tie it to another delivery mechanism if you want.
Thanks for suggestion! I didn't profile myself how long latency in writing single row to DB, numbers from Internet about 1 ms. I think that in case of embedded DB latency would be much smaller. According to: http://www.oracle.com/technetwork/database/berkeleydb/overview/index-093405.... BerkelyDB add 1 MiB overhead in distribution which is acceptable. Don't know about runtime overhead. Embedded DB provides reliable solution for data consistency and allows keep it structured. I am not sure if DB size stay low if you add/del data over period of time. I don't like to maintain this DB. With some file rotation solution you can write to new file and process old one (and just delete it at the end).