Logging encoded google protocol buffers with logback rolling file appender
Hi, I'd like to log google protocol buffers received by my server to a file, for later offline processing (the data is in a byte[]). This is something I only do in when a debug mode is enabled, but it is useful for diagnostics. Ideally I'd like to use logback, but it doesn't seem to cater for binary data. I saw this issue: http://jira.qos.ch/browse/LOGBACK-232 which is marked as fix, and is a request for the same thing. But I can't figure out how to use this with a normal rolling file appender. Does anyone have any examples of doing this? Cheers
With the Anodyzed Onyx (http://code.google.com/p/anodyzed) wrapper around Logback you can use log.debug("Google Protocol Buffer\n{0,binary,dump}",myBuffer); to write a Hex Dump style entry in the logs. (*Chris*) On Thu, Aug 23, 2012 at 4:25 AM, Dave Johnston <dave.johnston@me.com> wrote:
Hi,
I'd like to log google protocol buffers received by my server to a file, for later offline processing (the data is in a byte[]). This is something I only do in when a debug mode is enabled, but it is useful for diagnostics.
Ideally I'd like to use logback, but it doesn't seem to cater for binary data.
I saw this issue: http://jira.qos.ch/browse/LOGBACK-232 which is marked as fix, and is a request for the same thing. But I can't figure out how to use this with a normal rolling file appender.
Does anyone have any examples of doing this?
Cheers
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
Thanks Chris, That looks like it might work what additional overhead does Anodyzed Onyx add to the log call? My system is processing quite a high volume of traffic, so I wan't to avoid any performance costs. In my case I don't need to mix readable text with the binary data - my output file should just be purely binary, so maybe there is a simpler solution ? Chris Pratt wrote:
With the Anodyzed Onyx (http://code.google.com/p/anodyzed) wrapper around Logback you can use log.debug("Google Protocol Buffer\n{0,binary,dump}",myBuffer); to write a Hex Dump style entry in the logs. (*Chris*)
On Thu, Aug 23, 2012 at 4:25 AM, Dave Johnston <dave.johnston@me.com> wrote:
Hi,
I'd like to log google protocol buffers received by my server to a file, for later offline processing (the data is in a byte[]). This is something I only do in when a debug mode is enabled, but it is useful for diagnostics.
Ideally I'd like to use logback, but it doesn't seem to cater for binary data.
I saw this issue: http://jira.qos.ch/browse/LOGBACK-232 which is marked as fix, and is a request for the same thing. But I can't figure out how to use this with a normal rolling file appender.
Does anyone have any examples of doing this?
Cheers
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
-- View this message in context: http://old.nabble.com/Logging-encoded-google-protocol-buffers-with-logback-r... Sent from the Logback User mailing list archive at Nabble.com.
Hi Dave, A logback encoder is fundamentally binary beast. It writes to an OutputStream. Have you considered rolling your own Encoder to deal with protobuf data? On 24.08.2012 12:30, DaveJohnston wrote:
Thanks Chris,
That looks like it might work what additional overhead does Anodyzed Onyx add to the log call? My system is processing quite a high volume of traffic, so I wan't to avoid any performance costs.
In my case I don't need to mix readable text with the binary data - my output file should just be purely binary, so maybe there is a simpler solution ?
Chris Pratt wrote:
With the Anodyzed Onyx (http://code.google.com/p/anodyzed) wrapper around Logback you can use log.debug("Google Protocol Buffer\n{0,binary,dump}",myBuffer); to write a Hex Dump style entry in the logs. (*Chris*)
On Thu, Aug 23, 2012 at 4:25 AM, Dave Johnston <dave.johnston@me.com> wrote:
Hi,
I'd like to log google protocol buffers received by my server to a file, for later offline processing (the data is in a byte[]). This is something I only do in when a debug mode is enabled, but it is useful for diagnostics.
Ideally I'd like to use logback, but it doesn't seem to cater for binary data.
I saw this issue: http://jira.qos.ch/browse/LOGBACK-232 which is marked as fix, and is a request for the same thing. But I can't figure out how to use this with a normal rolling file appender.
Does anyone have any examples of doing this?
Cheers
-- Ceki http://tinyurl.com/proLogback
Hi Chris, Not yet - I was hoping there was an off-the-shelf solution that would just allow me to pass through a byte[]? Maybe log.info("{}", byte[]) would work fine with this ? I thought the second argument was getting cast to a string, but maybe not. If not it looks like writing my own encoder is the way to go. I'll take a peak at the interface and see what I can put together. Ceki Gulcu wrote:
Hi Dave,
A logback encoder is fundamentally binary beast. It writes to an OutputStream. Have you considered rolling your own Encoder to deal with protobuf data?
On 24.08.2012 12:30, DaveJohnston wrote:
Thanks Chris,
That looks like it might work what additional overhead does Anodyzed Onyx add to the log call? My system is processing quite a high volume of traffic, so I wan't to avoid any performance costs.
In my case I don't need to mix readable text with the binary data - my output file should just be purely binary, so maybe there is a simpler solution ?
Chris Pratt wrote:
With the Anodyzed Onyx (http://code.google.com/p/anodyzed) wrapper around Logback you can use log.debug("Google Protocol Buffer\n{0,binary,dump}",myBuffer); to write a Hex Dump style entry in the logs. (*Chris*)
On Thu, Aug 23, 2012 at 4:25 AM, Dave Johnston <dave.johnston@me.com> wrote:
Hi,
I'd like to log google protocol buffers received by my server to a file, for later offline processing (the data is in a byte[]). This is something I only do in when a debug mode is enabled, but it is useful for diagnostics.
Ideally I'd like to use logback, but it doesn't seem to cater for binary data.
I saw this issue: http://jira.qos.ch/browse/LOGBACK-232 which is marked as fix, and is a request for the same thing. But I can't figure out how to use this with a normal rolling file appender.
Does anyone have any examples of doing this?
Cheers
-- Ceki http://tinyurl.com/proLogback _______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
-- View this message in context: http://old.nabble.com/Logging-encoded-google-protocol-buffers-with-logback-r... Sent from the Logback User mailing list archive at Nabble.com.
I ended up creating a very simple encoder based on the LayoutWrappingEncoder (although without the Layout) I then replaced the PatternEncoder that I had used with my logger with this ProtocolBuffer. When I call myLogger.info("", MyProtocolBufferObj); It logs the MyProtocolBufferObj in its serialized form. Just what I wanted, cheers. public class ProtocolBufferEncoder extends EncoderBase<ILoggingEvent> { public void doEncode(ILoggingEvent event) throws IOException { // Get the data // Object[] data = event.getArgumentArray(); // We should only have one argument, which is the // protocol buffer - I'm sure there maybe better // ways to do this - I think the ILoggingEvent is unique // to logback - so not sure how this plays with slf4j // // if (data.length == 1) { // If the data is a Message // if (data[0] instanceof Message) { Message message = (Message) data[0]; // Create and write a 4 byte header indicating the size of // the message // byte[] header = getMessageHeader(message.getSerializedSize()); outputStream.write(header); // Write the message and flush if immediate flush is on // outputStream.write(message.toByteArray()); if (isImmediateFlush()) { outputStream.flush(); } } } } } } DaveJohnston wrote:
Hi Chris,
Not yet - I was hoping there was an off-the-shelf solution that would just allow me to pass through a byte[]? Maybe log.info("{}", byte[]) would work fine with this ? I thought the second argument was getting cast to a string, but maybe not.
If not it looks like writing my own encoder is the way to go. I'll take a peak at the interface and see what I can put together.
Ceki Gulcu wrote:
Hi Dave,
A logback encoder is fundamentally binary beast. It writes to an OutputStream. Have you considered rolling your own Encoder to deal with protobuf data?
On 24.08.2012 12:30, DaveJohnston wrote:
Thanks Chris,
That looks like it might work what additional overhead does Anodyzed Onyx add to the log call? My system is processing quite a high volume of traffic, so I wan't to avoid any performance costs.
In my case I don't need to mix readable text with the binary data - my output file should just be purely binary, so maybe there is a simpler solution ?
Chris Pratt wrote:
With the Anodyzed Onyx (http://code.google.com/p/anodyzed) wrapper around Logback you can use log.debug("Google Protocol Buffer\n{0,binary,dump}",myBuffer); to write a Hex Dump style entry in the logs. (*Chris*)
On Thu, Aug 23, 2012 at 4:25 AM, Dave Johnston <dave.johnston@me.com> wrote:
Hi,
I'd like to log google protocol buffers received by my server to a file, for later offline processing (the data is in a byte[]). This is something I only do in when a debug mode is enabled, but it is useful for diagnostics.
Ideally I'd like to use logback, but it doesn't seem to cater for binary data.
I saw this issue: http://jira.qos.ch/browse/LOGBACK-232 which is marked as fix, and is a request for the same thing. But I can't figure out how to use this with a normal rolling file appender.
Does anyone have any examples of doing this?
Cheers
-- Ceki http://tinyurl.com/proLogback _______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
-- View this message in context: http://old.nabble.com/Logging-encoded-google-protocol-buffers-with-logback-r... Sent from the Logback User mailing list archive at Nabble.com.
The whole point of Onyx is to move the processing to where it makes sense. If you have the logging level set above the level of the current log call, there should be the same or less overhead than with most typical logging systems (Log4j, Logback, etc), since there should be no string concatenation or object dereferencing performed, so in a production system the overhead should go down. There is a small cost for this when running under development, since the system has to use reflection to look up the arguments it is slightly slower than natively dereferencing the objects yourself. If you're looking to build binary output file, I think logging may be the wrong avenue for you. You might want to look into the java.io and java.nio packages. They're chock full of support for writing out binary data. (*Chris*) On Fri, Aug 24, 2012 at 3:30 AM, DaveJohnston <dave.johnston@me.com> wrote:
Thanks Chris,
That looks like it might work what additional overhead does Anodyzed Onyx add to the log call? My system is processing quite a high volume of traffic, so I wan't to avoid any performance costs.
In my case I don't need to mix readable text with the binary data - my output file should just be purely binary, so maybe there is a simpler solution ?
Chris Pratt wrote:
With the Anodyzed Onyx (http://code.google.com/p/anodyzed) wrapper
around
Logback you can use log.debug("Google Protocol Buffer\n{0,binary,dump}",myBuffer); to write a Hex Dump style entry in the logs. (*Chris*)
On Thu, Aug 23, 2012 at 4:25 AM, Dave Johnston <dave.johnston@me.com> wrote:
Hi,
I'd like to log google protocol buffers received by my server to a file, for later offline processing (the data is in a byte[]). This is something I only do in when a debug mode is enabled, but it is useful for diagnostics.
Ideally I'd like to use logback, but it doesn't seem to cater for binary data.
I saw this issue: http://jira.qos.ch/browse/LOGBACK-232 which is marked as fix, and is a request for the same thing. But I can't figure out how to use this with a normal rolling file appender.
Does anyone have any examples of doing this?
Cheers
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
-- View this message in context: http://old.nabble.com/Logging-encoded-google-protocol-buffers-with-logback-r... Sent from the Logback User mailing list archive at Nabble.com.
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
participants (4)
-
ceki -
Chris Pratt -
Dave Johnston -
DaveJohnston