Question about a custom binary file appender.
Hi Ceki. I'd like to implement a file appender that writes the binary Lilith format, i.e. gzipped protobuf-serialized events, instead of Strings. I'd also like to have the same functionality that's supported by RollingFileAppender right now. Unfortunately, there seems to be no way to simply write bytes instead of a String. How would you go from here? Reimplementing everything from the start seems to be a pretty bad idea. What do you think about enhancing the RFA so it's using byte[] instead of Strings? The current behavior could be implemented using those methods + string.getBytes("UTF-8") or CharsetEncoder... Any idea, suggestions? Regards, Joern.
What about intrducing the Encoder interface I proposed some weeks ago ? http://code.google.com/p/firewood/source/browse/trunk/compare-formats/src/ma... ============================== import java.io.OutputStream; import java.io.IOException; import ch.qos.logback.classic.spi.ILoggingEvent; public interface Encoder <T extends OutputStream>{ void encode(ILoggingEvent event, T output) throws IOException; T decorate(OutputStream os) throws IOException; } ============================== import ch.qos.logback.classic.spi.ILoggingEvent; import java.io.OutputStream; import java.io.IOException; import java.io.ObjectOutputStream; /** * Encoder that uses plain Java Serialization */ public class ObjectEncoder implements Encoder<ObjectOutputStream> { public void encode(ILoggingEvent event, ObjectOutputStream output) throws IOException { output.writeObject(event); } public ObjectOutputStream decorate(OutputStream os) throws IOException { return new ObjectOutputStream(os); } } ===================================== By the way, Joern, trying out a recent version of Lilith is still on my todo list. regards, Maarten On Thu, Apr 23, 2009 at 6:38 PM, Joern Huxhorn <jhuxhorn@googlemail.com> wrote:
Hi Ceki.
I'd like to implement a file appender that writes the binary Lilith format, i.e. gzipped protobuf-serialized events, instead of Strings. I'd also like to have the same functionality that's supported by RollingFileAppender right now.
Unfortunately, there seems to be no way to simply write bytes instead of a String. How would you go from here? Reimplementing everything from the start seems to be a pretty bad idea.
What do you think about enhancing the RFA so it's using byte[] instead of Strings? The current behavior could be implemented using those methods + string.getBytes("UTF-8") or CharsetEncoder...
Any idea, suggestions?
Regards, Joern.
_______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
Hi Ceki. I've seen that you are working on the FileAppenders again. Have you seen this mail I wrote? What do you think about it? Joern. Begin forwarded message:
From: Joern Huxhorn <jhuxhorn@googlemail.com> Date: 23. April 2009 18:38:38 MESZ To: logback developers list <logback-dev@qos.ch> Subject: Question about a custom binary file appender.
Hi Ceki.
I'd like to implement a file appender that writes the binary Lilith format, i.e. gzipped protobuf-serialized events, instead of Strings. I'd also like to have the same functionality that's supported by RollingFileAppender right now.
Unfortunately, there seems to be no way to simply write bytes instead of a String. How would you go from here? Reimplementing everything from the start seems to be a pretty bad idea.
What do you think about enhancing the RFA so it's using byte[] instead of Strings? The current behavior could be implemented using those methods + string.getBytes("UTF-8") or CharsetEncoder...
Any idea, suggestions?
Regards, Joern.
Hi Joern, Plese enter a bug report with as much information as you can about the code you'd like to see changed. In principle, very little code is involved in actually writing to the file so the task would seem rather easy. However, WriterAppender on which FileAppender and RollingFileAppender are built upon uses a Writer to write whatever it is that needs to be written. Unfortunately, a Writer knows how to writes String or chars, but not bytes. Nevertheless, I think it's feasible with some work... Joern Huxhorn wrote:
Hi Ceki.
I've seen that you are working on the FileAppenders again. Have you seen this mail I wrote? What do you think about it?
Joern.
Begin forwarded message:
*From: *Joern Huxhorn <jhuxhorn@googlemail.com <mailto:jhuxhorn@googlemail.com>> *Date: *23. April 2009 18:38:38 MESZ *To: *logback developers list <logback-dev@qos.ch <mailto:logback-dev@qos.ch>> *Subject: **Question about a custom binary file appender.*
Hi Ceki.
I'd like to implement a file appender that writes the binary Lilith format, i.e. gzipped protobuf-serialized events, instead of Strings. I'd also like to have the same functionality that's supported by RollingFileAppender right now.
Unfortunately, there seems to be no way to simply write bytes instead of a String. How would you go from here? Reimplementing everything from the start seems to be a pretty bad idea.
What do you think about enhancing the RFA so it's using byte[] instead of Strings? The current behavior could be implemented using those methods + string.getBytes("UTF-8") or CharsetEncoder...
Any idea, suggestions?
Regards, Joern.
------------------------------------------------------------------------
_______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
-- Ceki Gülcü Logback: The reliable, generic, fast and flexible logging framework for Java. http://logback.qos.ch
Hi Ceki and Joern, Remember the Encoder interface we discussed in the past ? To make the aspect of encoding a LoggingEvent pluggable, and thus the Appender implementation more reusable. I have small prototype here : http://tinyurl.com/encoder-interface http://tinyurl.com/encoder-example regards, Maarten On Fri, Jul 31, 2009 at 5:51 PM, Ceki Gulcu <ceki@qos.ch> wrote:
Hi Joern,
Plese enter a bug report with as much information as you can about the code you'd like to see changed. In principle, very little code is involved in actually writing to the file so the task would seem rather easy. However, WriterAppender on which FileAppender and RollingFileAppender are built upon uses a Writer to write whatever it is that needs to be written. Unfortunately, a Writer knows how to writes String or chars, but not bytes.
Nevertheless, I think it's feasible with some work...
Joern Huxhorn wrote:
Hi Ceki.
I've seen that you are working on the FileAppenders again. Have you seen this mail I wrote? What do you think about it?
Joern.
Begin forwarded message:
*From: *Joern Huxhorn <jhuxhorn@googlemail.com <mailto:
jhuxhorn@googlemail.com>> *Date: *23. April 2009 18:38:38 MESZ *To: *logback developers list <logback-dev@qos.ch <mailto: logback-dev@qos.ch>> *Subject: **Question about a custom binary file appender.*
Hi Ceki.
I'd like to implement a file appender that writes the binary Lilith format, i.e. gzipped protobuf-serialized events, instead of Strings. I'd also like to have the same functionality that's supported by RollingFileAppender right now.
Unfortunately, there seems to be no way to simply write bytes instead of a String. How would you go from here? Reimplementing everything from the start seems to be a pretty bad idea.
What do you think about enhancing the RFA so it's using byte[] instead of Strings? The current behavior could be implemented using those methods + string.getBytes("UTF-8") or CharsetEncoder...
Any idea, suggestions?
Regards, Joern.
------------------------------------------------------------------------
_______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
-- Ceki Gülcü Logback: The reliable, generic, fast and flexible logging framework for Java. http://logback.qos.ch _______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
Thank you for reminding us about the prototype. I find it very informative and helpful. As I see things, the Encoder interface and the various implementations you have written show that it is not sufficient to return a byte array for a given ILoggingEvent, the "encoder" also needs to know how to write to the underlying OutputStream, it may also need to decorate the underlying OutputStream with its own (e.g. ObjectOutputStream for serialization). Consequently, I'd propose to rename the Encoder interface as ILoggingEventWriter. (In my mind at least, an Encoder is associated with transforming data into bytes.) So here is my revised interface: public interface Encoder <T extends OutputStream> { void write(ILoggingEvent event, T output) throws IOException; T decorate(OutputStream os) throws IOException; } What should be done about methods such as getFileHeader(), getFileFooter(), getPresentationHeader() and getPresentationFooter() from the Layout interface? My first reaction would be to rename getX() as writeX(OutputStream). This is all I have time for at this moment but I would really like to pursue this exchange. Cheers, Maarten Bosteels wrote:
Hi Ceki and Joern,
Remember the Encoder interface we discussed in the past ? To make the aspect of encoding a LoggingEvent pluggable, and thus the Appender implementation more reusable.
I have small prototype here :
http://tinyurl.com/encoder-interface http://tinyurl.com/encoder-example
regards, Maarten
On Fri, Jul 31, 2009 at 5:51 PM, Ceki Gulcu <ceki@qos.ch <mailto:ceki@qos.ch>> wrote:
Hi Joern,
Plese enter a bug report with as much information as you can about the code you'd like to see changed. In principle, very little code is involved in actually writing to the file so the task would seem rather easy. However, WriterAppender on which FileAppender and RollingFileAppender are built upon uses a Writer to write whatever it is that needs to be written. Unfortunately, a Writer knows how to writes String or chars, but not bytes.
Nevertheless, I think it's feasible with some work...
Joern Huxhorn wrote:
Hi Ceki.
I've seen that you are working on the FileAppenders again. Have you seen this mail I wrote? What do you think about it?
Joern.
Begin forwarded message:
*From: *Joern Huxhorn <jhuxhorn@googlemail.com <mailto:jhuxhorn@googlemail.com> <mailto:jhuxhorn@googlemail.com <mailto:jhuxhorn@googlemail.com>>>
*Date: *23. April 2009 18:38:38 MESZ *To: *logback developers list <logback-dev@qos.ch <mailto:logback-dev@qos.ch> <mailto:logback-dev@qos.ch <mailto:logback-dev@qos.ch>>>
*Subject: **Question about a custom binary file appender.*
Hi Ceki.
I'd like to implement a file appender that writes the binary Lilith format, i.e. gzipped protobuf-serialized events, instead of Strings. I'd also like to have the same functionality that's supported by RollingFileAppender right now.
Unfortunately, there seems to be no way to simply write bytes instead of a String. How would you go from here? Reimplementing everything from the start seems to be a pretty bad idea.
What do you think about enhancing the RFA so it's using byte[] instead of Strings? The current behavior could be implemented using those methods + string.getBytes("UTF-8") or CharsetEncoder...
Any idea, suggestions?
Regards, Joern.
------------------------------------------------------------------------
_______________________________________________ logback-dev mailing list logback-dev@qos.ch <mailto:logback-dev@qos.ch> http://qos.ch/mailman/listinfo/logback-dev
-- Ceki Gülcü Logback: The reliable, generic, fast and flexible logging framework for Java. http://logback.qos.ch _______________________________________________ logback-dev mailing list logback-dev@qos.ch <mailto:logback-dev@qos.ch> http://qos.ch/mailman/listinfo/logback-dev
------------------------------------------------------------------------
_______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
-- Ceki Gülcü Logback: The reliable, generic, fast and flexible logging framework for Java. http://logback.qos.ch
On Sat, Aug 1, 2009 at 8:33 PM, Ceki Gulcu <ceki@qos.ch> wrote:
Thank you for reminding us about the prototype. I find it very informative and helpful.
As I see things, the Encoder interface and the various implementations you have written show that it is not sufficient to return a byte array for a given ILoggingEvent, the "encoder" also needs to know how to write to the underlying OutputStream, it may also need to decorate the underlying OutputStream with its own (e.g. ObjectOutputStream for serialization).
Consequently, I'd propose to rename the Encoder interface as ILoggingEventWriter. (In my mind at least, an Encoder is associated with transforming data into bytes.)
So here is my revised interface:
public interface Encoder <T extends OutputStream> { void write(ILoggingEvent event, T output) throws IOException; T decorate(OutputStream os) throws IOException; }
I guess you forgot to rename the interface itself ;-)
From http://en.wikipedia.org/wiki/Encoder "An encoder is a device, circuit, transducer, software program, algorithm or person that converts information from one format, or code to another..." So I think calling it Encoder is appropriate, but I have no string feeling about the name of the interface.
The terminology also used in MINA: http://mina.apache.org/report/trunk/apidocs/org/apache/mina/filter/codec/Pro...
What should be done about methods such as getFileHeader(), getFileFooter(), getPresentationHeader() and getPresentationFooter() from the Layout interface? My first reaction would be to rename getX() as writeX(OutputStream).
Sounds OK to me.
This is all I have time for at this moment but I would really like to pursue this exchange.
That's cool ! I have no idea how much work it is to support encoders in the logback.xml file. I mean, we should be able to tell logback to use a XXXAppender with a ZZZEncoder ? Regards Maarten
Cheers,
Maarten Bosteels wrote:
Hi Ceki and Joern,
Remember the Encoder interface we discussed in the past ? To make the aspect of encoding a LoggingEvent pluggable, and thus the Appender implementation more reusable.
I have small prototype here : http://tinyurl.com/encoder-interface http://tinyurl.com/encoder-example
regards, Maarten
On Fri, Jul 31, 2009 at 5:51 PM, Ceki Gulcu <ceki@qos.ch <mailto: ceki@qos.ch>> wrote:
Hi Joern,
Plese enter a bug report with as much information as you can about the code you'd like to see changed. In principle, very little code is involved in actually writing to the file so the task would seem rather easy. However, WriterAppender on which FileAppender and RollingFileAppender are built upon uses a Writer to write whatever it is that needs to be written. Unfortunately, a Writer knows how to writes String or chars, but not bytes.
Nevertheless, I think it's feasible with some work...
Joern Huxhorn wrote:
Hi Ceki.
I've seen that you are working on the FileAppenders again. Have you seen this mail I wrote? What do you think about it?
Joern.
Begin forwarded message:
*From: *Joern Huxhorn <jhuxhorn@googlemail.com <mailto:jhuxhorn@googlemail.com> <mailto:jhuxhorn@googlemail.com <mailto:jhuxhorn@googlemail.com>>>
*Date: *23. April 2009 18:38:38 MESZ *To: *logback developers list <logback-dev@qos.ch <mailto:logback-dev@qos.ch> <mailto:logback-dev@qos.ch <mailto:logback-dev@qos.ch>>>
*Subject: **Question about a custom binary file appender.*
Hi Ceki.
I'd like to implement a file appender that writes the binary Lilith format, i.e. gzipped protobuf-serialized events, instead of Strings. I'd also like to have the same functionality that's supported by RollingFileAppender right now.
Unfortunately, there seems to be no way to simply write bytes instead of a String. How would you go from here? Reimplementing everything from the start seems to be a pretty bad idea.
What do you think about enhancing the RFA so it's using byte[] instead of Strings? The current behavior could be implemented using those methods + string.getBytes("UTF-8") or CharsetEncoder...
Any idea, suggestions?
Regards, Joern.
------------------------------------------------------------------------
_______________________________________________ logback-dev mailing list logback-dev@qos.ch <mailto:logback-dev@qos.ch> http://qos.ch/mailman/listinfo/logback-dev
-- Ceki Gülcü Logback: The reliable, generic, fast and flexible logging framework for Java. http://logback.qos.ch _______________________________________________ logback-dev mailing list logback-dev@qos.ch <mailto:logback-dev@qos.ch> http://qos.ch/mailman/listinfo/logback-dev
------------------------------------------------------------------------
_______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
-- Ceki Gülcü Logback: The reliable, generic, fast and flexible logging framework for Java. http://logback.qos.ch _______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
Maarten Bosteels wrote:
I guess you forgot to rename the interface itself ;-)
Yes, I did. :-)
From http://en.wikipedia.org/wiki/Encoder "An encoder is a device, circuit, transducer, software program, algorithm or person that converts information from one format, or code to another..." So I think calling it Encoder is appropriate, but I have no string feeling about the name of the interface.
The terminology also used in MINA: http://mina.apache.org/report/trunk/apidocs/org/apache/mina/filter/codec/Pro...
If there is an established terminology for instance in Mina, than that should be taken into consideration. However, being unfamiliar with Mina, I was a little surprised by the use of the term Encoder.
I have no idea how much work it is to support encoders in the logback.xml file. I mean, we should be able to tell logback to use a XXXAppender with a ZZZEncoder ?
If you can somehow reasonably express it in XML format, Joran can parse it, at least that's my experience so far. So, I would not worry about the parsing part and would concentrate on the "programmatic" API.
Regards Maarten
-- Ceki Gülcü Logback: The reliable, generic, fast and flexible logging framework for Java. http://logback.qos.ch
First of all, sorry that I didn't reply earlier. I was pretty occupied. I'm fine with Maartens Encoder suggestion. I've done something similar in a different way, see http://sulky.huxhorn.de/projects/de.huxhorn.sulky.codec/apidocs/index.html mainly because I need the amount of bytes before actually writing them but I could easily encapsulate my Encoder in the above interface. I'm creating a byte[] because one entry consists of <Length of byte array as int><actual bytes>. This has the downside that it occupies more memory but can't be circumvented in my case. I'd need the ability to write a header followed by an arbitrary number of the above structure, one for each event. I'm not sure how header (needed) or footers (not needed in my case, and arguably a bit dangerous) would be implemented, yet. Thanks for the discussion! Regards, Jörn. On 02.08.2009, at 17:59, Ceki Gulcu wrote:
Maarten Bosteels wrote:
I guess you forgot to rename the interface itself ;-)
Yes, I did. :-)
From http://en.wikipedia.org/wiki/Encoder "An encoder is a device, circuit, transducer, software program, algorithm or person that converts information from one format, or code to another..." So I think calling it Encoder is appropriate, but I have no string feeling about the name of the interface. The terminology also used in MINA: http://mina.apache.org/report/trunk/apidocs/org/apache/mina/filter/codec/Pro...
If there is an established terminology for instance in Mina, than that should be taken into consideration. However, being unfamiliar with Mina, I was a little surprised by the use of the term Encoder.
I have no idea how much work it is to support encoders in the logback.xml file. I mean, we should be able to tell logback to use a XXXAppender with a ZZZEncoder ?
If you can somehow reasonably express it in XML format, Joran can parse it, at least that's my experience so far. So, I would not worry about the parsing part and would concentrate on the "programmatic" API.
Regards Maarten -- Ceki Gülcü Logback: The reliable, generic, fast and flexible logging framework for Java. http://logback.qos.ch
logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
participants (3)
-
Ceki Gulcu -
Joern Huxhorn -
Maarten Bosteels