RFC 5424 and Structured Data support.
I would appreciate it if you would review git://github.com/rgoers/logback.git and git://github.com/rgoers/slf4j.git and incorporate that into the "official" logback source. The forks include fixes for LBCLASSIC-116, LBCLASSIC-154, possibly LBCLASSIC-153. The SLF4J changes include support LBCLASSIC-116 by way of fixing SLF4J bug 148 and also includes fixes for SLF4J bugs 31 and 127. Both of my forks include all of the latest changes in your forks. Ralph
Just wondering if you have any feedback on this. I have a project starting this week that needs to use the support for RFC 5424.
Ceki, I still haven't gotten any feedback from you. I realize I have made a few changes since first asking about this but this is becoming more important. Ralph
Hello Ralph, I am afraid other other matters are keeping me busy. I will not be able to work on logback before next week. Best regards, Ralph Goers wrote:
Ceki,
I still haven't gotten any feedback from you. I realize I have made a few changes since first asking about this but this is becoming more important.
Ralph
-- Ceki Gülcü Logback: The reliable, generic, fast and flexible logging framework for Java. http://logback.qos.ch
Thanks for letting me know. Ralph On Oct 26, 2009, at 1:40 AM, Ceki Gulcu wrote:
Hello Ralph,
I am afraid other other matters are keeping me busy. I will not be able to work on logback before next week.
Best regards,
Ralph Goers wrote:
Ceki, I still haven't gotten any feedback from you. I realize I have made a few changes since first asking about this but this is becoming more important. Ralph
-- Ceki Gülcü Logback: The reliable, generic, fast and flexible logging framework for Java. http://logback.qos.ch _______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
Once again, I'd appreciate this being integrated in as soon as possible. Ralph On Oct 26, 2009, at 6:12 AM, Ralph Goers wrote:
Thanks for letting me know.
Ralph
On Oct 26, 2009, at 1:40 AM, Ceki Gulcu wrote:
Hello Ralph,
I am afraid other other matters are keeping me busy. I will not be able to work on logback before next week.
Best regards,
Ralph Goers wrote:
Ceki, I still haven't gotten any feedback from you. I realize I have made a few changes since first asking about this but this is becoming more important. Ralph
-- Ceki Gülcü Logback: The reliable, generic, fast and flexible logging framework for Java. http://logback.qos.ch _______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
Hi Ralph, I've looked at the changed you made in relation to RFC 5424. It's quite a significant architectural change. The RFC 5424 route is in the opposite direction of logback where one creates a new module for each new event type. This offers the benefit of type safety but has significant overhead from a packaging pov, whereas the 5424 route has little packaging overhead but omits type safety. The RFC 5424 approach is not to be discarded. However, I don't think it's a route we should take at this *immediate* juncture. I propose to put it aside for the time being and come back to it, possibly with a vengeance, some time later. BR
Hi Ceki, did you have the chance to take a look at my latest SLF4J bug #31 comment, i.e. http://github.com/huxi/slf4j/tree/slf4j-redesign/slf4j-n-api/ ? (I know, I'm probably a nuisance...) It adds methods like logger.debug(Message message) that would essentially solve issues like that. For the common appender, such a Message would simply return a formatted String but custom appenders are able to handle an implementation they know however they like. If you take a look at my branch then please keep in mind that, while being fully (as far as I can tell yet) functional, it's merely meant as a common base for further discussion. The package names as well as method-names like e.g. getOldLogger() are, of course, just placeholders for whatever they might be called later. The StructuredDataMessage is also just a placeholder. I imagine a certain amount of "default" SLF4J-Messages that would be known to all appenders with the ability to add more. Those default messages could be serialized as they are (if they implement Serializable, of course) while messages not contained in the respective package would be serialzed as Strings (or, more specifically, as SimpleMessage instances). Regards, Joern. On 04.12.2009, at 23:12, Ceki Gülcü wrote:
Hi Ralph,
I've looked at the changed you made in relation to RFC 5424. It's quite a significant architectural change. The RFC 5424 route is in the opposite direction of logback where one creates a new module for each new event type. This offers the benefit of type safety but has significant overhead from a packaging pov, whereas the 5424 route has little packaging overhead but omits type safety.
The RFC 5424 approach is not to be discarded. However, I don't think it's a route we should take at this *immediate* juncture. I propose to put it aside for the time being and come back to it, possibly with a vengeance, some time later.
BR _______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
Joern, I looked at this and I have a question and a concern. First, I didn't see any code in your logback fork to support the Message. How do you envision that being integrated into the LoggingEvent? The concern is that Logger now would require Java 5. I personally have no problem with a new release train for that but if it means holding off on integrating the things that don't require Java 5 then I'd prefer the vararg support be separated. Other than that, I think this is a better approach. Ralph On Dec 2, 2009, at 2:48 PM, Joern Huxhorn wrote:
Hi Ceki,
did you have the chance to take a look at my latest SLF4J bug #31 comment, i.e. http://github.com/huxi/slf4j/tree/slf4j-redesign/slf4j-n-api/ ? (I know, I'm probably a nuisance...)
It adds methods like logger.debug(Message message) that would essentially solve issues like that. For the common appender, such a Message would simply return a formatted String but custom appenders are able to handle an implementation they know however they like.
If you take a look at my branch then please keep in mind that, while being fully (as far as I can tell yet) functional, it's merely meant as a common base for further discussion. The package names as well as method-names like e.g. getOldLogger() are, of course, just placeholders for whatever they might be called later.
The StructuredDataMessage is also just a placeholder. I imagine a certain amount of "default" SLF4J-Messages that would be known to all appenders with the ability to add more. Those default messages could be serialized as they are (if they implement Serializable, of course) while messages not contained in the respective package would be serialzed as Strings (or, more specifically, as SimpleMessage instances).
Regards, Joern.
On 04.12.2009, at 23:12, Ceki Gülcü wrote:
Hi Ralph,
I've looked at the changed you made in relation to RFC 5424. It's quite a significant architectural change. The RFC 5424 route is in the opposite direction of logback where one creates a new module for each new event type. This offers the benefit of type safety but has significant overhead from a packaging pov, whereas the 5424 route has little packaging overhead but omits type safety.
The RFC 5424 approach is not to be discarded. However, I don't think it's a route we should take at this *immediate* juncture. I propose to put it aside for the time being and come back to it, possibly with a vengeance, some time later.
BR _______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
_______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
Hi Ralph On 03.12.2009, at 11:31, Ralph Goers wrote:
Joern,
I looked at this and I have a question and a concern.
First, I didn't see any code in your logback fork to support the Message. How do you envision that being integrated into the LoggingEvent?
Well, Logback would, if it's implementing that API, change it's LoggingEvent to have a Message instead of pattern+args. StructuredDataMessage would be just another message. One could argue if it should be contained in the set of standard messages (at the moment SimpleMessage (just an encapsulation of a String), ParameterizedMessage and JavaUtilFormatterMessage) or not. From your point of view this wouldn't really matter since you write a custom appender anyway, right? Other appenders would simply handle your Message type similar to a String.
The concern is that Logger now would require Java 5. I personally have no problem with a new release train for that but if it means holding off on integrating the things that don't require Java 5 then I'd prefer the vararg support be separated.
Yes, that's true. This is my suggestion about how to proceed concerning the Java5 issue in SLF4J. I think that slf4j-api should stay as it is so that people that still depend on <=1.4 are still having the chance to use SLF4J without being excluded from future updates/fixes. Since we must not reduce or enhance the original SLF4J API so neither users nor implementors are hurt, I thought it would probably be best to simply define an entirely new API without any restrictions whatsoever. So, unfortunately, this can't be added to the current org.slf4j.Logger. The idea is, that >=1.5 logging frameworks like Logback would implement the new API directly, using a wrapper to support the old one. The opposite is the case for <=1.4 logging frameworks. Those wouldn't have explicit support of the new API since it would introduce >=1.5 as a requirement. The new API would return a wrapper that'S actually using the old Logger. All work concerning the wrapping is already done in my branch. See http://github.com/huxi/slf4j/tree/slf4j-redesign/slf4j-n-api-new-integration... and http://github.com/huxi/slf4j/tree/slf4j-redesign/slf4j-n-api-old-integration... for examples. Another nice thing is the simplicity of new Logger implementations. This is quite obvious if you compare the TestLogger implementations of the two integration-tests.
Other than that, I think this is a better approach.
Thanks, Joern.
Ralph
On Dec 2, 2009, at 2:48 PM, Joern Huxhorn wrote:
Hi Ceki,
did you have the chance to take a look at my latest SLF4J bug #31 comment, i.e. http://github.com/huxi/slf4j/tree/slf4j-redesign/slf4j-n-api/ ? (I know, I'm probably a nuisance...)
It adds methods like logger.debug(Message message) that would essentially solve issues like that. For the common appender, such a Message would simply return a formatted String but custom appenders are able to handle an implementation they know however they like.
If you take a look at my branch then please keep in mind that, while being fully (as far as I can tell yet) functional, it's merely meant as a common base for further discussion. The package names as well as method-names like e.g. getOldLogger() are, of course, just placeholders for whatever they might be called later.
The StructuredDataMessage is also just a placeholder. I imagine a certain amount of "default" SLF4J-Messages that would be known to all appenders with the ability to add more. Those default messages could be serialized as they are (if they implement Serializable, of course) while messages not contained in the respective package would be serialzed as Strings (or, more specifically, as SimpleMessage instances).
Regards, Joern.
On 04.12.2009, at 23:12, Ceki Gülcü wrote:
Hi Ralph,
I've looked at the changed you made in relation to RFC 5424. It's quite a significant architectural change. The RFC 5424 route is in the opposite direction of logback where one creates a new module for each new event type. This offers the benefit of type safety but has significant overhead from a packaging pov, whereas the 5424 route has little packaging overhead but omits type safety.
The RFC 5424 approach is not to be discarded. However, I don't think it's a route we should take at this *immediate* juncture. I propose to put it aside for the time being and come back to it, possibly with a vengeance, some time later.
BR _______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
_______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
_______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
On Dec 3, 2009, at 3:26 AM, Joern Huxhorn wrote:
Hi Ralph
On 03.12.2009, at 11:31, Ralph Goers wrote:
Joern,
I looked at this and I have a question and a concern.
First, I didn't see any code in your logback fork to support the Message. How do you envision that being integrated into the LoggingEvent?
Well, Logback would, if it's implementing that API, change it's LoggingEvent to have a Message instead of pattern+args. StructuredDataMessage would be just another message. One could argue if it should be contained in the set of standard messages (at the moment SimpleMessage (just an encapsulation of a String), ParameterizedMessage and JavaUtilFormatterMessage) or not.
From your point of view this wouldn't really matter since you write a custom appender anyway, right? Other appenders would simply handle your Message type similar to a String.
Actually, the IETFSyslogLayout and IETFSyslogAppender would be a base part of logback. But these are just new components and obviously wouldn't affect anything new. JMSAppender was modified to accept a layout instead of always sending a serialized LoggingEvent.
The concern is that Logger now would require Java 5. I personally have no problem with a new release train for that but if it means holding off on integrating the things that don't require Java 5 then I'd prefer the vararg support be separated.
Yes, that's true. This is my suggestion about how to proceed concerning the Java5 issue in SLF4J. I think that slf4j-api should stay as it is so that people that still depend on <=1.4 are still having the chance to use SLF4J without being excluded from future updates/fixes.
Since we must not reduce or enhance the original SLF4J API so neither users nor implementors are hurt, I thought it would probably be best to simply define an entirely new API without any restrictions whatsoever. So, unfortunately, this can't be added to the current org.slf4j.Logger.
The idea is, that >=1.5 logging frameworks like Logback would implement the new API directly, using a wrapper to support the old one. The opposite is the case for <=1.4 logging frameworks. Those wouldn't have explicit support of the new API since it would introduce >=1.5 as a requirement. The new API would return a wrapper that'S actually using the old Logger.
I don't think you can actually do that in a single jar but I could be wrong. You'd have to build with source = 1.5 and target = 1.3. I think it would complain about the varargs. In your project your pom.xml specifies source = 1.5 and target = 1.5 so applications trying to use it on <= 1.4 are going to get a wrong version exception. If it did build then that would imply you don't really need the wrapper.
All work concerning the wrapping is already done in my branch. See http://github.com/huxi/slf4j/tree/slf4j-redesign/slf4j-n-api-new-integration... and http://github.com/huxi/slf4j/tree/slf4j-redesign/slf4j-n-api-old-integration... for examples.
Another nice thing is the simplicity of new Logger implementations. This is quite obvious if you compare the TestLogger implementations of the two integration-tests.
I'll try to put this stuff in my branch and see what happens. Ralph
On Dec 3, 2009, at 6:33 AM, Ralph Goers wrote:
On Dec 3, 2009, at 3:26 AM, Joern Huxhorn wrote:
All work concerning the wrapping is already done in my branch. See http://github.com/huxi/slf4j/tree/slf4j-redesign/slf4j-n-api-new-integration... and http://github.com/huxi/slf4j/tree/slf4j-redesign/slf4j-n-api-old-integration... for examples.
Another nice thing is the simplicity of new Logger implementations. This is quite obvious if you compare the TestLogger implementations of the two integration-tests.
I'll try to put this stuff in my branch and see what happens.
I've modified my fork to support Messages rather than just StructuredData. The SLF4J api is backward compatible. Logback isn't completely. Calling getMessage on a LoggingEvent now returns the Message object, not the message format string. To get that you call getMessage().getMessageFormat(). I actually like this quite a bit more than trying to add the StructuredData as a parameter. I would have liked to do more but it would have meant breaking the TurboFilter interface and I didn't want to do that. The only issue I found was with LoggingEventSerializationPerfTest. It failed because averageSize exceeds averageSizeLimit. I've looked at that code but for the life of me I can't understand what it is attempting to validate. The size seems to be calculated by incrementing a counter everyting NoopOutputStream's write method is called. My guess is that this is an attempt to determine the length of the serialized data. The trouble is, when I changed the output stream to a FileOutputStream and serialized the object to it then size of the file was not even close to the value returned by NoopOutputStream's size method. Is that what is being attempted here? What is the real value in limiting the size? When I looked at the test the limit on the first test is 60 while the actual is 58. With the Message object replacing the message string and parameters the value is now 68. Ralph
Ralph Goers wrote:
The only issue I found was with LoggingEventSerializationPerfTest. It failed because averageSize exceeds averageSizeLimit. I've looked at that code but for the life of me I can't understand what it is attempting to validate. The size seems to be calculated by incrementing a counter everyting NoopOutputStream's write method is called. My guess is that this is an attempt to determine the length of the serialized data. The trouble is, when I changed the output stream to a FileOutputStream and serialized the object to it then size of the file was not even close to the value returned by NoopOutputStream's size method. Is that what is being attempted here? What is the real value in limiting the size? When I looked at the test the limit on the first test is 60 while the actual is 58. With the Message object replacing the message string and parameters the value is now 68.
LoggingEventSerializationPerfTest serves as a regression test. The idea was to catch unsolicited increases (regressions) in the size of the serialized messages. There is nothing sacred about the number 60.
Ralph
On Dec 13, 2009, at 2:14 PM, Ceki Gülcü wrote:
Ralph Goers wrote:
The only issue I found was with LoggingEventSerializationPerfTest. It failed because averageSize exceeds averageSizeLimit. I've looked at that code but for the life of me I can't understand what it is attempting to validate. The size seems to be calculated by incrementing a counter everyting NoopOutputStream's write method is called. My guess is that this is an attempt to determine the length of the serialized data. The trouble is, when I changed the output stream to a FileOutputStream and serialized the object to it then size of the file was not even close to the value returned by NoopOutputStream's size method. Is that what is being attempted here? What is the real value in limiting the size? When I looked at the test the limit on the first test is 60 while the actual is 58. With the Message object replacing the message string and parameters the value is now 68.
LoggingEventSerializationPerfTest serves as a regression test. The idea was to catch unsolicited increases (regressions) in the size of the serialized messages. There is nothing sacred about the number 60.
Ok. If you'd care to take a look it is available at git://github.com/rgoers/slf4j.git and git://github.com/rgoers/logback.git. Ralph
On Dec 13, 2009, at 4:45 PM, Ralph Goers wrote:
On Dec 13, 2009, at 2:14 PM, Ceki Gülcü wrote:
Ralph Goers wrote:
The only issue I found was with LoggingEventSerializationPerfTest. It failed because averageSize exceeds averageSizeLimit. I've looked at that code but for the life of me I can't understand what it is attempting to validate. The size seems to be calculated by incrementing a counter everyting NoopOutputStream's write method is called. My guess is that this is an attempt to determine the length of the serialized data. The trouble is, when I changed the output stream to a FileOutputStream and serialized the object to it then size of the file was not even close to the value returned by NoopOutputStream's size method. Is that what is being attempted here? What is the real value in limiting the size? When I looked at the test the limit on the first test is 60 while the actual is 58. With the Message object replacing the message string and parameters the value is now 68.
LoggingEventSerializationPerfTest serves as a regression test. The idea was to catch unsolicited increases (regressions) in the size of the serialized messages. There is nothing sacred about the number 60.
Ok. If you'd care to take a look it is available at git://github.com/rgoers/slf4j.git and git://github.com/rgoers/logback.git.
Ralph
Merry Christmas! I've made some minor changes to this. I'd appreciate a review of this and comments if you can find the time. Ralph
On Dec 24, 2009, at 3:59 PM, Ralph Goers wrote:
On Dec 13, 2009, at 4:45 PM, Ralph Goers wrote:
On Dec 13, 2009, at 2:14 PM, Ceki Gülcü wrote:
Ralph Goers wrote:
The only issue I found was with LoggingEventSerializationPerfTest. It failed because averageSize exceeds averageSizeLimit. I've looked at that code but for the life of me I can't understand what it is attempting to validate. The size seems to be calculated by incrementing a counter everyting NoopOutputStream's write method is called. My guess is that this is an attempt to determine the length of the serialized data. The trouble is, when I changed the output stream to a FileOutputStream and serialized the object to it then size of the file was not even close to the value returned by NoopOutputStream's size method. Is that what is being attempted here? What is the real value in limiting the size? When I looked at the test the limit on the first test is 60 while the actual is 58. With the Message object replacing the message string and parameters the value is now 68.
LoggingEventSerializationPerfTest serves as a regression test. The idea was to catch unsolicited increases (regressions) in the size of the serialized messages. There is nothing sacred about the number 60.
Ok. If you'd care to take a look it is available at git://github.com/rgoers/slf4j.git and git://github.com/rgoers/logback.git.
Ralph
Merry Christmas!
I've made some minor changes to this. I'd appreciate a review of this and comments if you can find the time.
Ralph
I hope you had a happy new year. I'm still hoping you can find time to review this. Ralph
On Jan 6, 2010, at 5:41 PM, Ralph Goers wrote:
On Dec 24, 2009, at 3:59 PM, Ralph Goers wrote:
On Dec 13, 2009, at 4:45 PM, Ralph Goers wrote:
On Dec 13, 2009, at 2:14 PM, Ceki Gülcü wrote:
Ralph Goers wrote:
The only issue I found was with LoggingEventSerializationPerfTest. It failed because averageSize exceeds averageSizeLimit. I've looked at that code but for the life of me I can't understand what it is attempting to validate. The size seems to be calculated by incrementing a counter everyting NoopOutputStream's write method is called. My guess is that this is an attempt to determine the length of the serialized data. The trouble is, when I changed the output stream to a FileOutputStream and serialized the object to it then size of the file was not even close to the value returned by NoopOutputStream's size method. Is that what is being attempted here? What is the real value in limiting the size? When I looked at the test the limit on the first test is 60 while the actual is 58. With the Message object replacing the message string and parameters the value is now 68.
LoggingEventSerializationPerfTest serves as a regression test. The idea was to catch unsolicited increases (regressions) in the size of the serialized messages. There is nothing sacred about the number 60.
Ok. If you'd care to take a look it is available at git://github.com/rgoers/slf4j.git and git://github.com/rgoers/logback.git.
Ralph
Merry Christmas!
I've made some minor changes to this. I'd appreciate a review of this and comments if you can find the time.
Ralph
I hope you had a happy new year. I'm still hoping you can find time to review this.
Ralph
Once again, I would appreciate feedback on this. I would especially value your input Joern since the move to using a Message was your idea. Ralph
Sorry, I'll have a look at it soon. I was kind of occupied with Twitter stuff and right now I have a head that merely fits through a door due to some cold. I guess I'll like your changes from what I've read previously, though. Regards, Joern. On 18.01.2010, at 18:27, Ralph Goers <rgoers@apache.org> wrote:
On Jan 6, 2010, at 5:41 PM, Ralph Goers wrote:
On Dec 24, 2009, at 3:59 PM, Ralph Goers wrote:
On Dec 13, 2009, at 4:45 PM, Ralph Goers wrote:
On Dec 13, 2009, at 2:14 PM, Ceki Gülcü wrote:
Ralph Goers wrote:
The only issue I found was with LoggingEventSerializationPerfTest. It failed because averageSize exceeds averageSizeLimit. I've looked at that code but for the life of me I can't understand what it is attempting to validate. The size seems to be calculated by incrementing a counter everyting NoopOutputStream's write method is called. My guess is that this is an attempt to determine the length of the serialized data. The trouble is, when I changed the output stream to a FileOutputStream and serialized the object to it then size of the file was not even close to the value returned by NoopOutputStream's size method. Is that what is being attempted here? What is the real value in limiting the size? When I looked at the test the limit on the first test is 60 while the actual is 58. With the Message object replacing the message string and parameters the value is now 68.
LoggingEventSerializationPerfTest serves as a regression test. The idea was to catch unsolicited increases (regressions) in the size of the serialized messages. There is nothing sacred about the number 60.
Ok. If you'd care to take a look it is available at git:// github.com/rgoers/slf4j.git and git://github.com/rgoers/ logback.git.
Ralph
Merry Christmas!
I've made some minor changes to this. I'd appreciate a review of this and comments if you can find the time.
Ralph
I hope you had a happy new year. I'm still hoping you can find time to review this.
Ralph
Once again, I would appreciate feedback on this. I would especially value your input Joern since the move to using a Message was your idea.
Ralph
_______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
Ok, I've had time to take a look now. My plan was that the logback-classic Logger would simply extend org.slf4j.n.helper.AbstractLoggerBase. That way, only the following 3 methods would need implementation: Threshold getThreshold() boolean isEnabled(Level level, Marker marker) void log(Level level, Marker marker, Message message, Throwable throwable) The log-method above would just call filterAndLog(String localFQCN, Marker marker, Level level, Message msg, Throwable t). The implementation of old and new SLF4J-API implementations must be separated so it's possible to either call org.slf4j.n.LoggerFactory (for org.slf4j.n.Logger) or org.slf4j.LoggerFactory (for org.slf4j.Logger) while deserializing a Logger, i.e. the Logback Logger implementation should *not* implement org.slf4j.Logger anymore. Instead, org.slf4j.impl.StaticLoggerBinder should return org.slf4j.n.helpers.UsingNewLoggerFactory which is implemented like this: Logger getLogger(String name) { return org.slf4j.n.LoggerFactory.getLogger(name).getOldLogger(); } Does this help as an explanation so far? Regards, Joern. On 18.01.2010, at 18:27, Ralph Goers wrote:
On Jan 6, 2010, at 5:41 PM, Ralph Goers wrote:
On Dec 24, 2009, at 3:59 PM, Ralph Goers wrote:
On Dec 13, 2009, at 4:45 PM, Ralph Goers wrote:
On Dec 13, 2009, at 2:14 PM, Ceki Gülcü wrote:
Ralph Goers wrote:
The only issue I found was with LoggingEventSerializationPerfTest. It failed because averageSize exceeds averageSizeLimit. I've looked at that code but for the life of me I can't understand what it is attempting to validate. The size seems to be calculated by incrementing a counter everyting NoopOutputStream's write method is called. My guess is that this is an attempt to determine the length of the serialized data. The trouble is, when I changed the output stream to a FileOutputStream and serialized the object to it then size of the file was not even close to the value returned by NoopOutputStream's size method. Is that what is being attempted here? What is the real value in limiting the size? When I looked at the test the limit on the first test is 60 while the actual is 58. With the Message object replacing the message string and parameters the value is now 68.
LoggingEventSerializationPerfTest serves as a regression test. The idea was to catch unsolicited increases (regressions) in the size of the serialized messages. There is nothing sacred about the number 60.
Ok. If you'd care to take a look it is available at git:// github.com/rgoers/slf4j.git and git://github.com/rgoers/ logback.git.
Ralph
Merry Christmas!
I've made some minor changes to this. I'd appreciate a review of this and comments if you can find the time.
Ralph
I hope you had a happy new year. I'm still hoping you can find time to review this.
Ralph
Once again, I would appreciate feedback on this. I would especially value your input Joern since the move to using a Message was your idea.
Ralph
_______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
In reading your response I'm not sure if you are just describing how you originally thought this should be done or critiquing what I've done. In your analysis below I really don't see any comments about what is wrong but rather, how you had anticipated it would be done. I've tried to answer the points in your email as best I can, but the main issue is that it seemed to me that your proposal was for a new version of SLF4J whereas I am trying to make the changes without the need for a major release or breaking any compatibility. On Jan 25, 2010, at 3:40 AM, Joern Huxhorn wrote:
Ok, I've had time to take a look now.
My plan was that the logback-classic Logger would simply extend org.slf4j.n.helper.AbstractLoggerBase.
I saw that in your original note. That class doesn't currently exist. I assume you would have created it. As I recall I thought about that but decided against it. Logback has a lot of logic in its Logger class and implementing AbstractLoggerBase in SLF4J doesn't save much as you might think. In addition, all the current Loggers implement the interface. Creating a new base class that includes the Message API and then having Logback extend that creates serious problems
That way, only the following 3 methods would need implementation: Threshold getThreshold() boolean isEnabled(Level level, Marker marker) void log(Level level, Marker marker, Message message, Throwable throwable)
That isn't really true. Again, look at the Logger class. While the public methods like info(), debug() etc, could be factored out there are a lot of other methods that can't be in an abstract class.
The log-method above would just call filterAndLog(String localFQCN, Marker marker, Level level, Message msg, Throwable t).
The implementation of old and new SLF4J-API implementations must be separated so it's possible to either call org.slf4j.n.LoggerFactory (for org.slf4j.n.Logger) or org.slf4j.LoggerFactory (for org.slf4j.Logger) while deserializing a Logger, i.e. the Logback Logger implementation should *not* implement org.slf4j.Logger anymore.
There are some issues with this. LoggerFactory returns a Logger. To maintain compatibility this interface cannot be changed. Instead I created MessageLoggerFactory which returns a MessageLogger which extends LocationAwareLogger. Logback's Logger now implements MessageLogger instead of LocationAwareLogger. Other applications, such as Sling, which implement the Logger interface will not be impacted by this at all. I don't see how using org.slf4j.message.MessageLoggerFactory is much different than using org.slf4j.new.LoggerFactory, except I believe my name is a bit clearer. Logback doesn't implement Logger directly in either Ceki's fork or mine. Currently, it implements LocationAwareLogger. In mine, MessageLogger.
Instead, org.slf4j.impl.StaticLoggerBinder should return org.slf4j.n.helpers.UsingNewLoggerFactory which is implemented like this: Logger getLogger(String name) { return org.slf4j.n.LoggerFactory.getLogger(name).getOldLogger(); }
Why? I left the existing getLogger alone. Instead the MessageLogger is a LoggerWrapper, so the existing API is untouched.
Does this help as an explanation so far?
Not really. I'm working under the premise that a) SLF4J is not being upgraded to Java 5 b) no new SLF4J version is required because c) backward compatibility must be maintained. If I was not working with those constraints I certainly would have made a few different choices, but then I'm sure I would have had a snowballs chance in hell of getting it adopted since it would break so much stuff. Ralph
Hi Ralph, On 28.01.2010, at 08:42, Ralph Goers wrote:
In reading your response I'm not sure if you are just describing how you originally thought this should be done or critiquing what I've done. In your analysis below I really don't see any comments about what is wrong but rather, how you had anticipated it would be done. I've tried to answer the points in your email as best I can, but the main issue is that it seemed to me that your proposal was for a new version of SLF4J whereas I am trying to make the changes without the need for a major release or breaking any compatibility.
I assumed you tried to do the Logback counterpart of my SLF4J redesign proposal - which wouldn't break compatibility but introduces an additional artifact (tentatively called slf4j-n-api). Instead of depending on slf4j-api, Logback classic would instead depend on slf4j-n-api (which, in turn, introduces the slf4j-api dependency for compatibility reasons).
On Jan 25, 2010, at 3:40 AM, Joern Huxhorn wrote:
Ok, I've had time to take a look now.
My plan was that the logback-classic Logger would simply extend org.slf4j.n.helper.AbstractLoggerBase.
I saw that in your original note. That class doesn't currently exist. I assume you would have created it. As I recall I thought about that but decided against it. Logback has a lot of logic in its Logger class and implementing AbstractLoggerBase in SLF4J doesn't save much as you might think. In addition, all the current Loggers implement the interface. Creating a new base class that includes the Message API and then having Logback extend that creates serious problems
But it does exist: http://github.com/huxi/slf4j/blob/slf4j-redesign/slf4j-n-api/src/main/java/o... Is it possible that you did not look at the slf4j-redesign branch but master? The Logback classic Logger isn't extending anything at the moment and the AbstractLoggerBase is simply implementing everything that will likely stay the same for all Logger implementations. It also supports proper deserialization of Logger instances already. It's not necessary to extend it but it's certainly easier to do so and doing so wouldn't introduce any problems in case of the Logback classic Logger. It would simply remove a lot of boilerplate code from the Logger implementation.
That way, only the following 3 methods would need implementation: Threshold getThreshold() boolean isEnabled(Level level, Marker marker) void log(Level level, Marker marker, Message message, Throwable throwable)
That isn't really true. Again, look at the Logger class. While the public methods like info(), debug() etc, could be factored out there are a lot of other methods that can't be in an abstract class.
Those would stay in the actual Logger implementation, unchanged, as before.
The log-method above would just call filterAndLog(String localFQCN, Marker marker, Level level, Message msg, Throwable t).
The implementation of old and new SLF4J-API implementations must be separated so it's possible to either call org.slf4j.n.LoggerFactory (for org.slf4j.n.Logger) or org.slf4j.LoggerFactory (for org.slf4j.Logger) while deserializing a Logger, i.e. the Logback Logger implementation should *not* implement org.slf4j.Logger anymore.
There are some issues with this. LoggerFactory returns a Logger. To maintain compatibility this interface cannot be changed. Instead I created MessageLoggerFactory which returns a MessageLogger which extends LocationAwareLogger. Logback's Logger now implements MessageLogger instead of LocationAwareLogger. Other applications, such as Sling, which implement the Logger interface will not be impacted by this at all. I don't see how using org.slf4j.message.MessageLoggerFactory is much different than using org.slf4j.new.LoggerFactory, except I believe my name is a bit clearer.
The original Logger interface is unchanged and would stay that way. No other implementation would be affected at all since they would be automatically wrapped the other way around, i.e. there's a n.ILoggerFactory implementation already implemented that is used automatically if no explicit implementation is available. It wraps an original Logger into the n.Logger interface. Both scenarios are tested in http://github.com/huxi/slf4j/tree/slf4j-redesign/slf4j-n-api-new-integration... and http://github.com/huxi/slf4j/tree/slf4j-redesign/slf4j-n-api-old-integration...
Logback doesn't implement Logger directly in either Ceki's fork or mine. Currently, it implements LocationAwareLogger. In mine, MessageLogger.
There's an additional (optional) interface for that, too: http://github.com/huxi/slf4j/blob/slf4j-redesign/slf4j-n-api/src/main/java/o...
Instead, org.slf4j.impl.StaticLoggerBinder should return org.slf4j.n.helpers.UsingNewLoggerFactory which is implemented like this: Logger getLogger(String name) { return org.slf4j.n.LoggerFactory.getLogger(name).getOldLogger(); }
Why? I left the existing getLogger alone. Instead the MessageLogger is a LoggerWrapper, so the existing API is untouched.
Because Logger and n.Logger are compiletime- but not binary-compatible. Assuming that more and more modules would switch to the n API it means that more and more Loggers are used without an additional wrapper. The current LoggerFactory-related code would be moved to org.slf4j.n.impl.StaticLoggerBinder and an org.slf4j.n.ILoggerFactory implementation with very minimal changes.
Does this help as an explanation so far?
Not really. I'm working under the premise that a) SLF4J is not being upgraded to Java 5
Yes, only an additional, optional module would be added. The original SLF4J API would not require Java 5.
b) no new SLF4J version is required because c) backward compatibility must be maintained.
That's the whole point. By leaving the original API untouched and adding an additional one we'd have the possibility to change the API without constraints. My constraining goal was compiletime compatibility, but not binary compatibility. Binary compatibility simply can't be achieved without sacrificing JDK<1.5 compatibility in SLF4J API - and I think that would be a bad idea. So I aimed for the next-best thing: compiletime compatibility requiring only a change of the Logger and LoggerFactory import. All modules using the original API could still do so while other that use JDK>=1.5 could opt-in using the "modern" API by simply changing the import. I've put a lot of thought into that whole concept. One benefit of this approach is that it does not require the user to learn anything new. If he's not interested in the new API he'll be perfectly fine ignoring it entirely. If, on the other hand, he has the need and/or motivation for the new Message/Varargs logging then a very simple upgrade path is in existence. It can't get any simpler than just adding a .n to the imports (given the constraint that the original API must be left unchanged). The .n is just a suggestion, though. I really don't care how that package should be named or if everything is put into a new namespace like org.lf4j. What I tried to achieve is that the class-names don't have to be changed, just the import/package-name.
If I was not working with those constraints I certainly would have made a few different choices, but then I'm sure I would have had a snowballs chance in hell of getting it adopted since it would break so much stuff.
It wouldn't break anything. I tested it. If you can find any breakage please let me know but I'm quite confident that I didn't miss any point this time around. For example: during the evolution of SLF4J bug #31 I suggested that n.Logger should extend Logger. This would have resulted in breakage since it wouldn't have been clear, during deserialization, what is expected by the deserializing code. readResolve() needs to execute LoggerFactory.getLogger(loggerName). How should we decide about wrapping or not without creating kludgy code? Wrapping it each and every time if n.Logger wasn't implemented by the returned instance? Nope, this would have been quite bad. Cheers, Joern.
On Jan 28, 2010, at 3:06 AM, Joern Huxhorn wrote:
Hi Ralph,
On 28.01.2010, at 08:42, Ralph Goers wrote:
In reading your response I'm not sure if you are just describing how you originally thought this should be done or critiquing what I've done. In your analysis below I really don't see any comments about what is wrong but rather, how you had anticipated it would be done. I've tried to answer the points in your email as best I can, but the main issue is that it seemed to me that your proposal was for a new version of SLF4J whereas I am trying to make the changes without the need for a major release or breaking any compatibility.
I assumed you tried to do the Logback counterpart of my SLF4J redesign proposal - which wouldn't break compatibility but introduces an additional artifact (tentatively called slf4j-n-api). Instead of depending on slf4j-api, Logback classic would instead depend on slf4j-n-api (which, in turn, introduces the slf4j-api dependency for compatibility reasons).
I tried to implement your proposal to some degree. The most important part to me was introducing the Message concept. For minimal impact on users I did not want to introduce or change any dependencies, other than requiring version changes.
On Jan 25, 2010, at 3:40 AM, Joern Huxhorn wrote:
Ok, I've had time to take a look now.
My plan was that the logback-classic Logger would simply extend org.slf4j.n.helper.AbstractLoggerBase.
I saw that in your original note. That class doesn't currently exist. I assume you would have created it. As I recall I thought about that but decided against it. Logback has a lot of logic in its Logger class and implementing AbstractLoggerBase in SLF4J doesn't save much as you might think. In addition, all the current Loggers implement the interface. Creating a new base class that includes the Message API and then having Logback extend that creates serious problems
But it does exist: http://github.com/huxi/slf4j/blob/slf4j-redesign/slf4j-n-api/src/main/java/o...
Is it possible that you did not look at the slf4j-redesign branch but master?
Sorry. It has been quite a while since I wrote the code. Yes, I looked at your code. IIRC the reason I didn't use it was that I don't like calling the isEnabled methods that way. The log method is going to result it Logback calling filterAndLog which results in filtering being performed twice for each call. If the log method doesn't call filterAndLog then the behavior of the log method would be different than how Logback currently behaves. Of course, not calling isEnabled would result in creating ParameterizedMessages when no logging will occur. My implementation doesn't have this problem.
The Logback classic Logger isn't extending anything at the moment and the AbstractLoggerBase is simply implementing everything that will likely stay the same for all Logger implementations. It also supports proper deserialization of Logger instances already.
It's not necessary to extend it but it's certainly easier to do so and doing so wouldn't introduce any problems in case of the Logback classic Logger. It would simply remove a lot of boilerplate code from the Logger implementation.
That would be true if my comment above wasn't true.
b) no new SLF4J version is required because c) backward compatibility must be maintained.
That's the whole point. By leaving the original API untouched and adding an additional one we'd have the possibility to change the API without constraints. My constraining goal was compiletime compatibility, but not binary compatibility.
Binary compatibility simply can't be achieved without sacrificing JDK<1.5 compatibility in SLF4J API - and I think that would be a bad idea. So I aimed for the next-best thing: compiletime compatibility requiring only a change of the Logger and LoggerFactory import.
??? My code should be maintaining binary compatibility. Applications written to older versions of SLF4J can drop in the new jars with no change. New code can use the new API. They can coexist happily.
If I was not working with those constraints I certainly would have made a few different choices, but then I'm sure I would have had a snowballs chance in hell of getting it adopted since it would break so much stuff.
It wouldn't break anything. I tested it.
I won't believe that until I see a fork of Logback that depends on your SLF4J implementation. It was very difficult to get Logback to accept the Message and deal with it correctly.
If you can find any breakage please let me know but I'm quite confident that I didn't miss any point this time around.
For example: during the evolution of SLF4J bug #31 I suggested that n.Logger should extend Logger. This would have resulted in breakage since it wouldn't have been clear, during deserialization, what is expected by the deserializing code. readResolve() needs to execute LoggerFactory.getLogger(loggerName). How should we decide about wrapping or not without creating kludgy code? Wrapping it each and every time if n.Logger wasn't implemented by the returned instance? Nope, this would have been quite bad.
If you notice there really is no extra overhead since in my fork Logback's getLogger() always returns a MessageLogger. It isn't really wrapped. No kludgy code is needed. Since MessageLogger extends Logger anyone calling LoggerFactory.getLogger() will work just fine. The wrapper is only needed if a Logger implementation returns a Logger. Then MessageLoggerFactory will wrap it. Ralph
On 29.01.2010, at 04:43, Ralph Goers wrote:
On Jan 28, 2010, at 3:06 AM, Joern Huxhorn wrote:
Hi Ralph,
On 28.01.2010, at 08:42, Ralph Goers wrote:
In reading your response I'm not sure if you are just describing how you originally thought this should be done or critiquing what I've done. In your analysis below I really don't see any comments about what is wrong but rather, how you had anticipated it would be done. I've tried to answer the points in your email as best I can, but the main issue is that it seemed to me that your proposal was for a new version of SLF4J whereas I am trying to make the changes without the need for a major release or breaking any compatibility.
I assumed you tried to do the Logback counterpart of my SLF4J redesign proposal - which wouldn't break compatibility but introduces an additional artifact (tentatively called slf4j-n-api). Instead of depending on slf4j-api, Logback classic would instead depend on slf4j-n-api (which, in turn, introduces the slf4j-api dependency for compatibility reasons).
I tried to implement your proposal to some degree. The most important part to me was introducing the Message concept. For minimal impact on users I did not want to introduce or change any dependencies, other than requiring version changes.
On Jan 25, 2010, at 3:40 AM, Joern Huxhorn wrote:
Ok, I've had time to take a look now.
My plan was that the logback-classic Logger would simply extend org.slf4j.n.helper.AbstractLoggerBase.
I saw that in your original note. That class doesn't currently exist. I assume you would have created it. As I recall I thought about that but decided against it. Logback has a lot of logic in its Logger class and implementing AbstractLoggerBase in SLF4J doesn't save much as you might think. In addition, all the current Loggers implement the interface. Creating a new base class that includes the Message API and then having Logback extend that creates serious problems
But it does exist: http://github.com/huxi/slf4j/blob/slf4j-redesign/slf4j-n-api/src/main/java/o...
Is it possible that you did not look at the slf4j-redesign branch but master?
Sorry. It has been quite a while since I wrote the code. Yes, I looked at your code. IIRC the reason I didn't use it was that I don't like calling the isEnabled methods that way. The log method is going to result it Logback calling filterAndLog which results in filtering being performed twice for each call. If the log method doesn't call filterAndLog then the behavior of the log method would be different than how Logback currently behaves. Of course, not calling isEnabled would result in creating ParameterizedMessages when no logging will occur. My implementation doesn't have this problem.
Good point. I've changed AbstractLoggerBase a bit. It now directly implements log(level, marker, message, throwable) and defines protected abstract void performLogging(level, marker, message, throwable) An implementation is expected to not call isEnabled before performing additional filtering and logging. What I don't know right now: Is it possible to reenable logging that would be disabled due to level by using turbo-filters? This wouldn't work anymore using my AbstractLoggerBase. (which doesn't mean that the rest of the concept doesn't work) Calling isEnabled was indeed meant to prevent creation of messages that won't be logged. This could probably be prevented by another method like protected abstract boolean isEnabled(level, marker, messagePattern, arguments) that implemented the filtering logic. I've never used TurboFilters so I tend to forget about their powers. One problem of your implementation that I've found in the meantime is that you are not using the Throwable that might have been detected by ParameterizedMessage. The logic looks like this: public void log(Level level, Marker marker, Message message) { if (!isEnabled(level, marker)) return; if (message instanceof ParameterizedMessage) { performLogging(level, marker, message, ((ParameterizedMessage) message).getThrowable()); } else { performLogging(level, marker, message, null); } } This is necessary to use both varargs and Throwables at the same time. Assuming that TurboFilters can act on the optional Throwables, the Message needs to be created before filtering. Assuming the filters can NOT override a level, checking for level and marker using isEnabled(level, marker) makes sense before filtering to prevent the Message creation. I really don't know.
The Logback classic Logger isn't extending anything at the moment and the AbstractLoggerBase is simply implementing everything that will likely stay the same for all Logger implementations. It also supports proper deserialization of Logger instances already.
It's not necessary to extend it but it's certainly easier to do so and doing so wouldn't introduce any problems in case of the Logback classic Logger. It would simply remove a lot of boilerplate code from the Logger implementation.
That would be true if my comment above wasn't true.
b) no new SLF4J version is required because c) backward compatibility must be maintained.
That's the whole point. By leaving the original API untouched and adding an additional one we'd have the possibility to change the API without constraints. My constraining goal was compiletime compatibility, but not binary compatibility.
Binary compatibility simply can't be achieved without sacrificing JDK<1.5 compatibility in SLF4J API - and I think that would be a bad idea. So I aimed for the next-best thing: compiletime compatibility requiring only a change of the Logger and LoggerFactory import.
??? My code should be maintaining binary compatibility. Applications written to older versions of SLF4J can drop in the new jars with no change. New code can use the new API. They can coexist happily.
With my concept it stays binary compatible for the old API, i.e. org.slf4j.Logger and org.slf4j.LoggerFactory. Any code built against those would simply work but couldn't use the Message logging and varargs since the original SLF4J API is frozen and JDK14. If you want Varargs (incl. Throwable evaluation) & Message you'd use org.slf4j.n.Logger and org.slf4j.n.LoggerFactory. The n-API is Java 5, the original one isn't.
If I was not working with those constraints I certainly would have made a few different choices, but then I'm sure I would have had a snowballs chance in hell of getting it adopted since it would break so much stuff.
It wouldn't break anything. I tested it.
I won't believe that until I see a fork of Logback that depends on your SLF4J implementation. It was very difficult to get Logback to accept the Message and deal with it correctly.
Well, you did the hardest part of the work already, i.e. the Message handling throughout the rest of the API. I'd honestly like to have some input from Ceki because I don't want to invest too much time if all of this ends in a "nope".
If you can find any breakage please let me know but I'm quite confident that I didn't miss any point this time around.
For example: during the evolution of SLF4J bug #31 I suggested that n.Logger should extend Logger. This would have resulted in breakage since it wouldn't have been clear, during deserialization, what is expected by the deserializing code. readResolve() needs to execute LoggerFactory.getLogger(loggerName). How should we decide about wrapping or not without creating kludgy code? Wrapping it each and every time if n.Logger wasn't implemented by the returned instance? Nope, this would have been quite bad.
If you notice there really is no extra overhead since in my fork Logback's getLogger() always returns a MessageLogger. It isn't really wrapped. No kludgy code is needed. Since MessageLogger extends Logger anyone calling LoggerFactory.getLogger() will work just fine. The wrapper is only needed if a Logger implementation returns a Logger. Then MessageLoggerFactory will wrap it.
What MessageLoggerFactory? Now I'm lost. ;) Joern.
On Jan 29, 2010, at 3:00 AM, Joern Huxhorn wrote:
One problem of your implementation that I've found in the meantime is that you are not using the Throwable that might have been detected by ParameterizedMessage.
That's a bug. The constructor for LoggingEvent takes a Message and a throwable. If the throwable is present (meaning it was specified on the call) then it should be used. If it is null then the throwable in the Message should be used. The same thing should be done when filtering, but there the Message, if present on the api call used, needs to be decomposed into the raw message format and parameters so that calls that don't have a Message don't create one. Fixing this problem is pretty easy.
I won't believe that until I see a fork of Logback that depends on your SLF4J implementation. It was very difficult to get Logback to accept the Message and deal with it correctly.
Well, you did the hardest part of the work already, i.e. the Message handling throughout the rest of the API.
I'd honestly like to have some input from Ceki because I don't want to invest too much time if all of this ends in a "nope".
I agree.
If you notice there really is no extra overhead since in my fork Logback's getLogger() always returns a MessageLogger. It isn't really wrapped. No kludgy code is needed. Since MessageLogger extends Logger anyone calling LoggerFactory.getLogger() will work just fine. The wrapper is only needed if a Logger implementation returns a Logger. Then MessageLoggerFactory will wrap it.
What MessageLoggerFactory? Now I'm lost. ;)
In my fork. Look at the org.slf4j.message package. With Logback the only difference between MessageLoggerFactory and LoggerFactory is that the return type is MessageLogger. A user calling LoggerFactory would still get a MessageLogger but the return type is Logger. Ralph
On Dec 4, 2009, at 2:12 PM, Ceki Gülcü wrote:
Hi Ralph,
I've looked at the changed you made in relation to RFC 5424. It's quite a significant architectural change. The RFC 5424 route is in the opposite direction of logback where one creates a new module for each new event type.
I'm confused. What is this "module" that one creates? I assume you mean an event type is a Marker? If not, what embodies it?
This offers the benefit of type safety but has significant overhead from a packaging pov, whereas the 5424 route has little packaging overhead but omits type safety.
Type safety of what? Everything is a String. With the latest changes Joern suggested that is enforced.
The RFC 5424 approach is not to be discarded. However, I don't think it's a route we should take at this *immediate* juncture. I propose to put it aside for the time being and come back to it, possibly with a vengeance, some time later.
Obviously I have no control over what gets adopted into SLF4J/Logback but I find this rather disappointing as it means I will either have to hack something messy on top of Logback or finally get off my butt and work on Log4j 2.0 which will require far more time than I want to spend. Structured data support (RFC 5424) is an absolute requirement for my project. I haven't looked at Joern's version yet but from what he has described it sounds promising. I hope I can find some time tonight. Ralph
Ralph Goers wrote:
On Dec 4, 2009, at 2:12 PM, Ceki Gülcü wrote:
Hi Ralph,
I've looked at the changed you made in relation to RFC 5424. It's quite a significant architectural change. The RFC 5424 route is in the opposite direction of logback where one creates a new module for each new event type.
I'm confused. What is this "module" that one creates? I assume you mean an event type is a Marker? If not, what embodies it?
I was referring to modules like logback-classic for ILoggingEvent, logback-access for AccessEvent, AuditEvent for logback-audit.
This offers the benefit of type safety but has significant overhead from a packaging pov, whereas the 5424 route has little packaging overhead but omits type safety.
Type safety of what? Everything is a String. With the latest changes Joern suggested that is enforced.
I have not seen Joern's changes. By using modules you can access each field directly with getters instead of get(eventFieldId). Many you write components this makes a palpabable difference.
The RFC 5424 approach is not to be discarded. However, I don't think it's a route we should take at this *immediate* juncture. I propose to put it aside for the time being and come back to it, possibly with a vengeance, some time later.
Obviously I have no control over what gets adopted into SLF4J/Logback but I find this rather disappointing as it means I will either have to hack something messy on top of Logback or finally get off my butt and work on Log4j 2.0 which will require far more time than I want to spend. Structured data support (RFC 5424) is an absolute requirement for my project.
If structured data support is that important, them I encourage you continue to develop your fork. SLF4J is changing very slowly so keeping in sync should not be too difficult. Logback is a different matter, but if your modifications are localized, then rebasing with the logback trunk should be possible as well. Just an idea...
Ralph
On Dec 2, 2009, at 11:37 PM, Ceki Gulcu wrote:
Ralph Goers wrote:
On Dec 4, 2009, at 2:12 PM, Ceki Gülcü wrote:
Hi Ralph,
I've looked at the changed you made in relation to RFC 5424. It's quite a significant architectural change. The RFC 5424 route is in the opposite direction of logback where one creates a new module for each new event type.
I'm confused. What is this "module" that one creates? I assume you mean an event type is a Marker? If not, what embodies it?
I was referring to modules like logback-classic for ILoggingEvent, logback-access for AccessEvent, AuditEvent for logback-audit.
Oh. I never understood the point of logback-access or logback-audit. It just seems like a nice way to make things very complicated. I've glanced at both but never figured out how they could be useful to me. I don't understand what benefit AuditEvent has over LoggingEvent if LoggingEvent can contain some sort of structured data object (as in EventData). I understand why it is great to provide support for Jetty and Tomcat but it has always seemed odd to me that the support isn't done as a layer on top of logback-classic instead of logback-core. But then, I've never had a good reason to really look very hard at the code in logback-access.
This offers the benefit of type safety but has significant overhead from a packaging pov, whereas the 5424 route has little packaging overhead but omits type safety.
Type safety of what? Everything is a String. With the latest changes Joern suggested that is enforced.
I have not seen Joern's changes. By using modules you can access each field directly with getters instead of get(eventFieldId). Many you write components this makes a palpabable difference.
Again, having to write a whole new module per event type just isn't practical. But then, I don't think you really mean "event type". logback-access and logback-audit certainly can handle more than one kind of "event".
The RFC 5424 approach is not to be discarded. However, I don't think it's a route we should take at this *immediate* juncture. I propose to put it aside for the time being and come back to it, possibly with a vengeance, some time later.
Obviously I have no control over what gets adopted into SLF4J/Logback but I find this rather disappointing as it means I will either have to hack something messy on top of Logback or finally get off my butt and work on Log4j 2.0 which will require far more time than I want to spend. Structured data support (RFC 5424) is an absolute requirement for my project.
If structured data support is that important, them I encourage you continue to develop your fork. SLF4J is changing very slowly so keeping in sync should not be too difficult. Logback is a different matter, but if your modifications are localized, then rebasing with the logback trunk should be possible as well.
The main issue I faced was there is no good way to pass a data object through SLF4J to the underlying logging implementation. Really, the most significant change I made was to add XLocationAwareLogger to SLF4J to accomplish that, and even that is fairly transparent. I also still haven't looked at Joern's code, but if he is generalizing that to a more generic Message interface then that could be a very good thing. Within Logback the main issue was determining if the first parameter is a StructuredData object. At first I tried adding it to the LoggingEvent but soon realized that would be far more extensive a change. Again, I'm interested to see what Joern has done. One of the main reasons I am enamored with RFC 5424 is that it is a fabulous way of sending serialized event data around without being tied to any particular logging implementation, with the exception of the names of the keys. I don't think sending a throwable with its call stack would be a good idea, but virtually everything else could be captured in structured data such as the mdc (which I am already doing) and even the caller data. I don't really think of it as a syslog message but as an industry standard way of sending event messages through a system. Using a message bus one can easily inspect the data as it passes through the bus. Plus, I just noticed yesterday that there are two new RFCs, 5674 - Alarms in Syslog and 5675 - Mapping SNMP Notifications to Syslog Messages, that are building on top of the 5424 spec and making it even more useful. As far as developing on my fork, there is no way I can have my development teams use SLF4J and Logback from my private branches. Unfortunately, at this point all I can do is build a kludge around EventData - which is more expensive because it has to be serialized and deserialized in the Appender - and format that as structured data in my own IETFSyslogLayout class. Plus, the fact that the JMSAppender won't use a layout also means I have to write my own version of that or the IETFSyslogAppender if I use that directly instead of JMS. All of this code would then have to reside in my own source control system at Intuit which is exactly what I am hoping to avoid. Ralph
Ralph Goers wrote:
Oh. I never understood the point of logback-access or logback-audit. It just seems like a nice way to make things very complicated. I've glanced at both but never figured out how they could be useful to me. I don't understand what benefit AuditEvent has over LoggingEvent if LoggingEvent can contain some sort of structured data object (as in EventData). I understand why it is great to provide support for Jetty and Tomcat but it has always seemed odd to me that the support isn't done as a layer on top of logback-classic instead of logback-core. But then, I've never had a good reason to really look very hard at the code in logback-access.
When you can rely on the type of the event you are dealing with, you can write specialized logic for that type much more easily. This is pretty basic OO stuff as you can imagine.
Again, having to write a whole new module per event type just isn't practical. But then, I don't think you really mean "event type". logback-access and logback-audit certainly can handle more than one kind of "event".
Well, it's kind of a pain in the neck but once you put a structure in place, such a module, writing code becomes much easier than dealing with generic types. But I agree with or at least am sympathetic to the general point.
The main issue I faced was there is no good way to pass a data object through SLF4J to the underlying logging implementation. Really, the most significant change I made was to add XLocationAwareLogger to SLF4J to accomplish that, and even that is fairly transparent. I also still haven't looked at Joern's code, but if he is generalizing that to a more generic Message interface then that could be a very good thing. Within Logback the main issue was determining if the first parameter is a StructuredData object. At first I tried adding it to the LoggingEvent but soon realized that would be far more extensive a change. Again, I'm interested to see what Joern has done.
One of the main reasons I am enamored with RFC 5424 is that it is a fabulous way of sending serialized event data around without being tied to any particular logging implementation, with the exception of the names of the keys. I don't think sending a throwable with its call stack would be a good idea, but virtually everything else could be captured in structured data such as the mdc (which I am already doing) and even the caller data. I don't really think of it as a syslog message but as an industry standard way of sending event messages through a system. Using a message bus one can easily inspect the data as it passes through the bus. Plus, I just noticed yesterday that there are two new RFCs, 5674 - Alarms in Syslog and 5675 - Mapping SNMP Notifications to Syslog Messages, that are building on top of the 5424 spec and making it even more useful. >
RFC 5424 is a big mental leap in the sense that one needs to start seeing logging events as a map with named fields, with the entailing changes at the API level. At the same time, and very contradictorily, it can be viewed as just a new encoding technique, i.e. business as usual. Could you, without going into the business details, give an example of a use case. In particular, I am curious with the kind of post-processing you do with the logs. Do you do any post processing?
On Dec 3, 2009, at 1:05 AM, Ceki Gülcü wrote:
RFC 5424 is a big mental leap in the sense that one needs to start seeing logging events as a map with named fields, with the entailing changes at the API level. At the same time, and very contradictorily, it can be viewed as just a new encoding technique, i.e. business as usual.
Could you, without going into the business details, give an example of a use case. In particular, I am curious with the kind of post-processing you do with the logs. Do you do any post processing?
Sure. We deal with internet banking for 1800 financial institutions. The applications are clustered with a user for any institution being serviced by any of the nodes in the cluser. Here is a flow that isn't completely accurate but is representative of what would happen. 1. The user hits the web site and gets the sign on screen. Typically a session will be created and information such as the user's ip address and the bank's id will be stored. On every subsequent interaction the MDC will be populated with that data along with a unique request id. 2. The user enters their log in data. A "login attempt" audit event will be generated and the user's id will be part of the event data (the MDC data is always included in every event). 3. The login suceeds. The user's id will be added to the MDC. A "login succeeded" audit event will be generated (no extra event data is necessary). 4. The user goes to their account summary page. A "displayAccountSummary" audit event will be generated and the account number will be included in the event data. 5. The request to obtain the summary data will be routed to an internal service (which could be running in a separate data center). The MDC data is passed to the service where it populates an MDC with that data. A "getSummaryData" event will be generated and the relevant data included in the event data (this is not an "audit" event). 6. The request succeeds and the service generates a "getSummaryData succeeded" event and returns the data. 7. The application generates a "displayAccountSummary succeeded" audit event. This will go on for every activity the end user performs and each back end service that participates. The only difference between an "Audit" event and a non-Audit event is that Financial Institution administrators can view and generate reports on audit events while our internal administrators can view audit and non-audit events co-mingled. The basic design calls for RFC 5424 messages to be routed through a message bus to a "logging service" - I am still working with other Architects within Intuit to determine what that will be. However, for my purposes I will have portlets running in our Administrative Portal that will allow FI administrators to view end user activity, generate daily or weekly reports (or raw data for download) that meet certain filtering criteria (for example, showing all transactions where money was transferred from one account to another over a certain dollar amount). In addition, we might use the data to add promotional messages to the end user's screen. So having the data in a self describing format is of great benefit. I've already seen that Splunk handles the structured data pretty well without me having to do anything. Syslog-ng can also filter on the structured data even to the point of identifying it as SD-ID.key. If, on the other hand, the data was to be stored in a database it could easily be done in the same manner that the DBAppender stores the MDC data. In short, post-processing is one of the most important aspects of the solution. Ralph
participants (6)
-
Ceki Gulcu -
Ceki Gülcü
-
ceki＠qos.ch
-
Joern Huxhorn -
Ralph Goers -
ralph.goers ＠dslextreme.com