Hi! Logback has two levels of formatting: One for the complete log entry and one for the log message itself. While it's flexible with the first, it's restrictive with the second, as the MessageFormatter class is right now a static-methods-only singleton. By making it an interface with a default implementation, one could reconfigure a Layout to use a different Formatter with other features. We'd like to have a message formatter that: 1. Appends all arguments that don't have a formatting anchor ("{}"). 2. Configurably allow special formatting of arguments of some class. The workaround we are living with now is to use mainly a copy of the current MessageFormatter class. Before I file an JIRA entry and maybe even provide a patch, I'd like to know what others think. Regards Rü
On 18.01.2010, at 13:39, Ruediger zu Dohna wrote:
Hi!
Logback has two levels of formatting: One for the complete log entry and one for the log message itself. While it's flexible with the first, it's restrictive with the second, as the MessageFormatter class is right now a static-methods-only singleton. By making it an interface with a default implementation, one could reconfigure a Layout to use a different Formatter with other features.
We'd like to have a message formatter that: 1. Appends all arguments that don't have a formatting anchor ("{}"). 2. Configurably allow special formatting of arguments of some class.
The workaround we are living with now is to use mainly a copy of the current MessageFormatter class.
Before I file an JIRA entry and maybe even provide a patch, I'd like to know what others think.
My proposal for a refined slf4j API over at http://bugzilla.slf4j.org/show_bug.cgi?id=31#c69 has the ability to use custom-made Message implementations. Changing the default message format (or making it configurable) of Logback, on the other hand, isn't a good idea since the contract of it is derived from SLF4J. Any SLF4J implementation must be able to support the new formatting. I'd be happy if you could take a few minutes and check out my proposal. Thanks, Joern.
Regards Rü _______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
Hi Joern, Joern Huxhorn-2 wrote:
I'd be happy if you could take a few minutes and check out my proposal.
You where kidding, right?!? "a few minutes"... well, I still did take a look. Nice proposal and it's good to know that there might be some things like that coming. Passing your own Message instances into the API would be powerful and definetely a gain. But I think that's not only an API change, it's a completely different design. I'd say: That's a too big a step for SLF4J to reasonably take. Why not put it into a separate structured logging artifact that wrapps SLF4J. The impact on performance the wrapping will have is probably in the same range a that you'd introduce by replacing e.g. debug(msg, arg0) with debug(msg, new Object[] { arg0 }) by only using varargs... hotspot is really smart, you know? For the varargs discussion, I like Cekis suggestion http://bugzilla.slf4j.org/show_bug.cgi?id=31#c31 31-31 best.
Changing the default message format (or making it configurable) of Logback, on the other hand, isn't a good idea since the contract of it is derived from SLF4J. Any SLF4J implementation must be able to support the new formatting.
Your warning is legitimate! But I don't want to change the message format, I want to change the formatter! I don't want to e.g. put extra stuff within the curly bracket placeholders, generally preventing other SLF4J implementations from working properly. But I do want to change what goes in there. I.e. when I want to log a message with an argument that's an instance of a class that doesn't override toString to my liking (most notably: not at all), I want to provide my own formatter for all instances of that class. Anything else would be my own risk... if I'd ever choose to go away from logback, that is ;) Put simply, your proposal requires the caller code to change (as well as SLF4J, Logback, and many others) while my change would work with logging calls from third party libraries as well ;-) Regards Rü -- View this message in context: http://old.nabble.com/MessageFormatter-tp27209919p27241839.html Sent from the Logback Dev mailing list archive at Nabble.com.
On 20.01.2010, at 14:36, Rü wrote:
Hi Joern,
Joern Huxhorn-2 wrote:
I'd be happy if you could take a few minutes and check out my proposal.
You where kidding, right?!? "a few minutes"... well, I still did take a look. Nice proposal and it's good to know that there might be some things like that coming. Passing your own Message instances into the API would be powerful and definetely a gain. But I think that's not only an API change, it's a completely different design. I'd say: That's a too big a step for SLF4J to reasonably take. Why not put it into a separate structured logging artifact that wrapps SLF4J.
The original SLF4J API would stay untouched for compatibility reasons. Sorry about the "few minutes" ;) It's possible to provide the new API by wrapping the old one (I've prepared exactly that already) but implementing it "natively" would enable Appenders (in case of Logback) to work on Messages instead of Strings, opening up lots of new opportunities.
The impact on performance the wrapping will have is probably in the same range a that you'd introduce by replacing e.g. debug(msg, arg0) with debug(msg, new Object[] { arg0 }) by only using varargs... hotspot is really smart, you know?
For the varargs discussion, I like Cekis suggestion http://bugzilla.slf4j.org/show_bug.cgi?id=31#c31 31-31 best.
That comment refers to a previous proposal that merely changed the Object[] into Object... I've instead added a new, additional API that you can either use instead of the old one or ignore. This has the advantage that projects requiring JDK1.4 could still be supported.
Changing the default message format (or making it configurable) of Logback, on the other hand, isn't a good idea since the contract of it is derived from SLF4J. Any SLF4J implementation must be able to support the new formatting.
Your warning is legitimate! But I don't want to change the message format, I want to change the formatter! I don't want to e.g. put extra stuff within the curly bracket placeholders, generally preventing other SLF4J implementations from working properly. But I do want to change what goes in there. I.e. when I want to log a message with an argument that's an instance of a class that doesn't override toString to my liking (most notably: not at all), I want to provide my own formatter for all instances of that class. Anything else would be my own risk... if I'd ever choose to go away from logback, that is ;)
I think the main problem is that adding such an ability will have an impact on the standard formatter performance. Additional lookup is needed to check if a special formatter is registered - for every parameter. I don't say that this will be terribly expensive but a certain cost will be created.
Put simply, your proposal requires the caller code to change (as well as SLF4J, Logback, and many others) while my change would work with logging calls from third party libraries as well ;-)
That's not really the case. You can either leave everything as it is right now, since SLF4J API would stay binary-compatible, or switch to the new API by simply changing the import of Logger and LoggerFactory. This can easily be done by a quick search & destroy/replace. The same is the case for third-party libraries/implementations of SLF4J other than Logback. They will still work as before, even if the new API is used, since the original SLF4J would simply be wrapped. In case of a more modern implementation that natively implements the new SLF4J API, the old API would be implemented by wrapping the new API. Thoroughly confused? ;) Regards, Jörn.
Regards Rü -- View this message in context: http://old.nabble.com/MessageFormatter-tp27209919p27241839.html Sent from the Logback Dev mailing list archive at Nabble.com.
_______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
Hi Joern, Joern Huxhorn-2 wrote:
It's possible to provide the new API by wrapping the old one (I've prepared exactly that already) but implementing it "natively" would enable Appenders (in case of Logback) to work on Messages instead of Strings, opening up lots of new opportunities.
I was very well aware of your point. I just would prefer to leave SLF4J more or less as it is (only tiny steps like binary compatible varargs, like Ceki had suggested) and put your API into a separate package. I think it's a too different use case and a too fast pace for the core logging API that SLF4J is becomming. Your new API could be implemented natively e.g. by Logback or as a wrapper for other logging systems. As I've said: I can see the great benefit of such an API. We actually use the Logging-APIs to inform monitoring tools e.g. about call latency: Something like monitoring.info("{} took {}", system, time); could end up in a logfile and/or as a http request for our monitoring framework. We can't use Log4j here, as it formats messages before it passes LoggingEvents to Appenders and it would be ridiculous to parse the numbers out of the message string again. So we switched to Logback ;-) And passing such information as structured, eventually typesafe data sounds even better!
Rü wrote:
Your warning is legitimate! But I don't want to change the message format, I want to change the formatter!
I think the main problem is that adding such an ability will have an impact on the standard formatter performance.
The standard formatter should stay exactly as it is... it should just not be one more of those terrible singletons (the modern form of global variables). Then I could configure Logback to use one that I write whenever I think I can live with the extra performance cost.
Put simply, your proposal requires the caller code to change (as well as SLF4J, Logback, and many others) while my change would work with logging calls from third party libraries as well ;-)
That's not really the case. ... Thoroughly confused? ;)
I was talking about third party libraries like Hibernate, that log for themselves. When they log objects that've been passed to them, and those objects don't know how to String (pun)...... a special MessageFormatter could make it readable; but even if Hibernate would pass it's own Message object to your API, the objects within would not be formatted to my liking. Regards Rü -- View this message in context: http://old.nabble.com/MessageFormatter-tp27209919p27244635.html Sent from the Logback Dev mailing list archive at Nabble.com.
FWIW, I've implemented Joern's proposal at http://github.com/rgoers/slf4j and http://github.com/rgoers/logback. The impact on SLF4J is minimal. In SLF4J I added a message package where the support is encapsulated. Since Logback's client API is SLF4J the new methods have to be implemented there. The overhead of doing this is negligable. In reality this is the only good way to do what you are asking for. As Ceki has pointed out, changing the message formatting for a "standard" message creates endless problems since many components all use the the SLF4J api and expect the current formatting. Joern's proposal, where the formatting is dictated by the message type provides the required flexibility in a much more deterministic manner. We are using SLF4J/Logback for Audit/Event/Activity logging. Structured data is a perfect fit for that use case. Not having that available in SLF4J makes audit logging difficult. The EventData I added to slf4j-ext was a first stab at this (before I knew RFC 5424 existed), but it is very costly since SLF4J doesn't support passing an Object through LocationAwareLogger. In addition, logback-classic isn't dependent on slf4j-ext so adding support for EventData would introduce an undesirable new dependency. Finally, RFC 5424 is the new syslog spec. I have recently completed a review of all the commercial log management products, such as ArcSight, Splunk, SenSage, LogLogic, etc and they all support this format, as do syslog-ng and rsyslog. So I view SLF4J/Logback not supporting this format as a serious deficiency. Ralph On Jan 20, 2010, at 8:33 AM, Rü wrote:
Hi Joern,
Joern Huxhorn-2 wrote:
It's possible to provide the new API by wrapping the old one (I've prepared exactly that already) but implementing it "natively" would enable Appenders (in case of Logback) to work on Messages instead of Strings, opening up lots of new opportunities.
I was very well aware of your point. I just would prefer to leave SLF4J more or less as it is (only tiny steps like binary compatible varargs, like Ceki had suggested) and put your API into a separate package. I think it's a too different use case and a too fast pace for the core logging API that SLF4J is becomming. Your new API could be implemented natively e.g. by Logback or as a wrapper for other logging systems.
As I've said: I can see the great benefit of such an API. We actually use the Logging-APIs to inform monitoring tools e.g. about call latency: Something like monitoring.info("{} took {}", system, time); could end up in a logfile and/or as a http request for our monitoring framework. We can't use Log4j here, as it formats messages before it passes LoggingEvents to Appenders and it would be ridiculous to parse the numbers out of the message string again. So we switched to Logback ;-) And passing such information as structured, eventually typesafe data sounds even better!
Rü wrote:
Your warning is legitimate! But I don't want to change the message format, I want to change the formatter!
I think the main problem is that adding such an ability will have an impact on the standard formatter performance.
The standard formatter should stay exactly as it is... it should just not be one more of those terrible singletons (the modern form of global variables). Then I could configure Logback to use one that I write whenever I think I can live with the extra performance cost.
Put simply, your proposal requires the caller code to change (as well as SLF4J, Logback, and many others) while my change would work with logging calls from third party libraries as well ;-)
That's not really the case. ... Thoroughly confused? ;)
I was talking about third party libraries like Hibernate, that log for themselves. When they log objects that've been passed to them, and those objects don't know how to String (pun)...... a special MessageFormatter could make it readable; but even if Hibernate would pass it's own Message object to your API, the objects within would not be formatted to my liking.
Regards Rü
-- View this message in context: http://old.nabble.com/MessageFormatter-tp27209919p27244635.html Sent from the Logback Dev mailing list archive at Nabble.com.
_______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
Hi Ralph, I thought that you'd be the second one to reply ;-))) rgoers wrote:
In reality this is the only good way to do what you are asking for. As Ceki has pointed out, changing the message formatting for a "standard" message creates endless problems since many components all use the the SLF4J api and expect the current formatting. Joern's proposal, where the formatting is dictated by the message type provides the required flexibility in a much more deterministic manner.
I don't want to change the format strings in my log statements!!! I don't want to change what's going in, just what's going out!!! Nobody relies on that. I am free to write any Layout I like. For example in version 0.9.18 the EchoLayout just uses toString and not event.getFormattedMessage(). LoggingEvent.toString luckily calls event.getFormattedMessage() itself, but LoggingEventVO doesn't even override toString(). You could argue that this is a bug, but: What contract is broken?!? I could even write my own SLF4J implementation that completely ignores the formatting anchors and just appends the arguments with a fixed delimiter... no contract broken! Only when I start to omit the formatting anchors, then I'm breaking the SLF4J contract... I can't switch to another implementation any more. Same thing, when I start to add some kind of formatting instructions into the formatting anchor: Broke the SLF4J contract and can't switch any more.
We are using SLF4J/Logback for Audit/Event/Activity logging. Structured data is a perfect fit for that use case. Not having that available in SLF4J makes audit logging difficult.
We have this use case, too, but it is a different use case! I need to simply format objects that anyone (maybe even Hibernate) tries to log.
Finally, RFC 5424 is the new syslog spec. I have recently completed a review of all the commercial log management products, such as ArcSight, Splunk, SenSage, LogLogic, etc and they all support this format, as do syslog-ng and rsyslog. So I view SLF4J/Logback not supporting this format as a serious deficiency.
Is this review available anywhere... we have similar discussions with our system admins. Would be helpful to have more fuel ;-))) And structured messages would be great to store into the database, too! But it's again two use cases: Either you want to put structured data into your logging api, or you want to put structured data out onto the channel. The full power would only unfold, if you have both, of course! But the other combinations make sense in their own right. Regards Rü -- View this message in context: http://old.nabble.com/MessageFormatter-tp27209919p27257209.html Sent from the Logback Dev mailing list archive at Nabble.com.
Hello, As far as the requirements you have described, assuming you are using PatternLayout to format the output, you can easily create your own conversion specifier (see [1]) to format the message according to your wishes. You are not forced to use %m / %msg / %message in your conversion pattern. For example, assuming you defined %myMsg the conversion word, you could have "%d %level - %myMsg%n" as your conversion pattern, which would 1) appendall arguments lacking an associated {}, 2) allow special formatting of arguments of certain types. Would the above work for you? [1] http://logback.qos.ch/manual/layouts.html#customConversionSpecifier On 18/01/2010 1:39 PM, Ruediger zu Dohna wrote:
Hi!
Logback has two levels of formatting: One for the complete log entry and one for the log message itself. While it's flexible with the first, it's restrictive with the second, as the MessageFormatter class is right now a static-methods-only singleton. By making it an interface with a default implementation, one could reconfigure a Layout to use a different Formatter with other features.
We'd like to have a message formatter that: 1. Appends all arguments that don't have a formatting anchor ("{}"). 2. Configurably allow special formatting of arguments of some class.
The workaround we are living with now is to use mainly a copy of the current MessageFormatter class.
Before I file an JIRA entry and maybe even provide a patch, I'd like to know what others think.
Regards Rü
Hi Ceki, Ceki Gulcu wrote:
You are not forced to use %m / %msg / %message in your conversion pattern.
I even replace the MessageConverter with my own ExtendedMessageConverter. But that still means, that I'll have to do all the parsing of the formatting anchors and things for myself! I'd be branching your code, and maybe that could be done for the benefit of all Logback users. Regards Rü -- View this message in context: http://old.nabble.com/MessageFormatter-tp27209919p27257288.html Sent from the Logback Dev mailing list archive at Nabble.com.
On 21/01/2010 1:59 PM, Rü wrote:
Ceki Gulcu wrote:
You are not forced to use %m / %msg / %message in your conversion pattern.
I even replace the MessageConverter with my own ExtendedMessageConverter. But that still means, that I'll have to do all the parsing of the formatting anchors and things for myself! I'd be branching your code, and maybe that could be done for the benefit of all Logback users.
Reading your previous posts I was under the impression that the formatting code was specific to your environment and not general purpose. If it is general purpose, you can contribute it here or distribute it separately. As far as I can tell, the conversion specifier extension mechanism as exists currently in PatternLayout meets your needs. I am missing something here? -- Ceki
participants (5)
-
Ceki Gülcü -
Joern Huxhorn -
Ralph Goers -
Ruediger zu Dohna -
Rü