separate task log files (slf4j + logback)
The MDC does not help when I have several different long-running tasks in one thread and I just want to immediately separate their logs. It seems that this is not such a fantastic wish. It is much easier to separate logical logging flows at once, than to filter them later in databases. The amount of data is large, logs per day can accumulate several hundred megabytes. It turns out that we will have to write our own implementation of such a log.
On Wed, Oct 18, 2017 at 9:41 AM, Денис Матвеев <denism78@gmail.com> wrote:
The MDC does not help when I have several different long-running tasks in one thread and I just want to immediately separate their logs.
If they different they should use different packages for logger name and you can separate based on base package. I can't imaging that regular Java app uses single thread for different task simultaneously. Unless you use some cooperative threading framework (Akka?) or some obscure setup. I see you lazy to make two calls MDC.put and MDC.remove. It can be that you are looking for dumping of app data instead of logging app events. In that case just open as many files for write as you wish. You don't need Logback for such task.
It seems that this is not such a fantastic wish. It is much easier to separate logical logging flows at once, than to filter them later in databases. The amount of data is large, logs per day can accumulate several hundred megabytes. It turns out that we will have to write our own implementation of such a log.
You can separate logs in filtering/routing software like I wrote: Flume, Fluentd, Logstash, Kafka, or whatever popular today. Direct events to intermediate agent that able to split and transform logging events. Of course write to local file is most reliable way to save data as 3rd party network service can be unavailable...
Hello Denis, The SiftingAppender uses a "discriminator" to sift events. This discriminator *can* be based on any part of the logging event and in particular the logger name. Here is an example of a possible LoggerBasedDiscriminator: ------------------------------- package org.example; public class LoggerBasedDiscriminator extends AbstractDiscriminator<ILoggingEvent> { private String key; @Override public void start() { if (OptionHelper.isEmpty(key)) { addError("The \"Key\" property must be set"); started = false; } else { started = true; } } public String getDiscriminatingValue(ILoggingEvent event) { return event.getLoggerName(); } public String getKey() { return key; } public void setKey(String key) { this.key = key; } } ------------------------------- Here is an example configuration for the sifting appender: <appender name="SIFT" class="ch.qos.logback.classic.sift.SiftingAppender"> <discriminator class="org.example.LoggerBasedDiscriminator "> <key>task</key> </discriminator> <sift> <appender name="FILE-${task}" class="ch.qos.logback.core.FileAppender"> <file>${task}.log</file> <append>false</append> <layout class="ch.qos.logback.classic.PatternLayout"> <pattern>%d [%thread] %level %logger{35} - %msg%n</pattern> </layout> </appender> </sift> </appender> As far as I can tell, the above is fits your initial requirements. However, I have not tested the above which might not even compile. -- Ceki On 18.10.2017 08:41, Денис Матвеев wrote:
The MDC does not help when I have several different long-running tasks in one thread and I just want to immediately separate their logs. It seems that this is not such a fantastic wish. It is much easier to separate logical logging flows at once, than to filter them later in databases. The amount of data is large, logs per day can accumulate several hundred megabytes. It turns out that we will have to write our own implementation of such a log.
participants (3)
-
Ceki Gulcu -
Oleksandr Gavenko -
Денис Матвеев