What would cause "java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy" in long running daemon
Hi, We have multiple production processes that run 24 hours a day, 7 days a week. All use slf4j/logback and recently we had several of these (seemingly coincidentally) spew: java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy These are long running processes where nothing has changed recently. Obviously the logback jars are available, in the right place, etc. I'm stumped. There are some hints of some possible system related problem (like missing NFS mount, possibly). Does logback dynamically load the above class? Again, this error is out of the blue, from a process that may have been running for days or weeks. Any ideas? We're using version logback 1.0.13. If I google the above, there are some references to "Spring Boot" which we are not using. However, we are using Spring Integration, in case that matters. Any ideas? Thanks, Fred
Are you storing your jar files on the NFS mounts? Seem like a bad idea if you are. If a network error causes an exception, and the exception handler calls a class that has not been loaded yet, the class loader will try to load it. The network is unreliable, you must be able to handle network failures. If you are using NFS for ease of deployment, say for batch jobs, there is another way. David
On 30 Aug 2015, at 01:58, Fred Toth <ftoth@synernet.com> wrote:
Hi,
We have multiple production processes that run 24 hours a day, 7 days a week. All use slf4j/logback and recently we had several of these (seemingly coincidentally) spew:
java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy
These are long running processes where nothing has changed recently. Obviously the logback jars are available, in the right place, etc.
I'm stumped. There are some hints of some possible system related problem (like missing NFS mount, possibly).
Does logback dynamically load the above class? Again, this error is out of the blue, from a process that may have been running for days or weeks.
Any ideas?
We're using version logback 1.0.13.
If I google the above, there are some references to "Spring Boot" which we are not using. However, we are using Spring Integration, in case that matters.
Any ideas?
Thanks, Fred
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
Hi David, For better or worse, these particular apps are on one of these modern disk systems where EVERYTHING is an NFS mount. Supposed to "just work", you know. Thanks, Fred On 8/31/15 6:55 AM, David Roussel wrote:
Are you storing your jar files on the NFS mounts?
Seem like a bad idea if you are. If a network error causes an exception, and the exception handler calls a class that has not been loaded yet, the class loader will try to load it.
The network is unreliable, you must be able to handle network failures.
If you are using NFS for ease of deployment, say for batch jobs, there is another way.
David
On 30 Aug 2015, at 01:58, Fred Toth <ftoth@synernet.com> wrote:
Hi,
We have multiple production processes that run 24 hours a day, 7 days a week. All use slf4j/logback and recently we had several of these (seemingly coincidentally) spew:
java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy
These are long running processes where nothing has changed recently. Obviously the logback jars are available, in the right place, etc.
I'm stumped. There are some hints of some possible system related problem (like missing NFS mount, possibly).
Does logback dynamically load the above class? Again, this error is out of the blue, from a process that may have been running for days or weeks.
Any ideas?
We're using version logback 1.0.13.
If I google the above, there are some references to "Spring Boot" which we are not using. However, we are using Spring Integration, in case that matters.
Any ideas?
Thanks, Fred
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
Hopefully /tmp is local disk? Write a bootstrapped that reads the jars off the network, writes them to /tmp, construct a class path then create a classloader and boot your main class. Or you could use the capsule open source project which pretty much does that for you. David
On 31 Aug 2015, at 16:03, Fred Toth <ftoth@synernet.com> wrote:
Hi David,
For better or worse, these particular apps are on one of these modern disk systems where EVERYTHING is an NFS mount. Supposed to "just work", you know.
Thanks, Fred
On 8/31/15 6:55 AM, David Roussel wrote: Are you storing your jar files on the NFS mounts?
Seem like a bad idea if you are. If a network error causes an exception, and the exception handler calls a class that has not been loaded yet, the class loader will try to load it.
The network is unreliable, you must be able to handle network failures.
If you are using NFS for ease of deployment, say for batch jobs, there is another way.
David
On 30 Aug 2015, at 01:58, Fred Toth <ftoth@synernet.com> wrote:
Hi,
We have multiple production processes that run 24 hours a day, 7 days a week. All use slf4j/logback and recently we had several of these (seemingly coincidentally) spew:
java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy
These are long running processes where nothing has changed recently. Obviously the logback jars are available, in the right place, etc.
I'm stumped. There are some hints of some possible system related problem (like missing NFS mount, possibly).
Does logback dynamically load the above class? Again, this error is out of the blue, from a process that may have been running for days or weeks.
Any ideas?
We're using version logback 1.0.13.
If I google the above, there are some references to "Spring Boot" which we are not using. However, we are using Spring Integration, in case that matters.
Any ideas?
Thanks, Fred
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
Seems like overkill. All you need to do is preload the classes you need to log your failures while NFS is still working normally, right? For example, you could log a test exception as part of your initialization sequence. Once the classes have been loaded into the JVM they won't be evicted. Thanks, Paul
On Aug 31, 2015, at 11:16, David Roussel <nabble@diroussel.xsmail.com> wrote:
Hopefully /tmp is local disk?
Write a bootstrapped that reads the jars off the network, writes them to /tmp, construct a class path then create a classloader and boot your main class.
Or you could use the capsule open source project which pretty much does that for you.
David
On 31 Aug 2015, at 16:03, Fred Toth <ftoth@synernet.com> wrote:
Hi David,
For better or worse, these particular apps are on one of these modern disk systems where EVERYTHING is an NFS mount. Supposed to "just work", you know.
Thanks, Fred
On 8/31/15 6:55 AM, David Roussel wrote: Are you storing your jar files on the NFS mounts?
Seem like a bad idea if you are. If a network error causes an exception, and the exception handler calls a class that has not been loaded yet, the class loader will try to load it.
The network is unreliable, you must be able to handle network failures.
If you are using NFS for ease of deployment, say for batch jobs, there is another way.
David
On 30 Aug 2015, at 01:58, Fred Toth <ftoth@synernet.com> wrote:
Hi,
We have multiple production processes that run 24 hours a day, 7 days a week. All use slf4j/logback and recently we had several of these (seemingly coincidentally) spew:
java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy
These are long running processes where nothing has changed recently. Obviously the logback jars are available, in the right place, etc.
I'm stumped. There are some hints of some possible system related problem (like missing NFS mount, possibly).
Does logback dynamically load the above class? Again, this error is out of the blue, from a process that may have been running for days or weeks.
Any ideas?
We're using version logback 1.0.13.
If I google the above, there are some references to "Spring Boot" which we are not using. However, we are using Spring Integration, in case that matters.
Any ideas?
Thanks, Fred
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
On 31 Aug 2015, at 16:47, Paul Krause <pkrause@investsoftech.com> wrote:
Seems like overkill. All you need to do is preload the classes you need to log your failures while NFS is still working normally, right?
That would solve this specific issue. But what about the next failure scenario?
Point taken. I only answered the question that was (implicitly) asked, namely "how do I fix these class-loader errors so I can see the real problem?" David answered the question that should have been asked, "how do I make my long running process more stable than the NFS appliance where it is running?" -----Original Message----- From: Logback-user [mailto:logback-user-bounces@qos.ch] On Behalf Of David Roussel Sent: Monday, August 31, 2015 3:49 PM To: logback users list <logback-user@qos.ch> Subject: Re: [logback-user] What would cause "java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy" in long running daemon
On 31 Aug 2015, at 16:47, Paul Krause <pkrause@investsoftech.com> wrote:
Seems like overkill. All you need to do is preload the classes you need to log your failures while NFS is still working normally, right?
That would solve this specific issue. But what about the next failure scenario? _______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
If literally everything is an NFS drive, you could edit your app startup script to create a RAM disk, copy the jar files, and use those instead. http://www.jamescoyle.net/how-to/943-create-a-ram-disk-in-linux Scott ----- Original Message ----- From: "Fred Toth" <ftoth@synernet.com> To: "logback users list" <logback-user@qos.ch> Sent: Monday, August 31, 2015 11:03:31 AM Subject: Re: [logback-user] What would cause "java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy" in long running daemon Hi David, For better or worse, these particular apps are on one of these modern disk systems where EVERYTHING is an NFS mount. Supposed to "just work", you know. Thanks, Fred On 8/31/15 6:55 AM, David Roussel wrote:
Are you storing your jar files on the NFS mounts?
Seem like a bad idea if you are. If a network error causes an exception, and the exception handler calls a class that has not been loaded yet, the class loader will try to load it.
The network is unreliable, you must be able to handle network failures.
If you are using NFS for ease of deployment, say for batch jobs, there is another way.
David
On 30 Aug 2015, at 01:58, Fred Toth <ftoth@synernet.com> wrote:
Hi,
We have multiple production processes that run 24 hours a day, 7 days a week. All use slf4j/logback and recently we had several of these (seemingly coincidentally) spew:
java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy
These are long running processes where nothing has changed recently. Obviously the logback jars are available, in the right place, etc.
I'm stumped. There are some hints of some possible system related problem (like missing NFS mount, possibly).
Does logback dynamically load the above class? Again, this error is out of the blue, from a process that may have been running for days or weeks.
Any ideas?
We're using version logback 1.0.13.
If I google the above, there are some references to "Spring Boot" which we are not using. However, we are using Spring Integration, in case that matters.
Any ideas?
Thanks, Fred
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
Are you redeploying over old jars and not shutting down your existing process and/or relying on hot deploy? Are you absolutely sure your apps jars are not changing while the process is running? Check timestamp. Check MD5 sums. Why you see ThrowableProxy coming up as an error probably has nothing to do with logback. I bet there was a previous NoClassDef errors that should have be thrown and the output should of been shown but when that error was logged your application had not yet logged any exceptions yet and thus loaded the ThrowableProxy class which was also missing/corrupt. -Adam On Mon, Aug 31, 2015 at 12:23 PM, Scott Plante <splante@insightsys.com> wrote:
If literally everything is an NFS drive, you could edit your app startup script to create a RAM disk, copy the jar files, and use those instead.
http://www.jamescoyle.net/how-to/943-create-a-ram-disk-in-linux
Scott
----- Original Message -----
From: "Fred Toth" <ftoth@synernet.com> To: "logback users list" <logback-user@qos.ch> Sent: Monday, August 31, 2015 11:03:31 AM Subject: Re: [logback-user] What would cause "java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy" in long running daemon
Hi David,
For better or worse, these particular apps are on one of these modern disk systems where EVERYTHING is an NFS mount. Supposed to "just work", you know.
Thanks, Fred
On 8/31/15 6:55 AM, David Roussel wrote:
Are you storing your jar files on the NFS mounts?
Seem like a bad idea if you are. If a network error causes an exception, and the exception handler calls a class that has not been loaded yet, the class loader will try to load it.
The network is unreliable, you must be able to handle network failures.
If you are using NFS for ease of deployment, say for batch jobs, there is another way.
David
On 30 Aug 2015, at 01:58, Fred Toth <ftoth@synernet.com> wrote:
Hi,
We have multiple production processes that run 24 hours a day, 7 days a week. All use slf4j/logback and recently we had several of these (seemingly coincidentally) spew:
java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy
These are long running processes where nothing has changed recently. Obviously the logback jars are available, in the right place, etc.
I'm stumped. There are some hints of some possible system related problem (like missing NFS mount, possibly).
Does logback dynamically load the above class? Again, this error is out of the blue, from a process that may have been running for days or weeks.
Any ideas?
We're using version logback 1.0.13.
If I google the above, there are some references to "Spring Boot" which we are not using. However, we are using Spring Integration, in case that matters.
Any ideas?
Thanks, Fred
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user _______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
-- CTO SnapHop (snaphop.com) (twitter) @agentgt (linkedin) http://www.linkedin.com/in/agentgt (cell) 781-883-5182
participants (5)
-
Adam Gent -
David Roussel -
Fred Toth -
Paul Krause -
Scott Plante