logback fails to roll log file - Shibboleth IDP 2.3.8 - Windows server 2008 - Jetty running as service via procrun
Hi all, My Shibboleth IDP version 2.3.8 is failing to roll over files daily. Its running on Windows Server 2008, and the configuration is the default Shibboleth one in logging.xml, except I have changed the slashes to be double back slashes as per the logback documentation in Appenders. I am running the IDP under Jetty 7.3, and have configured Jetty to run as a service using procrun, as the 'local system account' which I believe has all permissions necessary. It works on another IDP I look after (version 2.3.6, with Tomcat) on Windows Server 2003. I have read the posts referring to JNDI and rotation failing, but I can't see JNDI configured in my context file under Jetty, so guess this isn't affecting it ? I have looked through various logs in the Event Viewer but don't see anything to indicate a logback error on file rotation. My Shibboleth logging.xml contains: ----------------------------------------------------------------snip-- <appender name="IDP_PROCESS" class="ch.qos.logback.core.rolling.RollingFileAppender"> <File>C:\\opt\\shibboleth-idp\\logs\\idp-process.log</File> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <FileNamePattern>C:\\opt\\shibboleth-idp\\logs\\idp-process-%d{yyyy-MM-dd}.log</FileNamePattern> </rollingPolicy> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <charset>UTF-8</charset> <Pattern>%date{HH:mm:ss.SSS} - %level [%logger:%line] - %msg%n</Pattern> </encoder> </appender> --------------------------------------------------------------------- Does anyone have any ideas as to what might be causing roll over to fail ? Any pointers would be much appreciated, Kind Regards Karen -- *Karen Murphy* Systems Analyst - Bibliographic Services The Library at Queen's Queen's University Belfast Belfast BT7 1LP Tel: 028 90976260 Email: k.l.murphy@qub.ac.uk
My rollover problems have always been permissions problems. We usually find them by stopping the process that isn't rolling over its logs, switching user, and starting the process by hand as the user the process is supposed to run as. Virtually every time, standard out/standard error will show some IOException permissions thing. Jetty is doing its own logging of standard out/standard error, no? Is there a clue in that logfile? Bob On Tue, Mar 12, 2013 at 2:24 AM, Karen Murphy <k.l.murphy@qub.ac.uk> wrote:
Hi all,
My Shibboleth IDP version 2.3.8 is failing to roll over files daily. Its running on Windows Server 2008, and the configuration is the default Shibboleth one in logging.xml, except I have changed the slashes to be double back slashes as per the logback documentation in Appenders.
I am running the IDP under Jetty 7.3, and have configured Jetty to run as a service using procrun, as the 'local system account' which I believe has all permissions necessary.
It works on another IDP I look after (version 2.3.6, with Tomcat) on Windows Server 2003.
I have read the posts referring to JNDI and rotation failing, but I can't see JNDI configured in my context file under Jetty, so guess this isn't affecting it ?
I have looked through various logs in the Event Viewer but don't see anything to indicate a logback error on file rotation.
My Shibboleth logging.xml contains:
----------------------------------------------------------------snip-- <appender name="IDP_PROCESS" class="ch.qos.logback.core.rolling.RollingFileAppender"> <File>C:\\opt\\shibboleth-idp\\logs\\idp-process.log</File>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<FileNamePattern>C:\\opt\\shibboleth-idp\\logs\\idp-process-%d{yyyy-MM-dd}.log</FileNamePattern> </rollingPolicy>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <charset>UTF-8</charset> <Pattern>%date{HH:mm:ss.SSS} - %level [%logger:%line] - %msg%n</Pattern> </encoder> </appender> ---------------------------------------------------------------------
Does anyone have any ideas as to what might be causing roll over to fail ?
Any pointers would be much appreciated,
Kind Regards
Karen
-- *Karen Murphy* Systems Analyst - Bibliographic Services The Library at Queen's Queen's University Belfast Belfast BT7 1LP Tel: 028 90976260 Email: k.l.murphy@qub.ac.uk _______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
Hi Bob, Thanks for your reply and suggestions. Firstly I changed the FileNamePattern in the logback configuration to roll the log every minute (just until I crack this): <FileNamePattern>C:\\opt\\shibboleth-idp\\logs\\idp-process-%d{yyyy-MM-dd_HH-mm}.log</FileNamePattern> I then stopped the service and used Commons Deameon procrun command: prunsrv //TS/JettyService this runs the service "as a console application", and I'm logged in as "Administrator" account. Interestingly, the first thing it does is roll over the log file, but only once. After I make subsequent authentications (at least a minute apart) the new log lines are appended to idp-process.log, ie. it doesn't roll again after each minute. So I'm trying to understand whey this would happen. The Initial startup of the service as a console app (run as administrator) is able to roll the file, but any subsequent IdP operation is not able or allowed to roll it. Once the service is started as a console application (as Administrator) isn't all subsequent IdP processes run with Administrator permissions too ? I have also been playing around with the jetty logs (including creation of resources/jetty-logging.properties) , but have temporarily broken logging :-o Prior to breaking it the stderr and stdout files in JETTY_HOME/logs weren't showing any errors/exceptions. I increased the logging level in resource/log4j.properties to "log4j.rootLogger=DEBUG, stdout", the logging didn't increase. And now, I can't seem to get logging at all. I'm working on fixing it now, Thanks again for help and advice, Karen On 12/03/13 15:04, Robert Kuhar wrote:
My rollover problems have always been permissions problems. We usually find them by stopping the process that isn't rolling over its logs, switching user, and starting the process by hand as the user the process is supposed to run as. Virtually every time, standard out/standard error will show some IOException permissions thing. Jetty is doing its own logging of standard out/standard error, no? Is there a clue in that logfile?
Bob
On Tue, Mar 12, 2013 at 2:24 AM, Karen Murphy <k.l.murphy@qub.ac.uk> wrote:
Hi all,
My Shibboleth IDP version 2.3.8 is failing to roll over files daily. Its running on Windows Server 2008, and the configuration is the default Shibboleth one in logging.xml, except I have changed the slashes to be double back slashes as per the logback documentation in Appenders.
I am running the IDP under Jetty 7.3, and have configured Jetty to run as a service using procrun, as the 'local system account' which I believe has all permissions necessary.
It works on another IDP I look after (version 2.3.6, with Tomcat) on Windows Server 2003.
I have read the posts referring to JNDI and rotation failing, but I can't see JNDI configured in my context file under Jetty, so guess this isn't affecting it ?
I have looked through various logs in the Event Viewer but don't see anything to indicate a logback error on file rotation.
My Shibboleth logging.xml contains:
----------------------------------------------------------------snip-- <appender name="IDP_PROCESS" class="ch.qos.logback.core.rolling.RollingFileAppender"> <File>C:\\opt\\shibboleth-idp\\logs\\idp-process.log</File>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<FileNamePattern>C:\\opt\\shibboleth-idp\\logs\\idp-process-%d{yyyy-MM-dd}.log</FileNamePattern> </rollingPolicy>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <charset>UTF-8</charset> <Pattern>%date{HH:mm:ss.SSS} - %level [%logger:%line] - %msg%n</Pattern> </encoder> </appender> ---------------------------------------------------------------------
Does anyone have any ideas as to what might be causing roll over to fail ?
Any pointers would be much appreciated,
Kind Regards
Karen
-- *Karen Murphy* Systems Analyst - Bibliographic Services The Library at Queen's Queen's University Belfast Belfast BT7 1LP Tel: 028 90976260 Email: k.l.murphy@qub.ac.uk _______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
-- *Karen Murphy* Systems Analyst - Bibliographic Services The Library at Queen's Queen's University Belfast Belfast BT7 1LP Tel: 028 90976260 Email: k.l.murphy@qub.ac.uk
Hi Bob, further to my email I have realised that the logging problem occurred as a direct result of running the service as a console application. ie, it failed to write the default log4j files: jettyservice-stderr.2013-03-13 jettyservice-stdout.2013-03-13 even when I removed the existing files, I switched back to running as a service (under local system account) and these jetty logs are again produced. Though ofcourse the idp logs are back to never being rolled. Looks like permissions are to blame, Thanks Karen On 13/03/13 12:38, Karen Murphy wrote:
Hi Bob,
Thanks for your reply and suggestions.
Firstly I changed the FileNamePattern in the logback configuration to roll the log every minute (just until I crack this):
<FileNamePattern>C:\\opt\\shibboleth-idp\\logs\\idp-process-%d{yyyy-MM-dd_HH-mm}.log</FileNamePattern>
I then stopped the service and used Commons Deameon procrun command: prunsrv //TS/JettyService
this runs the service "as a console application", and I'm logged in as "Administrator" account.
Interestingly, the first thing it does is roll over the log file, but only once. After I make subsequent authentications (at least a minute apart) the new log lines are appended to idp-process.log, ie. it doesn't roll again after each minute.
So I'm trying to understand whey this would happen. The Initial startup of the service as a console app (run as administrator) is able to roll the file, but any subsequent IdP operation is not able or allowed to roll it. Once the service is started as a console application (as Administrator) isn't all subsequent IdP processes run with Administrator permissions too ?
I have also been playing around with the jetty logs (including creation of resources/jetty-logging.properties) , but have temporarily broken logging :-o Prior to breaking it the stderr and stdout files in JETTY_HOME/logs weren't showing any errors/exceptions. I increased the logging level in resource/log4j.properties to "log4j.rootLogger=DEBUG, stdout", the logging didn't increase. And now, I can't seem to get logging at all. I'm working on fixing it now,
Thanks again for help and advice,
Karen
On 12/03/13 15:04, Robert Kuhar wrote:
My rollover problems have always been permissions problems. We usually find them by stopping the process that isn't rolling over its logs, switching user, and starting the process by hand as the user the process is supposed to run as. Virtually every time, standard out/standard error will show some IOException permissions thing. Jetty is doing its own logging of standard out/standard error, no? Is there a clue in that logfile?
Bob
On Tue, Mar 12, 2013 at 2:24 AM, Karen Murphy <k.l.murphy@qub.ac.uk> wrote:
Hi all,
My Shibboleth IDP version 2.3.8 is failing to roll over files daily. Its running on Windows Server 2008, and the configuration is the default Shibboleth one in logging.xml, except I have changed the slashes to be double back slashes as per the logback documentation in Appenders.
I am running the IDP under Jetty 7.3, and have configured Jetty to run as a service using procrun, as the 'local system account' which I believe has all permissions necessary.
It works on another IDP I look after (version 2.3.6, with Tomcat) on Windows Server 2003.
I have read the posts referring to JNDI and rotation failing, but I can't see JNDI configured in my context file under Jetty, so guess this isn't affecting it ?
I have looked through various logs in the Event Viewer but don't see anything to indicate a logback error on file rotation.
My Shibboleth logging.xml contains:
----------------------------------------------------------------snip-- <appender name="IDP_PROCESS" class="ch.qos.logback.core.rolling.RollingFileAppender"> <File>C:\\opt\\shibboleth-idp\\logs\\idp-process.log</File>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<FileNamePattern>C:\\opt\\shibboleth-idp\\logs\\idp-process-%d{yyyy-MM-dd}.log</FileNamePattern>
</rollingPolicy>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <charset>UTF-8</charset> <Pattern>%date{HH:mm:ss.SSS} - %level [%logger:%line] - %msg%n</Pattern> </encoder> </appender> ---------------------------------------------------------------------
Does anyone have any ideas as to what might be causing roll over to fail ?
Any pointers would be much appreciated,
Kind Regards
Karen
-- *Karen Murphy* Systems Analyst - Bibliographic Services The Library at Queen's Queen's University Belfast Belfast BT7 1LP Tel: 028 90976260 Email: k.l.murphy@qub.ac.uk _______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
-- *Karen Murphy* Systems Analyst - Bibliographic Services The Library at Queen's Queen's University Belfast Belfast BT7 1LP Tel: 028 90976260 Email: k.l.murphy@qub.ac.uk
Hi all, Just want to record my conclusion on this issue in case its useful to someone. It seems Apache Common-Deamon procrun has issues with log rotatation, Sounds like log rotation isn't really supported yet: See: https://issues.apache.org/jira/browse/DAEMON-259 I'm thinking that this is probably related to my problem with log rotation. Will wait and see how the issue progresses and in the meantime put a scheduled task in place for rotation, Thanks, Karen On 12/03/13 15:04, Robert Kuhar wrote:
My rollover problems have always been permissions problems. We usually find them by stopping the process that isn't rolling over its logs, switching user, and starting the process by hand as the user the process is supposed to run as. Virtually every time, standard out/standard error will show some IOException permissions thing. Jetty is doing its own logging of standard out/standard error, no? Is there a clue in that logfile?
Bob
On Tue, Mar 12, 2013 at 2:24 AM, Karen Murphy <k.l.murphy@qub.ac.uk> wrote:
Hi all,
My Shibboleth IDP version 2.3.8 is failing to roll over files daily. Its running on Windows Server 2008, and the configuration is the default Shibboleth one in logging.xml, except I have changed the slashes to be double back slashes as per the logback documentation in Appenders.
I am running the IDP under Jetty 7.3, and have configured Jetty to run as a service using procrun, as the 'local system account' which I believe has all permissions necessary.
It works on another IDP I look after (version 2.3.6, with Tomcat) on Windows Server 2003.
I have read the posts referring to JNDI and rotation failing, but I can't see JNDI configured in my context file under Jetty, so guess this isn't affecting it ?
I have looked through various logs in the Event Viewer but don't see anything to indicate a logback error on file rotation.
My Shibboleth logging.xml contains:
----------------------------------------------------------------snip-- <appender name="IDP_PROCESS" class="ch.qos.logback.core.rolling.RollingFileAppender"> <File>C:\\opt\\shibboleth-idp\\logs\\idp-process.log</File>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<FileNamePattern>C:\\opt\\shibboleth-idp\\logs\\idp-process-%d{yyyy-MM-dd}.log</FileNamePattern> </rollingPolicy>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <charset>UTF-8</charset> <Pattern>%date{HH:mm:ss.SSS} - %level [%logger:%line] - %msg%n</Pattern> </encoder> </appender> ---------------------------------------------------------------------
Does anyone have any ideas as to what might be causing roll over to fail ?
Any pointers would be much appreciated,
Kind Regards
Karen
-- *Karen Murphy* Systems Analyst - Bibliographic Services The Library at Queen's Queen's University Belfast Belfast BT7 1LP Tel: 028 90976260 Email: k.l.murphy@qub.ac.uk _______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
_______________________________________________ Logback-user mailing list Logback-user@qos.ch http://mailman.qos.ch/mailman/listinfo/logback-user
-- *Karen Murphy* Systems Analyst - Bibliographic Services The Library at Queen's Queen's University Belfast Belfast BT7 1LP Tel: 028 90976260 Email: k.l.murphy@qub.ac.uk
participants (2)
-
Karen Murphy -
Robert Kuhar