[JIRA] Created: (LBCORE-220) DBAppender.java performance improvement
DBAppender.java performance improvement --------------------------------------- Key: LBCORE-220 URL: http://jira.qos.ch/browse/LBCORE-220 Project: logback-core Issue Type: Bug Components: Other Affects Versions: 0.9.29 Reporter: Greg Thomas Assignee: Logback dev list We're seeing a distinct lack of performance with the DBAppender. Although it works fine in a single threaded environment, in a multi-threaded environment it slows down considerably. We've done some investigation, and narrowed it down to the pair of synchronized calls to subAppend() and selectEventId() (lines 107->110). I've had a look at this, and can see no reason for these to be synchronized; If using the JDBC3 getGeneratedKeys() method then as this applies to the thread specific statement there is no need to be synchronized. All the "Dialect" based calls are also connection specific; in detail; H2, HSQLDB and MySQL all call a database function, which is connection specific. MsSQL and Sybase both select a connection specific variable. Oracle and PostgreSQL both select the currval from a sequence, which again is connection specific. Having removed the synchronisation, we're seeing vastly improved logging throughput in our application. So to summarise; there is unnecessary synchronisation in the DBAppender that slows down multi-threaded applications; removing that synchronisation will improve performance in multi-threaded applications with no side affects. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.qos.ch/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[ http://jira.qos.ch/browse/LBCORE-220?page=com.atlassian.jira.plugin.system.i... ] Greg Thomas commented on LBCORE-220: ------------------------------------ I should have added at the time, I've submitted pull request 29 @ https://github.com/ceki/logback/pull/29 with a fix for this.
DBAppender.java performance improvement ---------------------------------------
Key: LBCORE-220 URL: http://jira.qos.ch/browse/LBCORE-220 Project: logback-core Issue Type: Bug Components: Other Affects Versions: 0.9.29 Reporter: Greg Thomas Assignee: Logback dev list
We're seeing a distinct lack of performance with the DBAppender. Although it works fine in a single threaded environment, in a multi-threaded environment it slows down considerably. We've done some investigation, and narrowed it down to the pair of synchronized calls to subAppend() and selectEventId() (lines 107->110). I've had a look at this, and can see no reason for these to be synchronized; If using the JDBC3 getGeneratedKeys() method then as this applies to the thread specific statement there is no need to be synchronized. All the "Dialect" based calls are also connection specific; in detail; H2, HSQLDB and MySQL all call a database function, which is connection specific. MsSQL and Sybase both select a connection specific variable. Oracle and PostgreSQL both select the currval from a sequence, which again is connection specific. Having removed the synchronisation, we're seeing vastly improved logging throughput in our application. So to summarise; there is unnecessary synchronisation in the DBAppender that slows down multi-threaded applications; removing that synchronisation will improve performance in multi-threaded applications with no side affects.
-- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.qos.ch/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[ http://jira.qos.ch/browse/LBCORE-220?page=com.atlassian.jira.plugin.system.i... ] Ceki Gulcu commented on LBCORE-220: ----------------------------------- Hi Greg, I had already seen the pull request thanks. I can see why relaxing synchronization would improve performance. However, it is not obvious to me why the result would still be safe. In particular, I don't see what "connection specific" entails. Regardless, I'd like to test the assumption that the proposed modifications preserve the safety of insertion ids. This could be done by making several hundred insertions from several threads and then checking that the resulting records are coherent. Would you care to implement this test and run it on say Postgres, MySQL, H2, MS SQLServer and Oracle? To be totally frank, I'd love to delegate responsibility of DBAppender to someone knowledgeable like yourself. Wink wink.
DBAppender.java performance improvement ---------------------------------------
Key: LBCORE-220 URL: http://jira.qos.ch/browse/LBCORE-220 Project: logback-core Issue Type: Bug Components: Other Affects Versions: 0.9.29 Reporter: Greg Thomas Assignee: Logback dev list
We're seeing a distinct lack of performance with the DBAppender. Although it works fine in a single threaded environment, in a multi-threaded environment it slows down considerably. We've done some investigation, and narrowed it down to the pair of synchronized calls to subAppend() and selectEventId() (lines 107->110). I've had a look at this, and can see no reason for these to be synchronized; If using the JDBC3 getGeneratedKeys() method then as this applies to the thread specific statement there is no need to be synchronized. All the "Dialect" based calls are also connection specific; in detail; H2, HSQLDB and MySQL all call a database function, which is connection specific. MsSQL and Sybase both select a connection specific variable. Oracle and PostgreSQL both select the currval from a sequence, which again is connection specific. Having removed the synchronisation, we're seeing vastly improved logging throughput in our application. So to summarise; there is unnecessary synchronisation in the DBAppender that slows down multi-threaded applications; removing that synchronisation will improve performance in multi-threaded applications with no side affects.
-- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.qos.ch/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[ http://jira.qos.ch/browse/LBCORE-220?page=com.atlassian.jira.plugin.system.i... ] Greg Thomas commented on LBCORE-220: ------------------------------------ Well, I'm most familiar with MySQL and MS SQL Server, but did check the documentation for the others. In each case, the method used to return the id of the inserted row returns the id of the row last inserted for that database connection; so a row inserted by another thread is ignored. However, I can see the desirability of some tests. I'll have a look to see how easy it would be to write some.
DBAppender.java performance improvement ---------------------------------------
Key: LBCORE-220 URL: http://jira.qos.ch/browse/LBCORE-220 Project: logback-core Issue Type: Bug Components: Other Affects Versions: 0.9.29 Reporter: Greg Thomas Assignee: Logback dev list
We're seeing a distinct lack of performance with the DBAppender. Although it works fine in a single threaded environment, in a multi-threaded environment it slows down considerably. We've done some investigation, and narrowed it down to the pair of synchronized calls to subAppend() and selectEventId() (lines 107->110). I've had a look at this, and can see no reason for these to be synchronized; If using the JDBC3 getGeneratedKeys() method then as this applies to the thread specific statement there is no need to be synchronized. All the "Dialect" based calls are also connection specific; in detail; H2, HSQLDB and MySQL all call a database function, which is connection specific. MsSQL and Sybase both select a connection specific variable. Oracle and PostgreSQL both select the currval from a sequence, which again is connection specific. Having removed the synchronisation, we're seeing vastly improved logging throughput in our application. So to summarise; there is unnecessary synchronisation in the DBAppender that slows down multi-threaded applications; removing that synchronisation will improve performance in multi-threaded applications with no side affects.
-- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.qos.ch/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
participants (2)
-
Ceki Gulcu (JIRA) -
Greg Thomas (JIRA)