[JIRA] Created: (LBCORE-115) SizeBasedTriggeringPolicy is not effecient enough because of using File.length() to check the active file size
SizeBasedTriggeringPolicy is not effecient enough because of using File.length() to check the active file size -------------------------------------------------------------------------------------------------------------- Key: LBCORE-115 URL: http://jira.qos.ch/browse/LBCORE-115 Project: logback-core Issue Type: Improvement Components: Rolling Affects Versions: 0.9.17 Environment: windows, linux Reporter: tomliliu Assignee: Logback dev list Hi, It looks like SizeBasedTriggeringPolicy is not effecient enough because it will check active file size by calling File.length(), which will add a fixed overhead every 16 messages. (The performance definitely has been improved from 0.9.15 to 0.9.17 a lot by checking the file size every 16 message instead of every messages, we saw 20% overhead cut down on linux local file system.) We think SizeBasedTriggeringPolicy could work the FileAppender which keep track of how many bytes has been written through, and it's more efficient to check the file size based on how many bytes has been written. We also quickly made a mock up which keeps track of how many bytes written for comparison. Test Enviornment: Windows XP SP2, Linux 2.4 Test #1: Writing 10 bytes message 1, 000, 000 times. (No rotation happens) Platform File.length() Tracking bytes written overhead (%) Windows ~ 12, 200 ms ~ 10, 000 ms 22% Linux (local) ~ 10, 200 ms ~ 9, 800 ms 4% Linux (NFS) ~ 37, 700 ms ~ 27, 300 ms 38% Test #2: Writing 100 bytes messages 1, 000, 000 times. (No rotation happens) Platform File.length() Tracking bytes written overhead (%) Windows ~ 14, 000 ms ~ 11, 300 ms 23% Linux (local) ~ 11, 100 ms ~ 10, 300 ms 7% Linux (NFS) ~ 414, 600 ms ~366, 300 ms 13% Could you please consider improve the performance of SizeBasedTriggeringPolicy so that there is minimal overhead. Great Thanks, Tom -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.qos.ch/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[ http://jira.qos.ch/browse/LBCORE-115?page=com.atlassian.jira.plugin.system.i... ] Ceki Gulcu commented on LBCORE-115: ----------------------------------- If I understand correctly, the current implementation is at worst 23% slower than the optimal implementation counting each byte written. Is that correct?
SizeBasedTriggeringPolicy is not effecient enough because of using File.length() to check the active file size --------------------------------------------------------------------------------------------------------------
Key: LBCORE-115 URL: http://jira.qos.ch/browse/LBCORE-115 Project: logback-core Issue Type: Improvement Components: Rolling Affects Versions: 0.9.17 Environment: windows, linux Reporter: tomliliu Assignee: Logback dev list
Hi, It looks like SizeBasedTriggeringPolicy is not effecient enough because it will check active file size by calling File.length(), which will add a fixed overhead every 16 messages. (The performance definitely has been improved from 0.9.15 to 0.9.17 a lot by checking the file size every 16 message instead of every messages, we saw 20% overhead cut down on linux local file system.) We think SizeBasedTriggeringPolicy could work the FileAppender which keep track of how many bytes has been written through, and it's more efficient to check the file size based on how many bytes has been written. We also quickly made a mock up which keeps track of how many bytes written for comparison. Test Enviornment: Windows XP SP2, Linux 2.4 Test #1: Writing 10 bytes message 1, 000, 000 times. (No rotation happens) Platform File.length() Tracking bytes written overhead (%) Windows ~ 12, 200 ms ~ 10, 000 ms 22% Linux (local) ~ 10, 200 ms ~ 9, 800 ms 4% Linux (NFS) ~ 37, 700 ms ~ 27, 300 ms 38% Test #2: Writing 100 bytes messages 1, 000, 000 times. (No rotation happens) Platform File.length() Tracking bytes written overhead (%) Windows ~ 14, 000 ms ~ 11, 300 ms 23% Linux (local) ~ 11, 100 ms ~ 10, 300 ms 7% Linux (NFS) ~ 414, 600 ms ~366, 300 ms 13% Could you please consider improve the performance of SizeBasedTriggeringPolicy so that there is minimal overhead. Great Thanks, Tom
-- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.qos.ch/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[ http://jira.qos.ch/browse/LBCORE-115?page=com.atlassian.jira.plugin.system.i... ] Ceki Gulcu edited comment on LBCORE-115 at 8/19/09 4:49 PM: ------------------------------------------------------------ Hello Tom, Thank you for creating this jira issue as well as the table listing performance figures. If I understand correctly, the current implementation is at worst 23% slower than the optimal implementation counting each byte written. Is that correct? was (Author: noreply.ceki@qos.ch): If I understand correctly, the current implementation is at worst 23% slower than the optimal implementation counting each byte written. Is that correct?
SizeBasedTriggeringPolicy is not effecient enough because of using File.length() to check the active file size --------------------------------------------------------------------------------------------------------------
Key: LBCORE-115 URL: http://jira.qos.ch/browse/LBCORE-115 Project: logback-core Issue Type: Improvement Components: Rolling Affects Versions: 0.9.17 Environment: windows, linux Reporter: tomliliu Assignee: Logback dev list
Hi, It looks like SizeBasedTriggeringPolicy is not effecient enough because it will check active file size by calling File.length(), which will add a fixed overhead every 16 messages. (The performance definitely has been improved from 0.9.15 to 0.9.17 a lot by checking the file size every 16 message instead of every messages, we saw 20% overhead cut down on linux local file system.) We think SizeBasedTriggeringPolicy could work the FileAppender which keep track of how many bytes has been written through, and it's more efficient to check the file size based on how many bytes has been written. We also quickly made a mock up which keeps track of how many bytes written for comparison. Test Enviornment: Windows XP SP2, Linux 2.4 Test #1: Writing 10 bytes message 1, 000, 000 times. (No rotation happens) Platform File.length() Tracking bytes written overhead (%) Windows ~ 12, 200 ms ~ 10, 000 ms 22% Linux (local) ~ 10, 200 ms ~ 9, 800 ms 4% Linux (NFS) ~ 37, 700 ms ~ 27, 300 ms 38% Test #2: Writing 100 bytes messages 1, 000, 000 times. (No rotation happens) Platform File.length() Tracking bytes written overhead (%) Windows ~ 14, 000 ms ~ 11, 300 ms 23% Linux (local) ~ 11, 100 ms ~ 10, 300 ms 7% Linux (NFS) ~ 414, 600 ms ~366, 300 ms 13% Could you please consider improve the performance of SizeBasedTriggeringPolicy so that there is minimal overhead. Great Thanks, Tom
-- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.qos.ch/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[ http://jira.qos.ch/browse/LBCORE-115?page=com.atlassian.jira.plugin.system.i... ] tomliliu commented on LBCORE-115: --------------------------------- Hi Ceki, Thanks for responding. I've benchmarked serveral times and 23% seems the average on my machine (windows), typically from 22% ~ 25%. The bytes counting are done by creating a decorator on writer in setWriter(...). Thanks, Tom
SizeBasedTriggeringPolicy is not effecient enough because of using File.length() to check the active file size --------------------------------------------------------------------------------------------------------------
Key: LBCORE-115 URL: http://jira.qos.ch/browse/LBCORE-115 Project: logback-core Issue Type: Improvement Components: Rolling Affects Versions: 0.9.17 Environment: windows, linux Reporter: tomliliu Assignee: Logback dev list
Hi, It looks like SizeBasedTriggeringPolicy is not effecient enough because it will check active file size by calling File.length(), which will add a fixed overhead every 16 messages. (The performance definitely has been improved from 0.9.15 to 0.9.17 a lot by checking the file size every 16 message instead of every messages, we saw 20% overhead cut down on linux local file system.) We think SizeBasedTriggeringPolicy could work the FileAppender which keep track of how many bytes has been written through, and it's more efficient to check the file size based on how many bytes has been written. We also quickly made a mock up which keeps track of how many bytes written for comparison. Test Enviornment: Windows XP SP2, Linux 2.4 Test #1: Writing 10 bytes message 1, 000, 000 times. (No rotation happens) Platform File.length() Tracking bytes written overhead (%) Windows ~ 12, 200 ms ~ 10, 000 ms 22% Linux (local) ~ 10, 200 ms ~ 9, 800 ms 4% Linux (NFS) ~ 37, 700 ms ~ 27, 300 ms 38% Test #2: Writing 100 bytes messages 1, 000, 000 times. (No rotation happens) Platform File.length() Tracking bytes written overhead (%) Windows ~ 14, 000 ms ~ 11, 300 ms 23% Linux (local) ~ 11, 100 ms ~ 10, 300 ms 7% Linux (NFS) ~ 414, 600 ms ~366, 300 ms 13% Could you please consider improve the performance of SizeBasedTriggeringPolicy so that there is minimal overhead. Great Thanks, Tom
-- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.qos.ch/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
participants (2)
-
Ceki Gulcu (JIRA) -
tomliliu (JIRA)