Hello all, Logback contains a small number of time-sensitive tests. The time needed to execute these tests depend on the capabilities of the host CPU. For example, if on my machine a test executes in 1 millisecond, I would expect it to finish in say 1*10 milliseconds on most machine -- allowing for 10 fold variation. Does anyone know of a junit extension which can help me normalize the coefficient, 10 in the previous example, so that it adapts to the speed of the host CPU? There is also the problem of the JIT compiler... Any recommendations on the subject? -- Ceki Gülcü
Ceki Gulcu skrev den 04-08-2008 15:58:
Hello all,
Logback contains a small number of time-sensitive tests. The time needed to execute these tests depend on the capabilities of the host CPU. For example, if on my machine a test executes in 1 millisecond, I would expect it to finish in say 1*10 milliseconds on most machine -- allowing for 10 fold variation.
Does anyone know of a junit extension which can help me normalize the coefficient, 10 in the previous example, so that it adapts to the speed of the host CPU? There is also the problem of the JIT compiler...
Any recommendations on the subject?
I have not heard of such an extension. I would however expect that a concept similar to the BogoMips might be applicable. Namely that you run a reference program for X miliseconds and see how many Y operations you could run. You then use that number Y for calibration. What is it you want to normalize against? -- Thorbjørn Ravn Andersen "... plus... Tubular Bells!"
Ceki Gulcu wrote:
Hello all,
Logback contains a small number of time-sensitive tests. The time needed to execute these tests depend on the capabilities of the host CPU. For example, if on my machine a test executes in 1 millisecond, I would expect it to finish in say 1*10 milliseconds on most machine -- allowing for 10 fold variation.
Does anyone know of a junit extension which can help me normalize the coefficient, 10 in the previous example, so that it adapts to the speed of the host CPU? There is also the problem of the JIT compiler...
Any recommendations on the subject?
Well, I don't have a real solution to your problem... The only performance-related JUnit framework I'm aware of is http://clarkware.com/software/JUnitPerf.html but I think it's outdated and I never used it. It could be useful for load-testing, though. If you are using JUnit 4.x I'd suggest something like the following: import org.junit.Test; import org.junit.BeforeClass; import org.junit.Before; import static junit.framework.Assert.assertEquals; import static junit.framework.Assert.fail; public class TimingTest { private static long referenceTime; @BeforeClass public static void initReferenceTime() { long time=System.currentTimeMillis(); // lets waste some time... StringBuffer msg=new StringBuffer(); for(int i=0;i<10000;i++) { msg.append(i); } System.out.println(msg.toString()); referenceTime=System.currentTimeMillis()-time; System.out.println("ReferenceTime: "+referenceTime+" millis"); } @Test(timeout = 1000) public void works() throws Exception { long time=System.currentTimeMillis(); Thread.sleep(referenceTime*5); time=System.currentTimeMillis()-time; long maxTime=referenceTime*10; if(time>maxTime) { fail("Test was not expected to take more than "+maxTime+" millis but took "+time+" millis!"); } } @Test(timeout = 1000) public void fails() throws Exception { long time=System.currentTimeMillis(); Thread.sleep(referenceTime*15); time=System.currentTimeMillis()-time; long maxTime=referenceTime*10; if(time>maxTime) { fail("Test was not expected to take more than "+maxTime+" millis but took "+time+" millis!"); } } } The problem with this approach, however, is that referenceTime isn't very stable. It fluctuates between 15ms and 40ms on my development system. I guess this fluctuation could be reduced by using an even higher number in the initReferenceTime loop. The time-wasting would have to be changed depending on the problem-domain of the real test, e.g. something like the above if String operations are performed in the actual code that is tested. The timeout I've given in @Test is supposed to be a definitive upper level that should never be reached. This could be left out, though. I have some doubts what the Hotspot-JIT will be doing if the above initReferenceTime is used the same way in more than one class. I have some hope that it does NOT realize that it's the same code but I'm not sure. The code above isn't perfect, neither is it especially elegant, but it's probably good enough for your use case if initReferenceTime and the factors are tuned a bit... I don't know. Joern.
Joern, Thanks for your input. Instead of forcing the test to sleep, another approach is to adjust the expected performance according to the capabilities of the host CPU. This is the approach adopted in BogoPerf. See http://svn.qos.ch/viewvc/logback/trunk/logback-classic/src/test/java/ch/qos/... http://svn.qos.ch/viewvc/logback/trunk/logback-classic/src/test/java/ch/qos/... Joern Huxhorn wrote:
Ceki Gulcu wrote:
Hello all,
Logback contains a small number of time-sensitive tests. The time needed to execute these tests depend on the capabilities of the host CPU. For example, if on my machine a test executes in 1 millisecond, I would expect it to finish in say 1*10 milliseconds on most machine -- allowing for 10 fold variation.
Does anyone know of a junit extension which can help me normalize the coefficient, 10 in the previous example, so that it adapts to the speed of the host CPU? There is also the problem of the JIT compiler...
Any recommendations on the subject?
Well, I don't have a real solution to your problem... The only performance-related JUnit framework I'm aware of is http://clarkware.com/software/JUnitPerf.html but I think it's outdated and I never used it. It could be useful for load-testing, though.
If you are using JUnit 4.x I'd suggest something like the following:
import org.junit.Test; import org.junit.BeforeClass; import org.junit.Before; import static junit.framework.Assert.assertEquals; import static junit.framework.Assert.fail;
public class TimingTest {
private static long referenceTime;
@BeforeClass public static void initReferenceTime() { long time=System.currentTimeMillis();
// lets waste some time... StringBuffer msg=new StringBuffer(); for(int i=0;i<10000;i++) { msg.append(i); } System.out.println(msg.toString()); referenceTime=System.currentTimeMillis()-time; System.out.println("ReferenceTime: "+referenceTime+" millis"); }
@Test(timeout = 1000) public void works() throws Exception { long time=System.currentTimeMillis(); Thread.sleep(referenceTime*5); time=System.currentTimeMillis()-time; long maxTime=referenceTime*10; if(time>maxTime) { fail("Test was not expected to take more than "+maxTime+" millis but took "+time+" millis!"); } }
@Test(timeout = 1000) public void fails() throws Exception { long time=System.currentTimeMillis(); Thread.sleep(referenceTime*15); time=System.currentTimeMillis()-time; long maxTime=referenceTime*10; if(time>maxTime) { fail("Test was not expected to take more than "+maxTime+" millis but took "+time+" millis!"); } } }
The problem with this approach, however, is that referenceTime isn't very stable. It fluctuates between 15ms and 40ms on my development system. I guess this fluctuation could be reduced by using an even higher number in the initReferenceTime loop. The time-wasting would have to be changed depending on the problem-domain of the real test, e.g. something like the above if String operations are performed in the actual code that is tested.
The timeout I've given in @Test is supposed to be a definitive upper level that should never be reached. This could be left out, though.
I have some doubts what the Hotspot-JIT will be doing if the above initReferenceTime is used the same way in more than one class. I have some hope that it does NOT realize that it's the same code but I'm not sure.
The code above isn't perfect, neither is it especially elegant, but it's probably good enough for your use case if initReferenceTime and the factors are tuned a bit... I don't know.
Joern. _______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
-- Ceki Gülcü QOS.ch is looking to hire talented developers located in Switzerland to work on cutting-edge software projects. If you think you are qualified, then please contact ceki@qos.ch.
On Mon, Aug 4, 2008 at 10:21 PM, Ceki Gulcu <listid@qos.ch> wrote:
Joern,
Thanks for your input. Instead of forcing the test to sleep, another approach is
The sleep was just meant as a placeholder for the code that will take some time, i.e. the actual code of the test. So I constructed one case where the test will work for sure, and one test that would always fail because it takes longer than the expected amount of time. I probably should have added some comments :)
to adjust the expected performance according to the capabilities of the host CPU. This is the approach adopted in BogoPerf. See
http://svn.qos.ch/viewvc/logback/trunk/logback-classic/src/test/java/ch/qos/...
http://svn.qos.ch/viewvc/logback/trunk/logback-classic/src/test/java/ch/qos/...
I'll take a look for sure. Joern.
Joern Huxhorn wrote:
Ceki Gulcu wrote:
Hello all,
Logback contains a small number of time-sensitive tests. The time needed to execute these tests depend on the capabilities of the host CPU. For example, if on my machine a test executes in 1 millisecond, I would expect it to finish in say 1*10 milliseconds on most machine -- allowing for 10 fold variation.
Does anyone know of a junit extension which can help me normalize the coefficient, 10 in the previous example, so that it adapts to the speed of the host CPU? There is also the problem of the JIT compiler...
Any recommendations on the subject?
Well, I don't have a real solution to your problem... The only performance-related JUnit framework I'm aware of is http://clarkware.com/software/JUnitPerf.html but I think it's outdated and I never used it. It could be useful for load-testing, though.
If you are using JUnit 4.x I'd suggest something like the following:
import org.junit.Test; import org.junit.BeforeClass; import org.junit.Before; import static junit.framework.Assert.assertEquals; import static junit.framework.Assert.fail;
public class TimingTest {
private static long referenceTime;
@BeforeClass public static void initReferenceTime() { long time=System.currentTimeMillis();
// lets waste some time... StringBuffer msg=new StringBuffer(); for(int i=0;i<10000;i++) { msg.append(i); } System.out.println(msg.toString()); referenceTime=System.currentTimeMillis()-time; System.out.println("ReferenceTime: "+referenceTime+" millis"); }
@Test(timeout = 1000) public void works() throws Exception { long time=System.currentTimeMillis(); Thread.sleep(referenceTime*5); time=System.currentTimeMillis()-time; long maxTime=referenceTime*10; if(time>maxTime) { fail("Test was not expected to take more than "+maxTime+" millis but took "+time+" millis!"); } }
@Test(timeout = 1000) public void fails() throws Exception { long time=System.currentTimeMillis(); Thread.sleep(referenceTime*15); time=System.currentTimeMillis()-time; long maxTime=referenceTime*10; if(time>maxTime) { fail("Test was not expected to take more than "+maxTime+" millis but took "+time+" millis!"); } } }
The problem with this approach, however, is that referenceTime isn't very stable. It fluctuates between 15ms and 40ms on my development system. I guess this fluctuation could be reduced by using an even higher number in the initReferenceTime loop. The time-wasting would have to be changed depending on the problem-domain of the real test, e.g. something like the above if String operations are performed in the actual code that is tested.
The timeout I've given in @Test is supposed to be a definitive upper level that should never be reached. This could be left out, though.
I have some doubts what the Hotspot-JIT will be doing if the above initReferenceTime is used the same way in more than one class. I have some hope that it does NOT realize that it's the same code but I'm not sure.
The code above isn't perfect, neither is it especially elegant, but it's probably good enough for your use case if initReferenceTime and the factors are tuned a bit... I don't know.
Joern. _______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
-- Ceki Gülcü
QOS.ch is looking to hire talented developers located in Switzerland to work on cutting-edge software projects. If you think you are qualified, then please contact ceki@qos.ch. _______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
On Mon, Aug 4, 2008 at 10:21 PM, Ceki Gulcu <listid@qos.ch> wrote:
Joern,
Thanks for your input. Instead of forcing the test to sleep, another approach is to adjust the expected performance according to the capabilities of the host CPU. This is the approach adopted in BogoPerf. See
http://svn.qos.ch/viewvc/logback/trunk/logback-classic/src/test/java/ch/qos/...
http://svn.qos.ch/viewvc/logback/trunk/logback-classic/src/test/java/ch/qos/...
Yes, that does look quite good :)
participants (4)
-
Ceki Gulcu -
Joern Huxhorn -
Jörn Huxhorn
-
Thorbjørn Ravn Andersen