On Wed, Mar 18, 2009 at 10:25 PM, Ceki Gulcu <ceki@qos.ch> wrote:
Are you using the standard corpus with 50'000? If so, 12MB for 50'000 elements translates to about 240 bytes per message which is pretty good. The CPU times, e.g. 1883 ms, are fantastic.
yep, ILoggingEvent[] eventArray = Corpus.makeStandardCorpus(); for (int i=0; i<10; i++) { ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream("/tmp/logback-java.ser." + i)); OutputStream os = new BufferedOutputStream(new FileOutputStream("/tmp/logback-protobuf.ser" + i)); GZIPOutputStream gos = new GZIPOutputStream(new FileOutputStream("/tmp/logback-proto." + i + ".gz"), 8000); DeflaterOutputStream dos = new DeflaterOutputStream( new FileOutputStream("/tmp/logback-protobuf.def." + i), new Deflater(Deflater.BEST_SPEED)); serialize(eventArray, oos); serializeWithProtobuf("normal", eventArray, os); serializeWithProtobuf("gzip ", eventArray, gos); serializeWithProtobuf("def ", eventArray, dos); oos.close(); os.close(); gos.close(); } Still have to create a multi-threaded test, but I suspect it won't make much difference. Maarten
Maarten Bosteels wrote:
Hi Ceki,
Keep in mind that I still have to double check my LoggingEvent-to-protobuf converter.
Serializing the corpus with different formats: java serialization => 16 MB (16108602) protobuf => 33 MB (34385867 bytes) protobuf + gzip => 9.9 MB (10354646 bytes) protbuf + deflater => 12 MB (12006206 bytes)
speed:
java serialization: 4330 ms protobuf 928 ms protobuf + gzip 3146 ms protobuf + deflater 1883 ms
So I would choose for protobuf + deflater : 25% smaller than java serilaization and twice as fast.
Deflater means using java.util.zip.Deflater.BEST_SPEED
regards, Maarten
On Wed, Mar 18, 2009 at 9:49 PM, Ceki Gulcu <ceki@qos.ch> wrote:
Hello Maarten,
It's interesting to learn that protobuf is faster than plain old java serialization. Thanks.
How about the size of the stored data? How many bytes does a logging event use on average?
Maarten Bosteels wrote:
Hi,
No objections at all.
Looking at my .proto file it's indeed pretty obvious that CallerData and StackTraceElement are quite similar :-)
http://code.google.com/p/firewood/source/browse/trunk/compare-formats/src/ma...
I've done some benchmarking with protobuf (using the Corpus), and it seems to be at least 4 times faster than Java serialization. Will soon post the results and the code.
regards, Maarten
-- Ceki Gülcü Logback: The reliable, generic, fast and flexible logging framework for Java. http://logback.qos.ch _______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
------------------------------------------------------------------------
_______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev
-- Ceki Gülcü Logback: The reliable, generic, fast and flexible logging framework for Java. http://logback.qos.ch _______________________________________________ logback-dev mailing list logback-dev@qos.ch http://qos.ch/mailman/listinfo/logback-dev