<br><br><div class="gmail_quote">On Thu, Nov 19, 2009 at 10:33 PM, Joern Huxhorn <span dir="ltr"><<a href="mailto:jhuxhorn@googlemail.com">jhuxhorn@googlemail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im"><br>
On 19.11.2009, at 19:06, Thorbjoern Ravn Andersen wrote:<br>
<br>
> Hi.<br>
><br>
> We have reached a situation where I basically want to log a data structure in order to be able to process it later.<br>
><br>
> After a bit of pondering, I have concluded that the best approach for us to do this would be to use the XMLEncoder/XMLDecoder in Java 1.4+ and log the generated XML snippets.<br>
><br>
> The issue I want to solve is that the XMLEncoder writes an UTF-8 encoded XML file to an OutputStream, i.e. a byte oriented destination. To the best of my knowledge the slf4j backends all deal with Strings, i.e. character oriented destinations and the output files are written in the default encoding for the platform.<br>
><br>
> The question now is, what is the best way to handle the OutputStream generated by XMLEncoder so it will survive all attempts to mess up any unicode characters inside due to encoding differences on the way. I will be using a custom layout anyway so much can be done :) A humanly readable transport format will be preferred.<br>
><br>
<br>
</div>XMLEncoder will have a severe impact on performance, I've tested this extensively.<br>
Have a look at <a href="http://sourceforge.net/apps/trac/lilith/wiki/SerializationPerformance" target="_blank">http://sourceforge.net/apps/trac/lilith/wiki/SerializationPerformance</a><br>
In my testcases, XMLEncoder serialized 300 events while a protobuf serializer managed to handle nearly 10.000!<br>
I'd therefore suggest that you take a mixed approach. Using protobuf to serialize the events to a file and writing an additional converter to convert that files to whatever you'd like as XML-Output as needed.<br>
<br>
A discussion about such a topic was started here: <a href="http://marc.info/?l=logback-dev&m=124905434331308&w=2" target="_blank">http://marc.info/?l=logback-dev&m=124905434331308&w=2</a> but I completely forgot to file an RFE for it.<br>
I've done just that now, thanks for the reminder!<br>
<a href="http://jira.qos.ch/browse/LBCORE-128" target="_blank">http://jira.qos.ch/browse/LBCORE-128</a><br></blockquote><div><br>I agree with Joern that XMLEncoder is not really suited when throughput is important to you. <br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im"><br>
> My current thoughts is to use a ByteArrayOutputStream and generate a String using the UTF-8 decoding. The resulting string contains a <?xml ... encoding="UTF-8"?> which is stripped resulting in an XML String containing Unicode chars (instead of encoded bytes). </div>
</blockquote><div><br>What is the difference between "Unicode chars" and "encoded bytes" ?<br>Every unicode codepoint has to be encoded somehow, no ? UTF-8 is one way to encode the codepoint (and imho the encoding everyone should use)<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="im">This can then be flattened to an ASCII version, by converting all characters outside 32..127 to their numeric entity (&#1234;), and THAT can be safely logged. I guess :)<br>
><br></div></blockquote><div><br>If you want to use XML, then I really don't see the problem with leaving it in UTF-8 ?<br>Especially since you state that "a humanly readable transport format will be preferred." I would prefer to see Σ instead of <span class="sc1">&#931;</span> <br>
<br>Of course, it should be possible to tell the XMLEncoder which encoding to use (instead of using the default encoding of the platform).<br><br>regards,<br>Maarten<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im">
<br>
</div>That would probably work but it would further decrease the serialization speed.<br>
Logback (assuming you use Logback) should really support binary, i.e. byte-based, logfiles since this would really make a major performance difference. This should be discussed over at logback-dev, though.<br>
<div class="im"><br>
> I'd appreciate comments on my thoughts, as this is a rather important intermediate step in us using log files to store information which can be used to simulate an external system when replaying an interesting sesion.<br>
><br>
><br>
<br>
</div>HTH,<br>
<font color="#888888">Joern.<br>
</font><div><div></div><div class="h5"><br>
_______________________________________________<br>
user mailing list<br>
<a href="mailto:user@slf4j.org">user@slf4j.org</a><br>
<a href="http://www.slf4j.org/mailman/listinfo/user" target="_blank">http://www.slf4j.org/mailman/listinfo/user</a><br>
</div></div></blockquote></div><br>