Caoilte O'Connor2020-07-19T12:39:53+00:00http://caoilte.orgCaoilte O'Connornospam@caoilte.orgBending JAXB to the will of Scala (Part 2 of 2)2014-10-06T00:00:00+00:00http://caoilte.org/scala/2014/10/06/bending-jaxb-to-the-will-of-scala-2-of-2<p>This blog post is the second in a two part series that bottoms out solutions for the corner cases that I ran into whilst getting JAXB to work in Scala. I am indebted to Martin Krasser’s <a href="http://krasserm.blogspot.co.uk/2012/02/using-jaxb-for-xml-and-json-apis-in.html">blog post on JAXB</a> for providing an essential first primer on the topic.</p>
<p>This week I enumerate the difficulties with lists and the work arounds I found for making them manageable. You can still read the first part of the series (on optionals) <a href="http://caoilte.org/scala/2014/09/29/bending-jaxb-to-the-will-of-scala-1-of-2/">here</a>.</p>
<!--more-->
<h2 id="lists-of-the-reasons-to-consider-giving-up-on-jaxb">Lists of the reasons to consider giving up on JAXB</h2>
<p>Lists in JAXB are where the gaps between Java and Scala start to become painful. They’re also where I gave up for a week in disgust and found solace in signing up for <a href="https://www.edx.org/course/delftx/delftx-fp101x-introduction-functional-2126">Erik Meijer’s new Haskell MOOC</a>. I eventually succeeded with two different ways of modeling lists, only the second of which turned out suitable for implementing the Atom specification.</p>
<h3 id="containerized-lists">Containerized Lists</h3>
<p>Martin Krasser’s JAXB blog post doesn’t explain how to model lists, but does link to extensive and very helpful examples in his <a href="https://github.com/krasserm/eventsourcing-example/tree/play-blog/modules/service/src/main/scala/dev/example/eventsourcing">event sourcing project</a>.</p>
<p>With a little bit of tweaking I managed to get them working for me,</p>
<script src="https://gist.github.com/caoilte/d0f1b86cebf68d88a04a.js?file=jaxb-in-scala-2_01.scala"> </script>
<p>This code will turn,</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="nc">Entry</span><span class="o">(</span><span class="s">"101"</span><span class="o">,</span>
<span class="nc">List</span><span class="o">(</span>
<span class="nc">Person</span><span class="o">(</span><span class="s">"Caoilte O'Connor"</span><span class="o">),</span>
<span class="nc">Person</span><span class="o">(</span><span class="s">"Martin Krasser"</span><span class="o">)</span>
<span class="o">)</span>
<span class="o">)</span></code></pre></figure>
<p>into,</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="cp"><?xml version="1.0" encoding="UTF-8"?></span>
<span class="nt"><entry></span>
<span class="nt"><id></span>101<span class="nt"></id></span>
<span class="nt"><authors></span>
<span class="nt"><person></span>
<span class="nt"><name></span>Caoilte O'Connor<span class="nt"></name></span>
<span class="nt"></person></span>
<span class="nt"><person></span>
<span class="nt"><name></span>Martin Krasser<span class="nt"></name></span>
<span class="nt"></person></span>
<span class="nt"></authors></span>
<span class="nt"></entry></span></code></pre></figure>
<p>The most notable property of the xml is the nesting of list elements within an <code class="language-plaintext highlighter-rouge">authors</code> element. A lot goes on in order to achieve this containerization, so let’s break it down into a list of bullets,</p>
<ul>
<li>JAXB doesn’t know how to handle Scala Lists, but it can map Java Lists and so the <code class="language-plaintext highlighter-rouge">AbstractListAdapter</code> turns a Scala List into a container of a Java List and vice versa.</li>
<li>Unfortunately one <code class="language-plaintext highlighter-rouge">XmlAdapter</code> cannot encapsulate the entire abstract transformational logic as it has for previous examples. The <code class="language-plaintext highlighter-rouge">AbstractList</code> trait describes the properties that the container requires.</li>
<li>The Person companion object then implements all of the abstract types described previously. This code is entirely boiler plate and seems like an excellent opportunity to re-watch Dave Gurnell’s <a href="https://www.parleys.com/play/53a7d2c4e4b0543940d9e542/chapter0/about">Macros for the Rest of Us</a> talk.</li>
<li>Finally, I have included the Person Case Class and an Entry Case class that incorporates a list of Persons.</li>
<li>The only change that I made to Martin Krasser’s code was because of the MOXy bug mentioned in the previous section, which required me to reverse the order of the type parameters on <code class="language-plaintext highlighter-rouge">AbstractListAdapter</code>.</li>
</ul>
<p>Unfortunately, despite how clever it is, this code isn’t suitable for the Atom Feed use case because that requires XML to be produced without the container element and this requires a completely different approach which I will discuss in the next section.</p>
<h3 id="mapping-scala-collections-to-java-collections-for-containerless-lists">Mapping Scala Collections to Java Collections for Containerless Lists</h3>
<p>Unfortunately the <code class="language-plaintext highlighter-rouge">XmlAdapter</code> can only be used to model the transformation for elements <em>in</em> a list. You cannot use it to model the transformation of the list itself. We got around this in the previous section by transforming a Scala List (which JAXB doesn’t recognise at all) into a <em>container</em> of lists. If you try and turn the Scala List into a Java List, <a href="https://gist.github.com/caoilte/d0f1b86cebf68d88a04a#file-jaxb-in-scala-2_02-txt">JAXB falls over hard</a>. (It recognises that the property will end up as a Java List but ignores the adapter and attempts to cast the Scala List to a Java Collection directly.)</p>
<p>Because of this limitation the entire approach needs to be re-thought. I tried utilizing other JAXB annotations (eg <a href="http://stackoverflow.com/questions/16202583/xmlelementwrapper-for-unwrapped-collections">@XmlElementWrapper</a>, <a href="http://stackoverflow.com/questions/11153599/unmarshalling-nested-collection-via-xmlpath-moxy-annotation">@XmlPath</a> and <a href="http://blog.bdoughan.com/2010/08/xmltransformation-going-beyond.html">@XmlTransformation</a>) but couldn’t get any of them to work with Scala. I had a similar lack of success with other Scala Collection types (<code class="language-plaintext highlighter-rouge">Seq</code>, <code class="language-plaintext highlighter-rouge">ArraySeq</code> and <code class="language-plaintext highlighter-rouge">WrappedArray</code>). JAXB managed to marshall some to XML but was unable to unmarshall any back to types.</p>
<p>The only type which worked was the Scala <code class="language-plaintext highlighter-rouge">Array</code>. This is probably because <code class="language-plaintext highlighter-rouge">Array</code> has some unique (and to be honest, rather awkward) properties. Unlike most (perhaps all?) other Scala types, <code class="language-plaintext highlighter-rouge">Array</code> compiles down to a Java Array in byte code and is indistinguishable to Java code. This has pros and cons,</p>
<p>On the plus side,</p>
<ul>
<li>JAXB knows how to map a Java Array to xml and back again</li>
<li>Scala makes all of its Collection mapping goodies available to an <code class="language-plaintext highlighter-rouge">Array</code></li>
<li>Arrays are more performant for very large data sets (I’m clutching at straws here)</li>
</ul>
<p>On the minus side,</p>
<ul>
<li>Arrays are mutable. Case Classes promote the use of immutable data because it is easier to reason with and it is a shame to be forced to give that up.</li>
<li>Because Arrays are really from Java, Scala cannot override the <code class="language-plaintext highlighter-rouge">equals</code>, <code class="language-plaintext highlighter-rouge">hashCode</code> and <code class="language-plaintext highlighter-rouge">toString</code> methods. This means they are pretty much the only type in Scala to compare with <em>reference</em> equality rather than <em>value</em> equality. Paul Phillips gives a great explanation <a href="http://stackoverflow.com/a/3738850/58005">here</a> about why that’s a good thing, but it still hurts and causes all sorts of problems using them in Case Classes (as we shall see).</li>
</ul>
<p>The comparison with reference equality can be worked around. There’s a great exploration of the possibilities <a href="http://blog.bruchez.name/2013/05/scala-array-comparison-without-phd.html">here</a>. Making Arrays work in a Case Class context is trickier, because a Case Class is really just <a href="https://groups.google.com/d/msg/scala-user/buGv-DNNvWs/QzdeSlHeCIEJ">a boilerplate generator</a> and the boilerplate generation makes no special exception for Arrays and the Java <code class="language-plaintext highlighter-rouge">equals</code>, <code class="language-plaintext highlighter-rouge">hashCode</code> and <code class="language-plaintext highlighter-rouge">toString</code> methods they expose.</p>
<p>I spent some time figuring out how to work around this. At first I got very excited to discover that Case Classes support two parameter lists and that
<a href="http://stackoverflow.com/questions/10373715/scala-ignore-case-class-field-for-equals-hascode/10373946#10373946">only parameters in the first parameter section are considered for equality and hashing.</a> (HT: Jason Zaugg, again.) However, there is no way to partially extend equals/toString/hashCode and so this didn’t gain me anything as I still had to override everything. A better solution presented itself after I saw this <a href="https://gist.github.com/okapies/2814608">this manual reimplementation of a case class</a> on a Stack Overflow question about <a href="http://stackoverflow.com/a/11396362/58005">overriding the equals method on a case class</a>. My eventual (and I believe, optimal) solution borrowed several tricks from this discussion,</p>
<script src="https://gist.github.com/caoilte/d0f1b86cebf68d88a04a.js?file=jaxb-in-scala-2_03.scala"> </script>
<p>This code will turn,</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="nc">Entry</span><span class="o">(</span><span class="s">"101"</span><span class="o">,</span>
<span class="nc">Array</span><span class="o">(</span>
<span class="nc">Person</span><span class="o">(</span><span class="s">"Caoilte O'Connor"</span><span class="o">),</span>
<span class="nc">Person</span><span class="o">(</span><span class="s">"Martin Krasser"</span><span class="o">)</span>
<span class="o">)</span>
<span class="o">)</span></code></pre></figure>
<p>into,</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="cp"><?xml version="1.0" encoding="UTF-8"?></span>
<span class="nt"><entry></span>
<span class="nt"><id></span>101<span class="nt"></id></span>
<span class="nt"><author></span>
<span class="nt"><name></span>Caoilte O'Connor<span class="nt"></name></span>
<span class="nt"></author></span>
<span class="nt"><author></span>
<span class="nt"><name></span>Martin Krasser<span class="nt"></name></span>
<span class="nt"></author></span>
<span class="nt"></entry></span></code></pre></figure>
<p>Let’s consider what is going on,</p>
<ul>
<li>One of the lesser known features of a Case Class is that the Scala compiler auto-generates it an implementation of the <a href="http://www.scala-lang.org/api/current/index.html#scala.Product">Product Trait</a>. The Product Trait provides an interface for programmatically accessing the attribute values of a class. The Scala Compiler then auto-generates <code class="language-plaintext highlighter-rouge">toString</code> and <code class="language-plaintext highlighter-rouge">hashCode</code> methods that rely on the Product Trait. This gives us a code seam where we can interpose our own implementation of the Product trait for a Case Class, whilst still benefitting from other auto-generated goodness.</li>
<li>The only difference between our implementation of Product and the auto-generated implementation is that we override Arrays returned by the <code class="language-plaintext highlighter-rouge">productElement</code> method in a <a href="http://www.scala-lang.org/api/current/index.html#scala.collection.mutable.WrappedArray">Wrapped Array</a>. The Wrapped Array is identical to a normal Array except that it overrides <code class="language-plaintext highlighter-rouge">toString</code>, <code class="language-plaintext highlighter-rouge">hashCode</code> and <code class="language-plaintext highlighter-rouge">equals</code> with more useful implementations.</li>
<li>The <a href="https://gist.github.com/okapies/2814608">manual reimplementation of a case class</a> showed <code class="language-plaintext highlighter-rouge">equals</code> being implemented by the <code class="language-plaintext highlighter-rouge">ScalaRuntime._equals</code> method, but this is not actually the case with real case classes auto-generated by the Scala Compiler. I asked <a href="https://groups.google.com/forum/#!topic/scala-user/EfEb8MPa5BM">why</a> on the scala-user list and was relieved to learn that it was a performance optimisation. Consequently, there is nothing to stop us from overriding equals with this ourselves.</li>
</ul>
<p>It is worth re-considering the pros and cons at this point. On the plus side,</p>
<ul>
<li>We’ve succeeded in defining a Collection containing Case Class that JAXB can marshall to XML and back again into the same Case Class.</li>
<li>The Case Class we have defined doesn’t break any of the conventions that the Case Class provides. It still has the same constructor, pattern-matching, equality and, product attribute properties expected.</li>
<li>The hand rolled boiler plate is easy to understand and relatively simple to spot errors in.</li>
<li>With a little work it might be possible to define a macro to do all of the auto-generation for us.</li>
</ul>
<p>On the minus side,</p>
<ul>
<li>With larger Case Classes the boiler plate <a href="https://github.com/caoilte/atomizer/blob/755f3c6bc980a8477b2ee0806153579680661fcd/atom-model/src/main/scala/org/caoilte/atomizer/model/Atom.scala#L162">becomes overwhelming</a>.</li>
<li>Arrays are mutable and many people are uncomfortable using mutable classes in a Case Class.</li>
<li>Our <code class="language-plaintext highlighter-rouge">equals</code> method uses the <code class="language-plaintext highlighter-rouge">ScalaRuntime._equals</code> helper method which is described on the Javadoc as being <em>outside the API and subject to change or removal without notice</em>. (In practice the method is three lines long and simple to re-implement.)</li>
</ul>
<h2 id="afterthoughts">Afterthoughts</h2>
<p>I have succeeded in bending JAXB to my use-case, but at what cost? If I use JAXB Annotated Case Classes for my domain model then I,</p>
<ul>
<li>have to suffer the heavy miasma of annotation pollution</li>
<li>am constrained to the use of <code class="language-plaintext highlighter-rouge">Arrays</code> to describe lists and have to re-implement a large part of each case class as a result</li>
<li>am at the mercy of any future Scala/Java interop bugs I find in the JAXB space (consider the map)</li>
</ul>
<p>On reflection, JAXB is not a good fit for Scala in most scenarios - especially ones where the specification is complex or could evolve. As I have nearly completely implemented the Atom spec it isn’t so bad, but I would think twice before using it again.</p>
Bending JAXB to the will of Scala (Part 1 of 2)2014-09-29T00:00:00+00:00http://caoilte.org/scala/2014/09/29/bending-jaxb-to-the-will-of-scala-1-of-2<p>Martin Krasser has written the definitive guide to <a href="http://krasserm.blogspot.co.uk/2012/02/using-jaxb-for-xml-and-json-apis-in.html">JAXB in Scala</a>. This blog post is the first in a two part series that bottoms out solutions for the corner cases that I ran into whilst appropriating all of his hard work.</p>
<p>This week I’ll focus on optionals. Next week I will cover lists.</p>
<!--more-->
<h2 id="background">Background</h2>
<p>The use-case that I am exploring in this series of blog posts was modelling the <a href="http://atomenabled.org/developers/syndication/">Atom Syndication Format</a> in appropriately typed Scala Case Classes that serialize to the corresponding XML as cleanly and concisely as possible. My (nearly complete) implementation is available <a href="https://github.com/caoilte/atomizer">here</a>.</p>
<p>I wanted to avoid scala-xml because of the (perhaps un-justified) bad press it gets. Jackson and json4s both have XML bindings, but they are not flexible enough to accurately map the complete Atom Specification. The only other well supported Scala xml library that I am aware of is <a href="http://scalaxb.org/">scalaxb</a>. I have used that before and it is very good. However,</p>
<ul>
<li>It relies on XSDs and although there are Atom XSDs <a href="https://www.google.co.uk/search?q=atom+xsd&oq=atom+xsd">out there</a>, none are official.</li>
<li>The Case Classes generated are not as friendly to use as hand rolled could be (as evidenced by <a href="https://gist.github.com/caoilte/aa7839935a29713a7a51#file-jaxb-in-scala-1_01-scala">these examples of Entries generated from different XSDs</a> I found).</li>
<li>Under the hood it is still using scala-xml</li>
</ul>
<p>Since none of the Scala libraries perfectly matched my requirements I decided (perhaps foolishly) to try falling back to the de-facto Java standard, <a href="https://jaxb.java.net/">JAXB</a>.</p>
<h2 id="basic-marshalling">Basic Marshalling</h2>
<p>Basic types (strings, nested types, un-parameterised custom types) are very simple to map once you understand how to map annotations correctly in Scala (and as Martin Krasser explains very well in <a href="http://krasserm.blogspot.co.uk/2012/02/using-jaxb-for-xml-and-json-apis-in.html">his blog post</a>).</p>
<h2 id="the-troubles-with-optionals">The Troubles with Optionals</h2>
<p>Martin also gives a great example of <a href="https://gist.github.com/krasserm/1891525#file-jaxb-02-scala">binding an optional string</a> to a case class but problems emerge when you actually use it.</p>
<h3 id="none-behaviour-on-strings">None Behaviour on Strings</h3>
<p>I expected,</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="nc">Person</span><span class="o">(</span><span class="s">"Martin Krasser"</span><span class="o">,</span> <span class="nc">None</span><span class="o">,</span> <span class="mi">30</span> <span class="cm">/* haha */</span><span class="o">)</span></code></pre></figure>
<p>to serialize to</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="cp"><?xml version="1.0" encoding="UTF-8" standalone="yes"?></span>
<span class="nt"><person></span>
<span class="nt"><name></span>Martin Krasser<span class="nt"></name></span>
<span class="nt"><age></span>30<span class="nt"></age></span>
<span class="nt"></person></span></code></pre></figure>
<p>ie to leave out the <code class="language-plaintext highlighter-rouge">Option[String]</code> username field because it was set to <code class="language-plaintext highlighter-rouge">None</code>. However, despite running JDK7u45 (Martin thought JDK7u4 and up should contain a fix for <a href="https://java.net/jira/browse/JAXB-415">this bug</a>) I still got a nasty <a href="https://gist.github.com/caoilte/aa7839935a29713a7a51#file-jaxb-in-scala-1_02-txt">NPE</a>. And then, even when I upgraded to the latest Reference Implementation version of JAXB (2.2.7) the case class would marshal to XML but left in an empty element for username despite the <code class="language-plaintext highlighter-rouge">XMLAdapter</code> specifically returning null.</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="cp"><?xml version="1.0" encoding="UTF-8" standalone="yes"?></span>
<span class="nt"><person></span>
<span class="nt"><name></span>Martin Krasser<span class="nt"></name></span>
<span class="nt"><username/></span>
<span class="nt"><age></span>30<span class="nt"></age></span>
<span class="nt"></person></span></code></pre></figure>
<p>An unneccessary empty <code class="language-plaintext highlighter-rouge"><username/></code> element breaks my requirement for <em>clean and concise xml</em>.</p>
<h3 id="optional-inner-value-only-types">Optional Inner Value-Only Types</h3>
<p>More serious problems emerged when I tried to map one very specific optional inner type. Consider the following code,</p>
<script src="https://gist.github.com/caoilte/aa7839935a29713a7a51.js?file=jaxb-in-scala-1_03.scala"> </script>
<p>It should marshal the following case class</p>
<figure class="highlight"><pre><code class="language-scala" data-lang="scala"><span class="nc">Person</span><span class="o">(</span><span class="s">"Caoilte O'Connor"</span><span class="o">,</span> <span class="nc">None</span><span class="o">)</span></code></pre></figure>
<p>to</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="cp"><?xml version="1.0" encoding="UTF-8" standalone="yes"?></span>
<span class="nt"><person></span>
<span class="nt"><name></span>Caoilte O'Connor<span class="nt"></name></span>
<span class="nt"></person></span></code></pre></figure>
<p>However, for me it <a href="https://gist.github.com/caoilte/aa7839935a29713a7a51#file-jaxb-in-scala-1_04-txt">NPEs</a> when using jdk7u45 (with or without adding the RI JAXB 2.2.7 as an explicit dependency). The problem is very specific and I was only able to reproduce it for types with one attribute that was <code class="language-plaintext highlighter-rouge">@xmlValue</code> annotated. (I have raised the bug as <a href="https://java.net/jira/browse/JAXB-1052">JAXB-1052</a>.)</p>
<h3 id="eclipselink-moxy--a-solution-for-optionals">EclipseLink MOXy : A Solution for Optionals</h3>
<p>The only solution I found for these problems was to switch JAXB implementation (as advised by Blaise Doughan in this <a href="http://stackoverflow.com/questions/11894193/jaxb-marshal-empty-string-to-null-globally/11931768#11931768">Stack Overflow question</a>). <a href="http://www.eclipse.org/eclipselink/moxy.php">EclipseLink MOXy</a> fixes both of the problems described above.</p>
<p>Switching was trivial. I just added a dependency on <code class="language-plaintext highlighter-rouge">"org.eclipse.persistence" % "org.eclipse.persistence.moxy" % "2.5.2"</code> and put the following <code class="language-plaintext highlighter-rouge">jaxb.properties</code> file</p>
<figure class="highlight"><pre><code class="language-properties" data-lang="properties"><span class="py">javax.xml.bind.context.factory</span><span class="p">=</span><span class="s">org.eclipse.persistence.jaxb.JAXBContextFactory</span></code></pre></figure>
<p>under the same package as the case classes that I was marshalling, except in the resources directory.</p>
<p>As an added bonus, MOXy also doesn’t print the <code class="language-plaintext highlighter-rouge">standalone</code> attribute in the xml fragment, which was breaking my requirement for <em>clean and concise xml</em> and which Metro (the Reference Implementation) cannot be configured to remove. My person case class would now marshal as,</p>
<figure class="highlight"><pre><code class="language-xml" data-lang="xml"><span class="cp"><?xml version="1.0" encoding="UTF-8"?></span>
<span class="nt"><person></span>
<span class="nt"><name></span>Caoilte O'Connor<span class="nt"></name></span>
<span class="nt"></person></span></code></pre></figure>
<p>Perfectly concise!</p>
<h3 id="mapping-other-peoples-types-to-optionals">Mapping Other People’s Types to Optionals</h3>
<p>The previous two examples describe the problems I found with using <code class="language-plaintext highlighter-rouge">OptionAdapter</code> to map a type that JAXB knows how to marshal/unmarshal. Taking a custom type that JAXB doesn’t know how to manage (ie one you have already created your own <code class="language-plaintext highlighter-rouge">XmlAdapter</code> for) and making it optional adds a whole new dimension to the problem space. You cannot simply pass your custom type as a type parameter to the <code class="language-plaintext highlighter-rouge">OptionAdapter</code> and let JAXB figure out the chain of transformations required (although I did try). I came up with the following solution,</p>
<script src="https://gist.github.com/caoilte/aa7839935a29713a7a51.js?file=jaxb-in-scala-1_05.scala"> </script>
<p>There’s nothing new, or even particularly pleasing about this code, but it is interesting because it was the most concise way that I was able to take an existing <code class="language-plaintext highlighter-rouge">XmlAdapter</code> (<code class="language-plaintext highlighter-rouge">DateTimeAdapter</code>) and make it optional.</p>
<p>I have tested that the <em>bug</em> described in the JavaDoc of <code class="language-plaintext highlighter-rouge">CustomOptionAdapter</code> has been fixed in the trunk of EclipseLink MOXy.</p>
<h2 id="afterthoughts">Afterthoughts</h2>
<p>I decided to write about JAXB in Scala because I wanted to highlight the problems that emerge when you scratch the surface with a real life use-case. That doesn’t mean I endorse the use of JAXB in Scala. I’m still very ambivalent about the library. Exactly why might become clearer in part two of this two part series when I scratch a little deeper on the problems for JAXB in Scala and talk about lists.</p>