Posts

scalaxb 0.5.2

  • Fixes <xs:sequence> handling. (GH-19)
  • Fixes <xs:attributeGroup> handling. (GH-21)
  • Improves error message when a referenced schema or referenced component is missing.
  • Searches for missing schema in the current directory. (GH-23)

navigating XML Schema

This is a memo to navigate around the case classes for XML Schema. Besides the derivation by extension that programming languages have as inheritance, XML Schema also allows derivation by restriction. Since Scala I don’t think expresses this easily, some types are wider than how they are specified in the schema.

XElement

  • known subclasses: XTopLevelElement, XLocalElementable, XNarrowMaxMin, XLocalElement
  • value member: xelementoption : Option[DataRecord[XElementOption]]
  • value member: name: Option[String]
  • value member: ref : Option[QName]
  • value member: typeValue : Option[QName]

XSimpleType

  • known subclasses: XTopLevelSimpleType, XLocalSimpleType

  • value member: arg1 : DataRecord[XSimpleDerivationOption]

  • XUnion

    • value member: memberTypes : Option[Seq[QName]]

  • XRestriction

    • value member: arg1 : XSimpleRestrictionModelSequence
    • XSimpleRestrictionModelSequence
      • value member: arg2 : Seq[DataRecord[XFacetsOption]]
        • known subclasses: XFacetable, XWhiteSpace, XNumFacetable, XTotalDigits, XNoFixedFacetable, XPattern, XNumFacet, XNoFixedFacet, XFacet. At runtime, XNoFixedFacetable will be parsed as XNoFixedFacet.
Read more

typeclass-based XML data binding

Ultimately, the users of scalaxb are interested the real problems that the entity objects express, not how they persist into XML. That’s why I knew I eventually had to vacate the singleton/companion object of the case class to implement the data binding. Until recently it has been generating the data binding implementation as follows:

object Address extends rt.ElemNameParser[Address] {
  val targetNamespace = "http://www.example.com/IPO"
 
  def parser(node: scala.xml.Node): Parser[Address] =
    ...
 
  def toXML(__obj: Address, __namespace: String, __elementLabel: String, __scope: scala.xml.NamespaceBinding): scala.xml.NodeSeq =
    ...
}

Now scalaxb has hijacked the prime real estate for the purpose of XML data binding, which has little to with Address.

Read more

scalaxb 0.5.1

  • Command line program uses Scala 2.8.1.
  • Fixes accessors to compositors by using type name as the param name. (GH-18)
  • Fixes generated type of nillable varargs. (also GH-18)

slides for 2010-11-29 ny-scala

I presented a 5-min lightening talk at 2010-11-29 New York Scala Enthusiasts meetup with 8 others. Here’s the pdf of the slides. The presentation fell apart in the middle because the sample code text was too dark over the black background (corrected in pdf). Nonetheless it was inspiring to get live feedback and see what other hackers are up to.

scalaxb 0.5.0

  • Generates typeclass-based XML data binding code in a separate file xmlprotocol.scala.
  • Renames runtime package name from org.scalaxb.rt to scalaxb and the helper file to scalaxb.scala.
  • Fixes handling of <xs:all> containing >22 items (GH-10).
  • Fixes handling of <xs:complexType> containing >22 attribtues (GH-11).
  • Fixes handling of <xs:complexType> with <xs:simpleContent>, which restricts a mixed <xs:complexContent> (GH-12).
  • Fixes output instability (GH-13).
  • Fixes handling of circular <xs:import> by consolidating the XML databinding code into a single trait (GH-14).
  • Adds warning when <xs:include> item is missing from compilation (GH-15).

I’d like to thank everyone who gave me feedback and reported issues. Special thanks go to Debasish Ghosh for his inspirational articles and Andreas Sewe for reporting all of the above bugs.

Read more

proguard

Since I’ve heard several issues installing scalaxb via sbaz, which I think is attributed to Scala version incompatibility, I’ve decided to include scala-library into scalaxb.jar using sbt-proguard-plugin.

--class-prefix and --param-prefix

I added two command line options --class-prefix and --param-prefix to add prefixes to the generated code. It shouldn’t be needed if the schema uses lower case for element names like many of the well-known schemas (slap me if you catch me say schemata or octopodes). This is because scalaxb capitalizes the class name, but leaves the params alone, so you’d end up with code like the following:

case class USAddress(name: String,
  street: String,
  city: String,
  usstate: USState,
  zip: Int) extends Addressable

Suppose the element names were also capitalized in the schema, then it would produce something like this:

Read more

narrower <choice>

rt.DataRecord came out of the necessity to fulfill two requirements. The first is to convert XML document into a native object so it could be consumed easier (also known as data binding); the second is to retain the ability to convert the native object back into the XML document (also known as round trip). The two requirements are sometimes not aligned with each other. For example, if I am just worried about getting the round trip done, I can just hold on to the scala.xml.Node and use that to write XML out, but it’s not very useful from the data binding perspective.

The first goal of the scalaxb was to cover the full range of XML Schema and implementing the round trip. Now that the goal is mostly fulfilled, my recent updates were aimed to improve the usability of generated code while still maintaining the roundtripability. One area where the pendulum has shifted too much towards round trip is the code generated for <choice>. I’d like to review the history of code generation for <choice> and propose a new solution at the end.

Read more

<any> revisited

In order for a DataRecord to not lose the freight during the round trip of <any>, I’ve been storing the scala.xml.Elem object within it. This is convenient for the round trip, but not for consuming the DataRecord. The problem with <any> is that it could be anything, and I wouldn’t know how to parse them. At least when they are not built-in types.

Now that mixed contents have been cleaned up, it seemed like a good time to start parsing XSD built-in types too. Here’s how it looks like:

Read more

mixed content revisited

scalaxb added support for mixed contents a while back. When <xs:complexType mixed="true">, text nodes are placed in conjunction with the subelements of the complex type, like XHTML. Since I implemented it, it’s been bothering me that the generated case class is not DRY.

For example,

<xs:element name="mixedTest">
  <xs:complexType mixed="true">
    <xs:choice maxOccurs="unbounded">
      <xs:element name="billTo" type="Address"/>
      <xs:any namespace="##other" processContents="lax" />
    </xs:choice>
  </xs:complexType>
</xs:element>

would generate case class Element3(arg1: Seq[rt.DataRecord[Any]], mixed: Seq[rt.DataRecord[Any]]). The first parameter arg1 contains the subelements; and the second mixed contains both the subelements and the text node. In order for the case class to round trip back to XML, it needed to store both text node and subelements in order in mixed; however because parsing was performed only for subelements, only arg1 would contain Address case class and mixed would contain unparsed scala.xml.Elem instance. Not very nice. All this was partially due to the fact the parsing logic was not very sophisticated.

Read more

working around Tuple22 limitation

Earlier this year I wrote about hitting some scala limitations:

Scala supports only up to Tuple22.

This means that Scala can define case classes only up to 22 parameters. I’ve implemented workarounds to use schema that includes complex types that has more than 22 particles within the sequence.

First, when a sequence contains more than MaxParticleSize (20) particles, it will split up the sequence into chunks of ChunkParticleSize (10). For example, a sequence that has 30 elements would be split into three chunks of 10 elements:

Read more

substitution group

XML Schema Part 0: Primer:

XML Schema provides a mechanism, called substitution groups, that allows elements to be substituted for other elements. More specifically, elements can be assigned to a special group of elements that are said to be substitutable for a particular named element called the head element. (Note that the head element as well as the substitutable elementsmust be declared as global elements.)

The following demonstrates substitution group:

Read more

<enumeration>

XML Schema Part 0: Primer:

The enumeration facet limits a simple type to a set of distinct values. For example, we can use the enumeration facet to define a new simple type called USState, derived from string, whose value must be one of the standard US state abbreviations:

<xsd:simpleType name="USState">
  <xsd:restriction base="xsd:string">
    <xsd:enumeration value="AK"/>
    <xsd:enumeration value="AL"/>
    <xsd:enumeration value="AR"/>
    <!-- and so on ... -->
  </xsd:restriction>
</xsd:simpleType>

We can use case objects to express these restrictions. Here’s what scalaxb generates:

Read more

scalaxb-appengine

scalaxb-appengine is a RESTful API to run scalaxb over the web. It’s implemented using n8han/Unfiltered and the full source is available on eed3si9n/scalaxb.

compile

URL:
http://scalaxb.appspot.com/compile/_{output}.{format}_
Formats:
scala, zip
HTTP method:
POST
HTTP encoding:
multipart/form-data
Parameters:
  • arg: Optional, multiple allowed. URL of the schema file.
  • argf: Optional, multiple allowed. Uploaded schema file.
  • defaultPackageName: Optional. Name of the target package.
  • namespaceURI: Optional, multiple allowed. URI of namespace to be packaged.
  • packageName: Optional, multiple allowed. Name of the target package for the namespaceURI.
  • classPrefix: Optional. Prefixes generated class names.
  • paramPrefix: Optional. Prefixes generated parameter names.
  • wrapContents: Optional, multiple allowed. Wraps inner contents into a seperate case class.
Notes:
  • At least one arg or argf must be present.
  • The n-th packageName specifies the target package for the n-th namespaceURI.
Read more

scalaxb 0.2.0

  • Implements support for <group>, <attributeGroup>, and <all>.
  • Fixed round trip of complex types containing sequences.
  • Generates toXML into the companion objects to simplify the case class.

For example,

<xs:complexType name="Address">
  <xs:sequence>
    <xs:element name="name"   type="xs:string"/>
    <xs:element name="street" type="xs:string"/>
    <xs:element name="city"   type="xs:string"/>
  </xs:sequence>
</xs:complexType>

turns to

case class Address(name: String,
  street: String,
  city: String)

<group> and <attributeGroup>

From XML Schema Part 1:

The XML representation for a model group definition schema component is a <group> element information item. It provides for naming a model group for use by reference in the XML representation of complex type definitions and model groups.

The <group> construct allows the schema designers to reuse part of the content model without resorting to type derivation. This comes in handy when dealing with complicated schema. Suppose we have a group named head.misc and an element named head.

Read more

scalaxb 0.1.0

The following code is an example of a generated parser:

object Address extends rt.ElemNameParser[Address] {
  val targetNamespace = "http://www.example.com/IPO"

  def parser(node: scala.xml.Node): Parser[Address] =
    (rt.ElemName(targetNamespace, "name")) ~ 
      (rt.ElemName(targetNamespace, "street")) ~ 
      (rt.ElemName(targetNamespace, "city")) ^^
        { case p1 ~ 
      p2 ~ 
      p3 => Address(p1.text,
      p2.text,
      p3.text) }
}

xsi:nillable

In XML, one can omit an element to express lack of value or use an empty element.

Sometimes it is desirable to represent an unshipped item, unknown information, or inapplicable information explicitly with an element, rather than by an absent element.

The trouble with using an empty element <foo></foo> is that the emptiness would no longer matches the specified type like xs:positiveInteger. It is possible to form a xs:union of xs:positiveInteger and an xs:enumeration with only empty string in it to allow either poistive integers or empty string. However, technically speaking, an empty string is different from pure emptiness. In terms of code, it’s the difference between null and "", or in Scala, None and Some("").

XML Schema resolves this issue by introducing a special attribute called xsi:nil. By writing

<price xsi:nil="true" />
Read more

parsing with parser combinators

I’ve known the limitation of hand parsing for a while. Parsing that relies on token positions quickly gets out of hand when there are more complex grammars like repetitions, options, and choices of sequences. At some point, I decided to use scala’s parser combinators to do the parsing of content types, but it’s been a long way to implement it.

First let’s look at a real-life example of such complex structure:

Read more

mixed content

Added support for mixed contents. in which text nodes are placed in conjunction with subelements within an element. Similar to the way I handled <any>, text nodes are placed in DataRecord[String] object along with other DataRecord objects under a member value called mixed.

Suppose we have a schema that looks like this:

<element name="Element3">
  <complexType mixed="true">
    <choice maxOccurs="unbounded">
      <element ref="ipo:Choice1"/>
      <element ref="ipo:Choice2"/>
      <any namespace="##other" processContents="lax" />
    </choice>
  </complexType>
</element>

The usage code from the unit test looks like this:

Read more

<any> and <anyAttribute>

Last time I wrote:

Next goal is to handle <any> without ignoring it completely using this generic container. I can probably store the scala.xml.Elem object “as is” in a collection. The user of round trip probably would expect that I don’t lose luggage during the flight.

That’s pretty much what I did for two usages of <any> handled by scalaxb. First pattern is that it could appear as part of the sequence.

Read more

round trip

I started implementation of round trip: xml document -> scala object -> back to xml document. Currently done with elements and attributes tracked in the case class. Suppose you have the following document:

val subject = <shipTo xmlns="http://www.example.com/IPO"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:ipo="http://www.example.com/IPO"
    xsi:type="ipo:USAddress">
  <name>Foo</name>
  <street>1537 Paper Street</street>
  <city>Wilmington</city>
  <state>DE</state>
  <zip>19808</zip>
</shipTo>

I can turn this into object by calling fromXML():

val obj = Addressable.fromXML(subject)

Then turn it back into xml document by calling toXML():

Read more

per-namespace package name

You can now generate classes under different packages per-namespace.

  -p:<namespaceURI>=<package> | --package:<namespaceURI>=<package>
        specifies the target package for <namespaceURI>

For example XML Signature and SAML 2.0 assertion probably should be under different package name:

$ scalaxb -p:http://www.w3.org/2000/09/xmldsig#=org.w3.xmldsig -p:http://www.w3.org/2001/04/xmlenc#=org.w3.xmlenc -p:urn:oasis:names:tc:SAML:2.0:assertion=org.xml.saml2.assertion -p:urn:oasis:names:tc:SAML:2.0:metadata=org.xml.saml2.metadata saml-schema-metadata-2.0.xsd saml-schema-assertion-2.0.xsd xenc-schema.xsd xmldsig-core-schema.xsd 
generated ./xmldsig-core-schema.scala
generated ./xenc-schema.scala
generated ./saml-schema-assertion-2.scala
generated ./saml-schema-metadata-2.scala
$ scalac xmldsig-core-schema.scala xenc-schema.scala saml-schema-assertion-2.scala saml-schema-metadata-2.scala Helper.scala

I’ve been thinking about this feature for a while. At least a week, since I implemented keyValueOpt in scopt a week ago. With this change, Helper object is now pushed out to its own namespace org.scalaxb.rt and file Helper.scala.

xml namespace

I’ve been working on Issue 1 submitted by Tsuresh Kumar for a while now. The idea is to compile the schema for SAML Metadata, which is saml-schema-metadata-2.0.xsd. I need to implement a few things so scalaxb can handle the schema.

The biggest issue is the handling of multiple schema documents. Real life schema like SAML Metadata defines the schema on top of existing schema documents/namespaces. The current implementation of scalaxb assumes that the given schema contains everything, and thus the namespace handling was sloppy. Now that it needs to handle multiple namespaces other than the targetNamespace and XSD namespace, I cleaned up the code to convert the prefixes into URI and keep them around in the ElemRef and AttributeRef object.

Read more