Blog

Creating and managing OCDS extensions

16 Jul 2018

By Tim Davies

This post in our technical series explains how to create an OCDS extension.

One of the major developments in version 1.1 of the Open Contracting Data Standard (OCDS) is full support for schema extensions. Extensions describe additional fields of data beyond those covered by the core standard and can be useful for documentation and validation of advanced data that helps users to understand contracting processes in more detail. 

The structure of extensions

Each extension consists of:

  • Documentation explaining the use case for the additional fields, and how they should be interpreted;
  • A JSON merge-patch against the schema introducing the technical definition of new fields; and/or
  • New or updated codelists; and
  • An extension definition (extension.json) file that describes the extension.

You can see the standard extension template here, and a range of example extensions here.

Using extensions

The documentation for each extension will explain the fields that it introduces and provide definitions for each. Publishers can simply include these fields in their OCDS files, and then declare that they are using the extension, in order for the data to be validated.

To declare the use of an extension publishers should include a link to the ‘extension.json’ file in the extensions array of a release or record package file.

For example, a publisher using the lots (core) extension because their tenders involve lots, and the community documentation extension which supports page numbers for linked documents, would start their release package like this:

{
    "version":"1.1",
    "extensions":[
       "https://raw.githubusercontent.com/open-contracting /ocds_lots_extension/v1.1/extension.json",
       "https://raw.githubusercontent.com/open-contracting /ocds_documentation_extension/master/extension.json"
    ],
    "releases":["..."]
}

This information is understood by the official OCDS validator, which will:

  • Display links to documentation for each extension – supporting users who want to interpret the data;
  • Fetch and apply each extension to the OCDS schema – checking that extensions are themselves valid in the process, and providing an extended schema for download for use in local validation workflows.
  • Validate supplied data against the extended schema – allowing publishers to check that their extended data has been correctly structured.

Third-party tools can also use the extensions array in a release or record package to detect when incoming data includes specific additional features, such as lots or locations, and can adapt how they process the data accordingly.

Developing an extension

To develop an extension you will need an understanding of how to write JSON Schema and an understanding of the underlying structure of the OCDS Schema. Extensions consist of fragments of JSON Schema which are merged into the main OCDS schema files to create an extended schema.

Example

The partyScale extension adds a simple ‘scale’ property to the Organization object, which is used to define entries in the parties array. In the partyScale repository there is a README.md file that defines the behaviour of the extension, and a release-schema.json file that provides JSON schema for the additional fields as below:

{
  "definitions": {
    "Organization": {
      "properties": {
        "details":{
            "properties":{
                "scale":{
                    "title":"Scale",
                    "description":"For commercial organization's, is this a micro (micro), Small or Medium Enterprise (sme) or large (large) entity according to the definitions used by the procuring entity or buyer. This field can be left blank if no such concepts apply.",
                    "type":["string","null"],
                    "enum": ["micro","sme","large",""]
                }
            }
        }
      }
    }
  }
}

Note that, as this is provided using the JSON merge patch model, we include all the parent hierarchy, even though we are only introducing ‘scale’, so that any tool applying the patch knows where to place this new entry.

Process

The process to develop an extension starts with discussion before moving to an iterative process of development, documentation, and demonstration.

Step 1: Discuss

After checking the current list of extensions and discovering your need is not met by one of these, you should search open and closed issues in the OCDS Standard issue tracker to make sure no one is already working on a related extension. If they are, consider getting in touch with them to collaborate. If they are not, create a new issue to describe the extension you plan to create, setting out the need that it meets, and giving as much information as you can about your current plans for modeling the data. A few tips to create a good issue are:

A description of the process you are trying to represent. Avoid too much detail about the data: describe the workflow and the information that people know at particular points in time. A diagram can help here.

Examples of existing data. Ideally, find two or three different examples of how this data is currently represented, to help with data modeling.

What belongs in OCDS, and what could belong in secondary datasets. For example, some budget and corporate information is better represented in separate datasets, as they are independent of individual contracting processes.

A description of the use cases for this data. In particular: Is there a strong use case for comparing between countries? Is the main use case creating common interfaces? This will inform the balance between interoperable abstraction versus direct representation of a publisher’s data.

For example, if three countries have similar but not identical processes, we may prefer an abstracted model that can represent all three, and which would provide enough information that the same interface could display ‘good enough’ information for each (i.e. interfaces wouldn’t need major customization to display the data correctly).

On the other hand, if the countries have very different processes, we should assume that any interfaces or analyses will need to be customized for each country. In such cases, we want to be careful with abstraction, because it adds complexity for the publisher and the user to convert between reality and the abstracted model.

Throughout the development of your extension, keep this issue updated. This provides an opportunity for others with an interest in the same or similar data to get involved in shaping the chosen approach to data modeling.

Step 2: Develop

To develop your extension, you will first need to decide on field names and structures.

It can be useful to start by mocking up an example of the data you are creating an extension for in JSON format. You can then consider:

  • How would different data fit into this structure?
  • Are the field names intuitive for users?
  • Is the structure easy for applications to consume?

At this stage, the more different examples of source data, and different use cases for the data you can consider, the better.

Once you have settled on a model, ideally with community feedback, you should then create the schema JSON merge patch. You can do this using the extension creator, or manually using the standard extension template.

There are also a number of OCDS schema conventions to consider:

  • If the extension provides an array of objects, each object must include an id property.
  • Use ‘Capitalised’ class names in the ‘definitions’ section.
  • Use camelCase for all property names.
  • English language field names should also be used for core and community extensions.

(The last two conventions can be relaxed for local extensions in cases where existing local properties are being inserted into OCDS data).

You will also need to find somewhere to host your extension online at a stable URL. GitHub provides a good option for this, as it also offers version control and issue tracking for your extension. Create a repository for your extension.

Step 3: Document

In parallel with the development of the extension schema, you will need to write a README file, and/or additional documentation.

A readme should consist of:

  • An introduction providing a short description of the motivation for the extension;
  • Definitions for all fields, matching the definitions provided in the schema itself;
  • Examples of data that could be provided using the extension.

Step 4: Demonstrate

Before placing an extension in the extension registry, it is important to test it out.

Create a full example data file containing extended data, and submit it to the validator.

Extensions do not need to be in the registry before the validator will recognize them.

Tools

The OCDS extension creator simplifies the process of creating a JSON merge patch and standard extension folder structure.

It provides three schema views:

  • Source – where you can select the base schema you are extending.
  • Target – where you can edit the schema to the updated version that should result from your extension. Select ‘Copy from source above’ to populate this with the starting schema you have selected.
  • Patch – which will show the difference between the source schema, and your update, providing the JSON merge patch. After editing ‘target’, click ‘Generate patch’ to see this in action.

The target view is editable via a tree structure or a code editor.

If you need a head start with working on what the schema for a particular fragment of JSON might look like, you may also find https://jsonschema.net/ provides some useful tools for generating template schema from example JSON.

Registering an extension

There are three kinds of extension:

  • Core extensions describe recommended fields. If it is relevant and feasible to provide the information described by a core extension (such as location or bid information), publishers are encouraged to do so. These extensions are kept stable by being versioned along with the standard, and going through the standard governance process. This means they can only be introduced or updated with a new version of the standard.
  • Community extensions describe common approaches to publishing optional fields. These may have been developed to model a common set of additional fields that publishers have, or to describe data required by a particular use case. Community extensions can be introduced and included in the extension registry at any time, allowing ongoing growth of the standard between version updates.
  • Local extensions describe fields required by a single publisher or context and are used to document and validate the extra data that a publisher is including. Local extensions are not included in the extension registry.

Details of how to register an extension can be found in the Extension Registry GitHub documentation.