Processing FHIR Bundles using HAPI

A month or so back, we talked about a project we had to import glucose results into a repository using FHIR. That post was focused on using the transaction search facility to indicate that there was a resource in the bundle that may or may not exist on the server, and giving the search parameters for the server to use to make the determination. The example we looked at was the Patient resource – specifying the identifier value to use as the lookup.

We didn’t really talk about the other aspects of processing a bundle such as this, so let’s see how we can implement this scenario using the HAPI library.

So what are our requirements?

Well, there will be a glucose meter that the patient uses to actually take the measurements. It communicates via a USB connection with a desktop application that can extract the results, and will be able to transmit them to the repository.

We want to store the data in structured form in our repository so that we can display them to clinical and other users, and use them as the basis of decision support. Ideally this should be done in real time (though we want to preserve the ability to use a message based paradigm in the future).

We have a few options for the way that the client application (let’s say it’s a rich desktop application) can update the repository.

1. We could first query the repository for the Patient and Device, adding them if necessary. Then for each glucose result we create and then POST an observation with appropriate references to Patient and Device.
2. We could create a bundle containing a ‘candidate’ Patient and Device resource, using the transaction search capability to create a real resource if necessary. We’d also have an Observation resource for each glucose result, referencing the patient as the Observation.subject and the Device as the Observation.performer. Then, we set up a custom service on our repository (with a url like: [FHIR]/services/glucoseprocessor) that knows how to process the bundle.
3. We could do the same as #2, except we send it to the mailbox So that the mailbox knows what to do with the message, we’d add a MessageHeader resource to the bundle, using the MessageHeader.event property to indicate what the bundle was. The mailbox would then hand over the bundle to an appropriate routine for processing.
4. Another variant on #2 would be to send the bundle to the transaction endpoint (the root of the server), and use a Profile to define the contents of the bundle. The transaction process would recognize the profile, and submit to a routine for processing, which could include specific validation as well as resource updating.

The first one sounds like a bit too much work for the client, as well as it having to manage any transactional issues – the other 3 all have a single call, which is attractive, so lets choose one of those.

The problem with a custom service (#2) is just that – it’s custom. FHIR doesn’t (yet) define a common way to define and describe these services, so it’s necessary that there is ‘out of band’ negotiation between sender & recipient.

The mailbox option (#3) would certainly work – except that we need to add a MessageHeader resource – and define a custom event. Nothing wrong with that – but also implies some out of band negotiation between sender & recipient

POSTing to the root (#4) is attractive –provided that the server knows how to perform the ‘search processing’ (i.e. recognize the search link for a resource in the bundle and process accordingly. We could possibly use a profile so that the server can validate the bundle first.

However, generic transaction processing is hard. Not only do you need to do the ‘ID re-write’ (if you support it) but you need to manage deletions in the bundle as well. Plus, there are other subtleties. If would be nice if you could use a profile to limit the responsibilities of the transaction processing – eg if the processor could recognize a particular profile, and use that for specific processing rather than needing to implement all the functionality – but I’m not sure whether that’s legal in FHIR.

Whichever we go with, there’s one thing we do have to remember – and that is the requirement to return a bundle containing the resources, and the ID’s that they were assigned during processing. This is a requirement when POSTING to the root – it’s a SHALL – but we ought to do the same with the other options.

For this exercise we’ll choose option 2 – Creating a specific service. We’ll wrap the specific processing code into a separate class so we could re-use it with a mailbox approach (hopefully that’s on HAPI’s radar to implement).

Here’s the class:

public class GlucoseBundleProcessor {

    private MyMongo _myMongo;   //the database helper class

    public GlucoseBundleProcessor(MyMongo myMongo){
        _myMongo = myMongo;
    }

    //process with a Bundle...
    public List<IResource> processGlucoseBundle(Bundle bundle) {
        //generate a list of resources...
        List<IResource> theResources = new ArrayList<IResource>();    //list of resources in bundle
        for (BundleEntry entry : bundle.getEntries()) {
            theResources.add(entry.getResource());
        }
        return this.process(theResources);
    }


    //process with a List of resources
    public List<IResource> processGlucoseUploads(List<IResource> theResources) {
        return this.process(theResources);
    }


    //process a bundle of glucose results, adding them to the repository...
    private List<IResource> process(List<IResource> theResources) {
        Patient patient = null;     //this will be the patient resource. There should only be one....
        Device device = null;       //this will be the device resource. There should only be one....
        List<IResource> insertList = new ArrayList<IResource>();    //list of resource to insert...

        //First pass: assign all the CID:'s to a new ID. For more complex scenarios, we'd keep track of
        //the changes, but for this profile, we don't need to...
        //Note that this processing is highly specific to this profile !!! (For example, we ignore resources we don't expect to see)
        for (IResource resource : theResources) {
            String currentID = resource.getId().getValue();
            if (currentID.substring(0,4).equals("cid:")) {
                //Obviouslym the base URL should not be hard coded...
                String newID = "http://myUrl/" + java.util.UUID.randomUUID().toString();
                resource.setId(new IdDt(newID));    //and here's the new URL
            }

            //if this resource is a patient or device, then set the appropriate objects. We'll use these to set
            // the references in the Observations in the second pass. In real life we'd want to be sure there is only one of each...
            if (resource instanceof Patient) {
                patient = (Patient) resource;
                //we need to see if there's already a patient with this identifier. If there is - and there is one,
                //then we use that Patient rather than adding a new one.
                // This could be triggered by a 'rel=search' link on the bundle entry in a generic routine...
                IdentifierDt identifier = patient.getIdentifier().size() >0 ? patient.getIdentifier().get(0) : null;
                if (identifier != null) {
                    List<IResource> lst = _myMongo.findResourcesByIdentifier("Patient",identifier);
                    if (lst.size() == 1) {
                        //there is a single patient with that identifier...
                        patient = (Patient) lst.get(0);
                        resource.setId(patient.getId());    //set the identifier in the list. We need to return this...
                    } else if (lst.size() > 1) {
                        //here is where we ought to raise an error - we cannot know which one to use.
                    } else {
                        //if there isn't a single resource with this identifier, we need to add a new one
                        insertList.add(patient);
                    }
                } else {
                    insertList.add(patient);
                }
            }

            //look up a Device in the same way as as for a Patient
            if (resource instanceof Device) {
                device = (Device) resource;
                IdentifierDt identifier = device.getIdentifier().size() >0 ? device.getIdentifier().get(0) : null;
                if (identifier != null) {
                    List<IResource> lst = _myMongo.findResourcesByIdentifier("Device", identifier);
                    if (lst.size() == 1) {
                        device = (Device) lst.get(0);
                        resource.setId(device.getId());    //set the identifier in the list. We need to retuen this...
                    } else {
                        insertList.add(device);
                    }
                } else {
                    insertList.add(device);
                }
            }

            if (resource instanceof Observation) {
                //we always add observations...
                insertList.add(resource);
            }
        }

        //Second Pass: Now we re-set all the resource references. This is very crude, and rather manual.
        // We also really ought to make sure that the patient and the device have been set.....
        for (IResource resource : theResources) {
            if (resource instanceof Observation) {
                Observation obs = (Observation) resource;

                //this will be the correct ID - either a new one from the bundle, or a pre-existing one...
                obs.setSubject(new ResourceReferenceDt(patient.getId()));

                //set the performer - there can be more than one in the spec, hence a list...
                List<ResourceReferenceDt> lstReferences = new ArrayList<ResourceReferenceDt>();
                lstReferences.add(new ResourceReferenceDt(device.getId()));
                obs.setPerformer(lstReferences);
            }
        }

        //Last pass - write out the resources
        for (IResource resource : insertList) {
            _myMongo.saveResource(resource);
        }

        //we return the bundle with the updated resourceID's - as per the spec...
        return theResources;
    }
}

some notes:

• The GlucoseBundleProcessor iterates through the resources, updating the cid: (Content-ID) references, and checking to see if the Patient & Device resources exist (based on their identifier – if it exists). According to the spec, if there is more than one matching resource then we really ought to reject the bundle, which we’re not doing at the moment.
• It’s completely specific to this use case – for example it ignores any resources it isn’t expecting. Obviously this is not appropriate in a production environment! There are likely better designs than ‘instanceOf’ statements, but it’ll do for now.
• We’re using a mongo database to store the resources. There’s a helper class that is instantiated when the servlet engine starts up and has some simple methods to save to and query from the database.
• The FHIR context and MongoDb helper classes are created when the server starts up and attached to the Servlet context

Now, for the code that processes the incoming bundle. This is a standard Java Servlet with a specific end point that maps to a URL.

Here’s the code:

@WebServlet(urlPatterns= {"/fhir/service/glucoseprocessor"}, displayName="Process Glucose bundle")
public class GlucoseResultServlet extends HttpServlet {

    private MyMongo _myMongo;
    private FhirContext _fhirContext;

    @Override
    public void init(ServletConfig config) throws ServletException {
        //get the 'global' resources from the servlet context
        ServletContext ctx = config.getServletContext();
        _myMongo =  (MyMongo) ctx.getAttribute("mymongo");
        _fhirContext = (FhirContext) ctx.getAttribute("fhircontext");
    }

    protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        PrintWriter out = response.getWriter();
        //first parse into a fhir bundle (need to check the mime type here. Should also check for _format paramters as well)
        //if we have to to this a lot, then a separate utility class would be good...

        Bundle bundle = null;
        IParser parser = null;
        String ct =  request.getHeader("content-type");
        if (ct.equals("application/xml+fhir")){
            parser = _fhirContext.newXmlParser();
            bundle = parser.parseBundle(request.getReader());
        } else if (ct.equals("application/json+fhir")){
            parser = _fhirContext.newJsonParser();
            bundle = parser.parseBundle(request.getReader());
        } else {
            OperationOutcome operationOutcome = new OperationOutcome();
            operationOutcome.getText().setDiv("Invalid content-type header: " + ct);
            parser = _fhirContext.newXmlParser();
            response.setStatus(415);    //unsupported media type
            out.println(parser.encodeResourceToString(operationOutcome));
            return;
        }

        //now we have a bundle we can process it...
        GlucoseBundleProcessor glucoseBundleProcessor = new GlucoseBundleProcessor(_myMongo);
        //process the bundle, and get back the list of resources with updated ID's...
        List<IResource> resources = glucoseBundleProcessor.processGlucoseBundle(bundle);
        Bundle newBundle = new Bundle();
        newBundle.getTitle().setValue("Processed Glucose results");
        newBundle.setId(new IdDt(java.util.UUID.randomUUID().toString()));
        newBundle.setPublished(new InstantDt());
        for (IResource resource : resources) {
            newBundle.addResource(resource,_fhirContext,"http://localServer/fhir");
        }

        response.setStatus(201);    //created
        out.println(parser.encodeBundleToString(newBundle));

    }



}

The code is pretty well documented, and is certainly not intended to be production level code – just to show concepts. There’s stuff we had to do manually (like choosing the correct HAPI parser based on HTTP headers) that HAPI does automatically elsewhere – makes you appreciate the heavy lifting that it does!

One thing we’ve not looked at is security. But – given that we’ve implemented OAuth2 as part of our SMART development, all we need to do is to ‘protect’ the root endpoint by requiring a valid Access Token to use the endpoint. Then, the client simply logs in before submitting a bundle.

So that’s a simple app to process a specific type of bundle. Bundles are a very useful mechanism in FHIR to manage complexity (especially transactional complexity) and reduce the number of REST calls required when there are multiple resources to create or update.

However, generic processing of bundles is quite complex – and specific use case processing will often be much simpler.

As with many things in FHIR, there’s more than one way to achieve a specific objective, so choose the one that best meets the requirements.