2. Create the FHIR Service

Generate the Service

For this tutorial, you will create a new FHIR service with the artifactId fhir-starter-service. The mvn archetype:generate command below generates the service from our FHIR archetype. Run it from the terminal in the root of your starter-project directory, on the branch you created:

mvn org.apache.maven.plugins:maven-archetype-plugin:3.4.1:generate \
   -DarchetypeGroupId=gov.va.mobile.archetypes \
   -DarchetypeArtifactId=fhir-module-archetype \
   -DarchetypeVersion=3.0.4 \
   -DgroupId=gov.va.mobile.service \
   -DartifactId=fhir-starter-service \
   -Dversion=1.0.0 \
   -Dshort-name=fhirstarter

The short-name property is required. It must start with a lowercase letter and contain only lowercase letters and digits, because the archetype uses it to build:

  • the Java package, gov.va.mobile.fhirstarter.v1

  • the service base path (context path), /fhirstarter/v1

  • the prefix of the service’s own properties, va.mobile.fhirstarter

The archetype also puts the FHIR server under /fhir (the fhir-server-path property), so the FHIR base URL of your service is /fhirstarter/v1/fhir. The dtr-org and jira-project properties default to ckm and CKM, which are right for the starter project.

Maven then asks for the remaining properties. Press Enter to accept each default, and enter Y at the end to confirm. Below is an example of the output:

[INFO] Using property: short-name = fhirstarter
[INFO] Using property: dtr-org = ckm
[INFO] Using property: jira-project = CKM
...
[INFO] Using property: fhir-server-path = /fhir/*
[INFO] Using property: groupId = gov.va.mobile.service
[INFO] Using property: artifactId = fhir-starter-service
[INFO] Using property: version = 1.0.0
Define value for property 'base-package' gov.va.mobile.fhirstarter.v1:
Define value for property 'package' gov.va.mobile.fhirstarter.v1:
Define value for property 'service-base-path' /fhirstarter/v1:
Define value for property 'formatted-short-name' Fhirstarter:
Confirm properties configuration:
short-name: fhirstarter
...
package: gov.va.mobile.fhirstarter.v1
service-base-path: /fhirstarter/v1
formatted-short-name: Fhirstarter
 Y: : y

Near the end of the output, Maven prints a few warnings such as [WARNING] CP Don’t override file …​/fhir-starter-service/.mvn. They are expected, and you can ignore them.

If everything goes well, you have a new folder named fhir-starter-service, and the archetype prints the next steps:

Generated fhir-starter-service in .../starter-project/fhir-starter-service
Next steps:
  1. Run the setup script. It initializes the git repository, generates the Kubernetes, Skaffold, and pipeline files
     with filegen, applies the single-module fixes to skaffold.yaml, and sets the Dynatrace labels:
       cd fhir-starter-service && ./setup.sh
  2. Build, deploy, and test the service locally:
       ./mvnw clean install -Pwith-skaffold
Java 21 is required to build this service. If you use SDKMAN, run sdk env install in the fhir-starter-service directory once; it installs the Java and Maven versions from the .sdkmanrc file. If sdkman_auto_env=true is set in ~/.sdkman/etc/config (the setup script sets it), SDKMAN switches to those versions whenever you cd into the directory. Otherwise, run sdk env in each new terminal.
At this point, open fhir-starter-service as a new project in your IDE. This will help with navigation and code completion as you work on the service.

Set Up the Service

Before you can build the service, a few more files need to be generated and adjusted: the Kubernetes, Skaffold, and Jenkins files, the deployment documentation, and the Backstage catalog files. The archetype includes a script, setup.sh, that does all of this for you.

Before you run it, make sure that:

  • You are connected to the VPN, and your ~/.m2/settings.xml has your coderepo credentials in the bitbucket server. The file generator (filegen) downloads its templates from coderepo.

  • Your git name and email are set (git config --global user.name and git config --global user.email).

Then run the script from the service directory:

cd fhir-starter-service
./setup.sh

Press Enter at every question to accept the default, or run ./setup.sh -y, which accepts every default without asking. The script shows each command before it runs it. It takes a few minutes, mostly while filegen downloads its templates from coderepo.

When the script finishes, your branch has two new commits, WIP: initial commit and Generated files added. Finally, the script offers to delete itself; press Enter (y).

The script lists SET_ME placeholders in catalog-info.yaml, mkdocs.yaml, and vv.yaml. filegen fills them in from the git remote of the service’s own repository, and your service lives inside the starter-project repository instead. This is expected for the starter project, and you can ignore them.

For what each step of setup.sh does, and the equivalent manual steps, see What setup.sh Does in the appendix. You don’t need them for this guide.

Tour the Generated Project

The archetype generated more than an empty service. It includes sample code for the patterns our FHIR services use, and you will build on that code throughout this guide. The Java code is in src/main/java/gov/va/mobile/fhirstarter/v1:

File What it shows Extended on page

resource/PatientProvider.java

A resource provider for Patient, with a search by identifier (@Search) and a read (@Read). Both validate the request and then return HTTP 501 Not Implemented. @JwtRestrictedGroup lets staff and veteran users call it, and PatientAccess makes sure veterans only request their own records

3

resource/PatientAccess.java

The ICN validation and the veteran access check, shared by every provider that returns patient data. The provider calls it in each operation

3

resource/SystemOperationsProvider.java

A system-level operation, $validate-icn, restricted to staff users. It returns a Parameters resource

6

config/FhirConfig.java

Customizes the FHIR server: it registers SystemOperationsProvider, and has commented examples for interceptors

6

exception/ErrorMessage.java, exception/InvalidIcnException.java

A catalog of the error messages returned to clients, and a HAPI exception that results in HTTP 400 with an OperationOutcome

5

AppProperties.java

The service’s own settings, bound from the va.mobile.fhirstarter.* properties and validated at startup

7

src/main/resources/application*.properties

The base configuration, and one file per Spring profile (dev, sandbox, sqa, prod), with commented examples for downstream clients

3

src/test/java/…​

Unit tests for the samples; SpringSmokeTest, which starts the service with each profile; and integration tests that run against the deployed service: PatientResourceITCase and SystemOperationsITCase (who may call what), HealthCheckITCase, and CapabilityStatementITCase, which also writes the CapabilityStatement to docs/

4

metadata.yaml, README.md, docs/

Service metadata and documentation, with TODO placeholders. docs/local-dev-testing.md describes the local development workflow

9

Three ideas from the generated code come up on every page:

  • Registration. The Mobile Framework registers every Spring bean that implements IResourceProvider (such as PatientProvider) with the FHIR server automatically. Anything else, such as SystemOperationsProvider, which only has operations, is registered in FhirConfig.

  • Who may call, and which patient. The @JwtRestricted and @JwtRestrictedGroup annotations decide which roles may call a provider. They can’t know which patient a request is for, so each operation checks that with PatientAccess: a veteran may only request their own records, identified by the ICN in their JWT.

  • Errors. A resource provider reports an error by throwing a HAPI exception, such as InvalidRequestException (HTTP 400) or ForbiddenOperationException (HTTP 403). HAPI turns it into the HTTP status and an OperationOutcome response.

Build and Run the Service

Build and Test Everything

First, make sure the generated service builds. Run a Maven install with the with-skaffold profile from the fhir-starter-service directory. It builds the service, deploys it and its dependencies to your local cluster, runs the unit tests (29 of them) and the integration tests (14), checks code coverage and code quality, and then removes the deployment:

./mvnw clean install -Pwith-skaffold
You will see a warning at the beginning of the build that says 'The requested profile "with-skaffold" could not be activated because it does not exist.' This is a side effect of the way the service pom file is constructed from tiles during the build process and is expected behavior. You can safely ignore it as long as the build completes successfully.
The service log in the build output contains ERROR lines for AuthenticationException and ForbiddenOperationException. They come from the integration tests that check that requests without permission are rejected.

If the build seems to stop after Running Skaffold RPC server, open skaffold.log in the service directory. Skaffold may have failed, for example with a TLS handshake timeout while downloading Kubernetes files from coderepo, and the build keeps waiting for it. Press Ctrl-C, check your VPN connection, and run the build again.
Make sure this build passes before you change anything. If it fails, work with the Onboarding team to resolve the problem.

Run the Service for Testing

To call the service yourself, keep it running with skaffold dev. This builds the service image, deploys it to your local cluster, forwards its ports (8080 for the service, 8081 for the actuator endpoints), and keeps showing the service log. The first start takes a minute or two, because Skaffold also runs the unit tests; wait until you see Press Ctrl+C to exit.

skaffold dev --rpc-port 50051 --port-forward=true

To stop the service, press Ctrl-C. To make sure that everything Skaffold deployed is cleaned up, run:

skaffold delete

If you would like to start the service with the remote debugging port (5005) exposed, run:

skaffold debug --rpc-port 50051 --port-forward=true

See Skaffold Debugging with IntelliJ for instructions on configuring the remote debugger.

Your Development Loop

On the following pages, you change the service again and again. This is how to get each kind of change into your running service, and how to test it. Come back to this table whenever a page asks you to restart skaffold dev or run the build.

When you…​ Do this

Change Java code

Nothing. skaffold dev copies the changed classes into the running container, and the service restarts in about 10 seconds.

Change files in the kubernetes directory, such as WireMock stubs

Stop skaffold dev with Ctrl-C, run ./scripts/localize.sh force, and start skaffold dev again. See The Kubernetes Files Skaffold Deploys for why.

Change pom.xml

Stop skaffold dev with Ctrl-C, and start it again.

Want a quick check of the unit tests and code quality

./mvnw verify. It takes about a minute and doesn’t need your cluster. It measures code coverage from the unit tests alone, so warnings such as Rule violated for bundle fhir-starter-service: lines covered ratio is 0.67, but expected minimum is 0.80 are expected. The full build adds the integration tests to the coverage, and that’s the number that has to reach 80%.

Want to run the integration tests against your running service

Start the service with skaffold dev -p integration --rpc-port 50051 --port-forward=true instead; the integration profile adds the code coverage agent. Then, in a second terminal, run ./mvnw verify -Pskaffold-integration.

Are about to commit

Stop skaffold dev, and run the full build, ./mvnw clean install -Pwith-skaffold.

The Kubernetes Files Skaffold Deploys

Skaffold doesn’t deploy the files in kubernetes/base and kubernetes/components directly. Before each deployment, scripts/localize.sh copies them, together with the shared files they reference on coderepo, into kubernetes/localized, and Skaffold deploys those copies. Never edit the files in kubernetes/localized; your changes would be overwritten.

To save time, localize.sh only makes new copies when the files git tracks in kubernetes change. A file you added but haven’t committed yet isn’t tracked, so a fix to it isn’t copied, and Skaffold deploys the old version again. ./scripts/localize.sh force always makes new copies; it takes about 10 seconds. (Running git add kubernetes before you restart works too.)

When Something Goes Wrong

skaffold dev doesn’t reach Press Ctrl+C to exit

Your service only becomes ready when the services it depends on are healthy: WireMock (mvi), and from page 7 on, cerner-mock (cerner). If one of them can’t start, for example because a stub file isn’t valid JSON, skaffold dev waits about 5 minutes, prints 1/2 deployment(s) failed or similar, and removes the deployment. Don’t wait that long. In another terminal, look at the pods:

kubectl get pods -n fhir-starter-service-test

A pod whose STATUS is CrashLoopBackOff or Error, or whose READY column stays at 0/1, is the problem. Read its log, using the start of the pod’s name:

kubectl logs -n fhir-starter-service-test deploy/wiremock
kubectl logs -n fhir-starter-service-test deploy/fhir-starter-service-v1

For example, a broken stub makes WireMock log MappingFileException: Error loading file …​ with the file name. Fix the file, stop skaffold dev, run ./scripts/localize.sh force, and start it again.

To check your stub and cerner-mock files before you start, run this from the service directory. It prints each file that isn’t valid JSON, and each WireMock stub that isn’t a single object with request and response:

for f in $(find kubernetes/components -name '*.json'); do
  case $f in
    */wiremock-data/*) check='has("request") and has("response")' ;;
    *) check='type == "object"' ;;
  esac
  jq -e "$check" "$f" > /dev/null 2>&1 || echo "Check $f"
done
The build fails with SkaffoldRenderCheckRule

The build’s service-lint check runs skaffold render for the staging and production profiles, and reports any failure as:

[ERROR] [SkaffoldRenderCheckRule] Failed to execute skaffold render for profile 'staging-build-test': skaffold render failed with exit code 1

The most common cause is a timeout while localize.sh downloads the shared Kubernetes files from coderepo. Check your VPN connection, and run the build again. If it fails again, run the same command that the check runs, which shows the real error:

SKAFFOLD_RPC_PORT=50052 skaffold render --digest-source=tag -p staging-build-test > /dev/null

Leave out --digest-source=tag, and skaffold render fails with an unrelated error, failed to resolve the digest of …​: does the image exist remotely?.

A curl request returns 401 No JWT included in request

Your JWT variables only exist in the terminal where you set them. Run source ~/fhir-starter-jwts.sh (see Get JWTs with jwtgen) in the terminal where you run curl. If the JWTs have expired, generate new ones.

Add the Generated Files to Git

The first build generates a few more files: the CapabilityStatement and OpenAPI documents in docs/, a local copy of the Kubernetes files in kubernetes/localized, and .kubernetes-hash.sha256. It also fills in the ${basePath} placeholders in docs/dibr.md. Commit them, then push your branch:

git add .
git commit -m "Add files generated by the first build"
git push -u origin fhir_starter_<firstname>_<lastname>
From now on, most of your commits include changes in kubernetes/localized, often 20 files or more, because localize.sh makes new copies whenever the Kubernetes files change. Commit them with your changes, but never edit them. To keep IntelliJ’s search from finding them, right-click kubernetes/localized and choose Mark Directory as > Excluded.
Every integration test run writes docs/capability-statement.json again, with a new id and date. Commit the file whenever the service interface changes.

Testing the Service

Start the service with skaffold dev --rpc-port 50051 --port-forward=true and wait until it is running.

Explore the Service

A few endpoints don’t need a JWT. Open them in your browser:

Get JWTs with jwtgen

The other endpoints are JWT protected. To generate JWTs for testing, you need a tool we have called jwtgen. Clone the jwtgen repository to a directory outside of your starter-project, and build it:

git clone https://coderepo.mobilehealth.va.gov/scm/iums/jwtgen.git
cd jwtgen
./mvnw clean package

If you already have a jwtgen clone, for example in ~/Projects, where the setup script puts one, run git pull in it before you build, instead of cloning it again.

You need jwtgen 3.1.0 or later, from its main branch. Earlier versions don’t put the veteran’s ICN in the JWT’s subject, so your service refuses the veteran JWT even for the veteran’s own record, and they can’t run the commands below. The jar name contains the version; if target has a newer jwtgen-<version>.jar, use that one.

You need two JWTs: one for a staff user, and one for a veteran, PATIENT FIFTEEN (10115V737384). You’ll use them for the rest of the guide, in many terminals and over several days, so save them in a file in your home directory, outside the repository. In the jwtgen directory, run:

cat > ~/fhir-starter-jwts.sh <<EOF
export STAFF_JWT=$(java -jar target/jwtgen-3.1.0.jar jwtgen-staff --long-expiration --token-only)
export VETERAN_JWT=$(java -jar target/jwtgen-3.1.0.jar jwtgen-veteran --icn 10115V737384 --long-expiration --token-only)
EOF

Each java -jar command runs one jwtgen command and prints the JWT: --long-expiration makes it valid for 10 years instead of 15 minutes, and --token-only prints only the JWT. Check the file: each line should end with a long JWT that starts with eyJ.

cat ~/fhir-starter-jwts.sh

Then load the JWTs in each terminal where you run curl:

source ~/fhir-starter-jwts.sh
Run java -jar target/jwtgen-3.1.0.jar without a command to use jwtgen’s interactive shell, which prompts for the values; type help there to see every command and option.

You’ll have several terminals open while you work: one running skaffold dev, which keeps showing the service log, and another for curl. Page 7 adds a third.

Your service identifies a veteran by the ICN in the subject (sub) of the JWT, which jwtgen sets to the ICN you entered. Your veteran JWT is for PATIENT FIFTEEN, 10115V737384: it may read that patient’s records, and no one else’s.

Don’t use jwtgen-super for Patient requests. The super user has both the staff and veteran roles, and your service treats any user with the veteran role as a veteran.

Call the Service with curl

The generated $validate-icn operation checks whether a patient identifier is a valid ICN. It’s restricted to staff users.

Request:

curl -s -H "x-vamf-jwt: $STAFF_JWT" 'http://localhost:8080/fhirstarter/v1/fhir/$validate-icn?icn=10115V737384' | jq

Response:

{
  "resourceType": "Parameters",
  "parameter": [
    {
      "name": "valid",
      "valueBoolean": true
    }
  ]
}
Put the URL in single quotes. In double quotes, your shell would replace $validate with the value of an environment variable.

Now try the requests below. To see the HTTP status, add -i to the curl command and leave out | jq, which can’t read the response headers:

  • $validate-icn?icn=10115V737385 with $STAFF_JWT: HTTP 200 with "valueBoolean":false, because the checksum of the ICN is wrong.

  • $validate-icn?icn=10115V737384 with $VETERAN_JWT: HTTP 403, because the operation is restricted to staff users:

    {
  "resourceType": "OperationOutcome",
  "issue": [
    {
      "severity": "error",
      "code": "processing",
      "diagnostics": "403 FORBIDDEN \"Access denied due to security policy for invalid: [role(s)]\""
    }
  ]
}

  • $validate-icn?icn=10115V737384 without the x-vamf-jwt header: HTTP 401, "diagnostics":"401 UNAUTHORIZED \"No JWT included in request\"".

Next, search for a patient by ICN. A FHIR token search parameter takes a system and a value, separated by |:

curl -s -H "x-vamf-jwt: $STAFF_JWT" 'http://localhost:8080/fhirstarter/v1/fhir/Patient?identifier=urn:oid:2.16.840.1.113883.4.349%7C10115V737384' | jq

Response (HTTP 501):

{
  "resourceType": "OperationOutcome",
  "issue": [
    {
      "severity": "error",
      "code": "processing",
      "diagnostics": "The Patient search operation is not yet implemented"
    }
  ]
}
Encode the | as %7C in URLs. The web server rejects a raw | with an HTML error page (HTTP 400). Alternatively, let curl encode it: curl -G --data-urlencode 'identifier=urn:oid:2.16.840.1.113883.4.349|10115V737384' …​

The search isn’t implemented yet; that’s the next page. But it already validates the request and checks access:

  • Patient?identifier=not-an-icn with $STAFF_JWT: HTTP 400, "diagnostics":"Invalid patient identifier, expected an ICN". The message doesn’t repeat the identifier, which could be another kind of patient identifier, such as an SSN; page 5 explains why.

  • Patient?identifier=http://hl7.org/fhir/sid/us-ssn%7C10115V737384 with $STAFF_JWT: HTTP 400, "diagnostics":"Invalid patient identifier system, expected urn:oid:2.16.840.1.113883.4.349".

  • Patient?identifier=10132V467385 with $VETERAN_JWT: HTTP 403, "diagnostics":"Access to the requested patient is not permitted", because a veteran may only request their own records.

  • Patient?identifier=10115V737384 with $VETERAN_JWT: HTTP 501, like the staff search. The veteran passes the access check for their own ICN, and gets as far as the search that isn’t implemented yet.

  • Encounter?patient=10115V737384 with $STAFF_JWT: HTTP 404, "diagnostics":"HAPI-0302: Unknown resource type 'Encounter' - Server knows how to handle: [Patient]". You will add Encounter on page 8.

Every error is an OperationOutcome, even the ones that the framework or HAPI return for you. The severity, code, and diagnostics of each issue tell the client what went wrong.

Other Clients

If you prefer Postman or your IDE’s HTTP client to curl, send the same requests there: add the x-vamf-jwt header with your JWT, and an Accept: application/fhir+json header, and write the | in a URL as %7C.

When you’re done testing, press Ctrl-C to stop skaffold dev, and run skaffold delete to clean up the deployment.

You’re Done When

  • Your service builds and its tests pass with ./mvnw clean install -Pwith-skaffold, and runs with skaffold dev

  • You have looked at the CapabilityStatement and Swagger UI of your running service

  • You have called $validate-icn and the Patient search with a staff JWT, and seen the 400, 401, 403, and 501 responses

  • Your branch is pushed to the starter-project repository

Continue to 3. Implement the Patient Resource Provider to implement the Patient read and search.