Hello, I am using FHIR view and its bigquery runner to query fhir data from a bigquery table on GCP. Wondering how we can work with extensions in this case?

Specifically, the bq table is exported from a fhir-store, and bq automatically strips away extensions (it maps each extension onto its own column). For example the following extension in a resource will actually create a column named retired in the bq table for the resource, and the bq table will NOT have a column with the name extension.

"extension": [
    {
      "url": "https://my-definition/StructureDefinition/retired",
      "valueString": "N"
    },
]

For context, due to our usecase, we are using the SQL on FHIR to specify the query, i.e

simple_pats_config = {
    'resource': 'Patient',
    'select': [
        {
            'alias': 'extensions',
            'path': 'extension',
        },
        ...
    ]
}

The problem with this is that the SQL generated will attempt to UNNEST(extension), which bigquery will error google.api_core.exceptions.BadRequest: 400 Unrecognized name: extension since such column does not exist in the bq table. So the question is how do we work with the bq table in this case? Or alternatively what settings/config do we need when exporting from fhir-store to bq to preserve the extension column so this works?

And a separate but related question is whether there will be support for extension.url matching. Currently, if the fhir-path used is extension.where(url = 'some-url'), views.from_view_definition(query) will throw an error (ValueError: Identifier url not in dict_keys(['id', 'extension', 'value'])). This is perhaps because (if i am correct) url is currently not part of child_defs of extension in FhirPathDataType. Maybe I am mistaken and if this is somehow supported it would be great!

Thanks in advance.

I am using https://colab.sandbox.google.com/ with these lines:

!git clone https://github.com/google/fhir-py.git
!cd fhir-py; ./install_all.sh

This is my output:

Processing ./google-fhir-core
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Preparing metadata (setup.py) ... error
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Processing ./google-fhir-r4
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Preparing metadata (setup.py) ... error
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Processing ./google-fhir-views
Preparing metadata (setup.py) ... done
Collecting absl-py~=0.10.0
Using cached absl_py-0.10.0-py3-none-any.whl (127 kB)
ERROR: Could not find a version that satisfies the requirement google-fhir-core~=0.8 (from google-fhir-views[bigquery,r4]) (from versions: none)
ERROR: No matching distribution found for google-fhir-core~=0.8

Hi! Our team is very interested in using this package, but the lack of 4.X protobuf support is blocking us pulling it into our repo (todos in a few setup.py ex).

# TODO(b/276635321): Fix compatibility issue with protobuf 4.x

Curious if there's any chance this will be addressed soon / if the incompatibilities could be elaborated on, we'd be willing to contribute.

Thanks!

I'm working with a BigQuery FHIR dataset from a Google FHIR store export (analytics_v2 schema). I need to extract the partOf.id of an Organization, but the FHIR Views library's FHIRPath builder doesn't seem to account for the analytics_v2 schema's structure where partOf.id is partOf.organizationId.

Here is the code,

org = views.view_of('Organization')

child_orgs = org.select({
    'id': org.id,
    'name': org.name,

    # partOf.id throws: BadRequest: 400 Field name id does not exist in STRUCT>, text STRING>, system STRING, ...>, reference STRING, ...> at [1:89]
    'org_id': org.partOf.id,

    # partOf.organizationId throws: AttributeError: ('No such field organizationId in partOf', "Expected something in ['display', 'extension', 'id', 'identifier', 'reference', 'type']")
    'org_id2': org.partOf.organizationId,
})

Any guidance on how to handle this would be greatly appreciated. Thanks!

Answer field is not captured while creating dataframe from QuestionnaireResponse parquet files using fhir-views.

Code for creating dataframe

import pandas
import json

from sqlalchemy import dialects
from sqlalchemy import engine

from google.fhir.views import r4
from google.fhir.views import spark_runner

fhir_dataset = 'default'
analysis_dataset = 'statin_analysis_example'

dialects.registry.register('hive', 'pyhive.sqlalchemy_hive', 'HiveDialect')

query_engine = engine.create_engine('hive://localhost:10001/default')

runner = spark_runner.SparkRunner(
query_engine=query_engine,
fhir_dataset=fhir_dataset,
view_dataset=analysis_dataset,
snake_case_resource_tables=True,
)

Load views based on the base FHIR R4 profile definitions.

views = r4.base_r4()
qrs = views.view_of('QuestionnaireResponse')

QuestionnaireResponse_df = runner.to_dataframe(
qrs.select(
{
"id": qrs.id,
"tag": qrs.meta.tag.code,
"patient_id":qrs.subject.reference,
"encounter_id": qrs.encounter.reference,
"questionnaire": qrs.questionnaire,
"item":qrs.item
}
)
)

when I do QuestionnaireResponse_df.iloc[0].item
dataframe_json.zip

when I query for the same data using sql

Please refer the attached Json file
qr.zip

Clearly the nested items and answer field are not captured in the dataframe. For example if I take the first object of the dataframe json and sql output

##Dataframe output
[
{
"id": null,
"linkId": "vitals",
"definition": null,
"text": null,
"answer": null,
"item": [
{
"id": null,
"linkId": "3.1",
"definition": null,
"text": "ANC service provided at *"
},
{
"id": null,
"linkId": "service-type-details",
"definition": null,
"text": null
},
{
"id": null,
"linkId": "machine-available",
"definition": null,
"text": "Is BP Machine Available? *"
},
{
"id": null,
"linkId": "3.2",
"definition": null,
"text": null
},
{
"id": null,
"linkId": "3.3",
"definition": null,
"text": null
},
{
"id": null,
"linkId": "3.4",
"definition": null,
"text": null
},
{
"id": null,
"linkId": "3.5",
"definition": null,
"text": null
}
]

##Sql output
[
{
"id": null,
"linkId": "vitals",
"definition": null,
"text": null,
"answer": null,
"item": [
{
"id": null,
"linkId": "3.1",
"definition": null,
"text": "ANC service provided at *",
"answer": [
{
"id": null,
"value": {
"boolean": null,
"decimal": null,
"integer": null,
"date": null,
"dateTime": null,
"time": null,
"string": null,
"uri": null,
"attachment": null,
"coding": {
"system": null,
"version": null,
"code": "home",
"display": "Home",
"userSelected": null
},
"quantity": null,
"reference": null
},
"item": null
}
],
"item": null
},
{
"id": null,
"linkId": "service-type-details",
"definition": null,
"text": null,
"answer": null,
"item": [
{
"id": null,
"linkId": "service-type",
"definition": null,
"text": null,
"answer": [
{
"id": null,
"value": {
"boolean": null,
"decimal": null,
"integer": null,
"date": null,
"dateTime": null,
"time": null,
"string": null,
"uri": null,
"attachment": null,
"coding": {
"system": "https://iprdgroup.com/custom-codes",
"version": null,
"code": "anc-visit",
"display": "ANC Visit",
"userSelected": null
},
"quantity": null,
"reference": null
},
"item": null
}
],
"item": null
}
]
},
{
"id": null,
"linkId": "machine-available",
"definition": null,
"text": "Is BP Machine Available? *",
"answer": [
{
"id": null,
"value": {
"boolean": false,
"decimal": null,
"integer": null,
"date": null,
"dateTime": null,
"time": null,
"string": null,
"uri": null,
"attachment": null,
"coding": null,
"quantity": null,
"reference": null
},
"item": null
}
],
"item": null
},
{
"id": null,
"linkId": "3.2",
"definition": null,
"text": null,
"answer": null,
"item": null
},
{
"id": null,
"linkId": "3.3",
"definition": null,
"text": null,
"answer": null,
"item": null
},
{
"id": null,
"linkId": "3.4",
"definition": null,
"text": null,
"answer": null,
"item": [
{
"id": null,
"linkId": "3.4.1",
"definition": null,
"text": null,
"answer": null,
"item": [
{
"id": null,
"linkId": "weight",
"definition": null,
"text": "Weight (in Kgs) *",
"answer": [
{
"id": null,
"value": {
"boolean": null,
"decimal": 79.0,
"integer": null,
"date": null,
"dateTime": null,
"time": null,
"string": null,
"uri": null,
"attachment": null,
"coding": null,
"quantity": null,
"reference": null
},
"item": null
}
],
"item": null
},
{
"id": null,
"linkId": "3.4.1.2",
"definition": null,
"text": null,
"answer": [
{
"id": null,
"value": {
"boolean": null,
"decimal": null,
"integer": null,
"date": null,
"dateTime": null,
"time": null,
"string": "Kgs",
"uri": null,
"attachment": null,
"coding": null,
"quantity": null,
"reference": null
},
"item": null
}
],
"item": null
}
]
},
{
"id": null,
"linkId": "3.4.2",
"definition": null,
"text": null,
"answer": null,
"item": [
{
"id": null,
"linkId": "3.4.2.1",
"definition": null,
"text": null,
"answer": [
{
"id": null,
"value": {
"boolean": null,
"decimal": null,
"integer": null,
"date": null,
"dateTime": null,
"time": null,
"string": null,
"uri": null,
"attachment": null,
"coding": {
"system": "http://fhir.org/guides/who/anc-cds/CodeSystem/anc-custom-codes",
"version": null,
"code": "ANC.B8.DE3",
"display": "Current weight",
"userSelected": null
},
"quantity": null,
"reference": null
},
"item": null
}
],
"item": null
},
{
"id": null,
"linkId": "3.4.2.2",
"definition": null,
"text": null,
"answer": [
{
"id": null,
"value": {
"boolean": null,
"decimal": null,
"integer": null,
"date": null,
"dateTime": null,
"time": null,
"string": null,
"uri": null,
"attachment": null,
"coding": {
"system": "http://hl7.org/fhir/observation-status",
"version": null,
"code": "preliminary",
"display": "Preliminary",
"userSelected": null
},
"quantity": null,
"reference": null
},
"item": null
}
],
"item": null
},
{
"id": null,
"linkId": "3.4.2.3",
"definition": null,
"text": null,
"answer": [
{
"id": null,
"value": {
"boolean": null,
"decimal": null,
"integer": null,
"date": null,
"dateTime": null,
"time": null,
"string": null,
"uri": null,
"attachment": null,
"coding": {
"system": "http://hl7.org/fhir/ValueSet/observation-category",
"version": null,
"code": "vital-signs",
"display": "Vital Signs",
"userSelected": null
},
"quantity": null,
"reference": null
},
"item": null
}
],
"item": null
}
]
}
]
},
{
"id": null,
"linkId": "3.5",
"definition": null,
"text": null,
"answer": null,
"item": [
{
"id": null,
"linkId": "3.5.1",
"definition": null,
"text": null,
"answer": null,
"item": [
{
"id": null,
"linkId": "3.5.1.1",
"definition": null,
"text": "Height (in cms) *",
"answer": [
{
"id": null,
"value": {
"boolean": null,
"decimal": 100.0,
"integer": null,
"date": null,
"dateTime": null,
"time": null,
"string": null,
"uri": null,
"attachment": null,
"coding": null,
"quantity": null,
"reference": null
},
"item": null
}
],
"item": null
},
{
"id": null,
"linkId": "3.5.1.2",
"definition": null,
"text": null,
"answer": [
{
"id": null,
"value": {
"boolean": null,
"decimal": null,
"integer": null,
"date": null,
"dateTime": null,
"time": null,
"string": "cms",
"uri": null,
"attachment": null,
"coding": null,
"quantity": null,
"reference": null
},
"item": null
}
],
"item": null
}
]
},
{
"id": null,
"linkId": "3.5.2",
"definition": null,
"text": null,
"answer": null,
"item": [
{
"id": null,
"linkId": "3.5.2.1",
"definition": null,
"text": null,
"answer": [
{
"id": null,
"value": {
"boolean": null,
"decimal": null,
"integer": null,
"date": null,
"dateTime": null,
"time": null,
"string": null,
"uri": null,
"attachment": null,
"coding": {
"system": "http://fhir.org/guides/who/anc-cds/CodeSystem/anc-custom-codes",
"version": null,
"code": "ANC.B8.DE1",
"display": "Height",
"userSelected": null
},
"quantity": null,
"reference": null
},
"item": null
}
],
"item": null
},
{
"id": null,
"linkId": "3.5.2.2",
"definition": null,
"text": null,
"answer": [
{
"id": null,
"value": {
"boolean": null,
"decimal": null,
"integer": null,
"date": null,
"dateTime": null,
"time": null,
"string": null,
"uri": null,
"attachment": null,
"coding": {
"system": "http://hl7.org/fhir/observation-status",
"version": null,
"code": "registered",
"display": "Registered",
"userSelected": null
},
"quantity": null,
"reference": null
},
"item": null
}
],
"item": null
},
{
"id": null,
"linkId": "3.5.2.3",
"definition": null,
"text": null,
"answer": [
{
"id": null,
"value": {
"boolean": null,
"decimal": null,
"integer": null,
"date": null,
"dateTime": null,
"time": null,
"string": null,
"uri": null,
"attachment": null,
"coding": {
"system": "http://hl7.org/fhir/ValueSet/observation-category",
"version": null,
"code": "vital-signs",
"display": "Vital Signs",
"userSelected": null
},
"quantity": null,
"reference": null
},
"item": null
}
],
"item": null
}
]
}
]
}
]

Hi. Thanks for the great work!

I was just wondering if support for the latest FHIR R5 specification is in the books or not.

Hi, we have a multi language project where we define APIs that depends on the fhir protos in google-fhir. So far we've been able to use the google-fhir python package for the generated protos.

Now it seems like the google-fhir python package is effectively deprecated and the fhir protos are provided in here (google-fhir-r4, google-fhir-core). However, it also seems like the fhir protos are copied in this repo with a different path, resulting a diffrent import path. And we have a problem when try to deprecate the use of google-fhir python package in favor of google-fhir-r4, I'll describe the problem below.

In our API proto definition, we have this (we clone the google-fhir repo's protos).

import "proto/google/fhir/proto/r4/core/resources/patient.proto";

Then, the generated python protobuf files will have

from proto.google.fhir.proto.r4.core.resource import patient_pb2

which works with the deprecated google-fhir python package.

However, if we want to use the google-fhir-r4/core package, it is using this import path (due to it's own folder structure for the protos)

from google.fhir.r4.proto.core.resources import patient_pb2

Do you have any suggestions on how we can migrate to google-fhir-r4? We can stay with google-fhir package for now but we also want to use the google-fhir-views package for our next project.