permitio / opal Goto Github PK
View Code? Open in Web Editor NEWPolicy and data administration, distribution, and real-time updates on top of Policy Agents (OPA, Cedar, ...)
Home Page: https://opal.ac
License: Apache License 2.0
opal's People
Contributors
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Stargazers
Watchers
Forkers
shimont doytsujin lightoyou pujan14 zechartiahsaher guyfrd gdmarsh r1999t hongbo-miao akoserwal gujun4990 rbanks-willow phi1010 syllogy natekat trendingtechnology ronniebasak manuelnemirovsky balaji1359 mburtless yangboyd anodynos opensourcerepositorys danielbass37 franklinharry authorizon onuratakan stremovsky py-go vasica38 walking-appa matanyastroh lamdalabs devopstoday11 demitrial suhovdg oishikg h9ing krishna-selectstar alokkulkarni daotlresearch kunalranjan19 i-am-yuvi scalifi-ai webclinic017 treysidechain michivi vijaynallagatla lakshyabhasin509 filipermit ndwannafly djun davide-ganito studioblvck bluepeople1 dservy jamielennox mohamedtaoufik brianhyder cloudnepal jenstancs loan75 tibotix garnerp vaibhavtechsophy nagarajskharvi jayanthvhavle avi-88 mirazxmondal clins10 adi9235 rishabh9450 calm1111 rahul-prasad-07 money8203 bharatveeryadav gvangest karndeb devopsgodd avo-sepp tlowerison appdirectory scarlier renatosc cclauss vivedo omriza veenaamb urspraveen2001 stevendaniels maurice-freitag oc002 thequantumquirk god0fcode saha-7 david-hamilton-bah gouravjazz thilak009 cbat98 l2dopal's Issues
Helm chart for kubernetes (or terraform module, or both :))
OPAL is already cloud-native ready and has prebuilt docker images ready to go.
However we want to make it even easier and faster to integrate OPAL quickly in your setup.
We (the maintainers) will probably get to it soon, but if someone is willing to work with us on this, it will speed up the process.
Statistics of opal-clients
Discussed in https://github.com/authorizon/opal/discussions/151
(^ Please go to the discussion there to check more info)
Originally posted by Hongbo-Miao October 14, 2021
Hi OPAL team, assume we have
- Multiple pods in a Kubernetes cluster
- Each/some pods has a main container and a sidecar container with opal-client inside
- Each/some pods has multiple replicas
- Each opal-client in the sidecar container can subscribe one or more topics
Is there a way such as letting opal-servers to return some statistics such as
- the total number of OPAL clients
- the number of OPAL clients per application/pod (each application/pod can have multiple replicas. All subscribe to same one or multiple topics)
the number of OPAL clients per topic(This might not be really useful as one opal-client can subscribe one or multiple topics)
Thanks!
Make OPAL support all common PEM key format for JWT encryption keys
Discussed in https://github.com/authorizon/opal/discussions/164
Originally posted by Hongbo-Miao October 21, 2021
Method 1 - RFC-4253 of SSH succeed in opal-server
OPAL_AUTH_PRIVATE_KEY_PASSPHRASE="ds6l3qYYx9UsYcgshmlbsMJTXs1lVH9ndf13Xp1xNKxbqjFdxFvdkJxpm0DfjAhh"
ssh-keygen -t rsa -b 4096 -m pem -f id_rsa -N "${OPAL_AUTH_PRIVATE_KEY_PASSPHRASE}"This will generate both id_rsa and id_rsa.pub
The public key is in RFC-4253 SSH format.
Above works in opal-server with default envs
OPAL_AUTH_PRIVATE_KEY_FORMAT=pem
OPAL_AUTH_PUBLIC_KEY_FORMAT=sshHowever, I am not sure how to validate it in Go. I have a question at https://stackoverflow.com/questions/69662259/key-is-of-invalid-type
Method 2 - PKCS#1 of PEM failed in opal-server
OPAL_AUTH_PRIVATE_KEY_PASSPHRASE="ds6l3qYYx9UsYcgshmlbsMJTXs1lVH9ndf13Xp1xNKxbqjFdxFvdkJxpm0DfjAhh"
ssh-keygen -t rsa -b 4096 -m pem -f id_rsa -N "${OPAL_AUTH_PRIVATE_KEY_PASSPHRASE}"
ssh-keygen -e -m pem -f id_rsa > id_rsa.pub -P ${OPAL_AUTH_PRIVATE_KEY_PASSPHRASE}The public key is in PKCS#1 PEM format.
This failed in opal-server with
OPAL_AUTH_PRIVATE_KEY_FORMAT=pem
OPAL_AUTH_PUBLIC_KEY_FORMAT=pemMethod 3 - X.509/SPKI of PEM failed in opal-server
OPAL_AUTH_PRIVATE_KEY_PASSPHRASE="ds6l3qYYx9UsYcgshmlbsMJTXs1lVH9ndf13Xp1xNKxbqjFdxFvdkJxpm0DfjAhh"
ssh-keygen -t rsa -b 4096 -m pem -f id_rsa -N "${OPAL_AUTH_PRIVATE_KEY_PASSPHRASE}"
ssh-keygen -e -m pkcs8 -f id_rsa > id_rsa.pub -P ${OPAL_AUTH_PRIVATE_KEY_PASSPHRASE}The public key is in X.509/SPKI PEM format.
This failed in opal-server with
OPAL_AUTH_PRIVATE_KEY_FORMAT=pem
OPAL_AUTH_PUBLIC_KEY_FORMAT=pemOPAL confirmation callback/webhook
confirmation callbacks
- OPAL client (if configured) will call a callback/webhook with confirmation of the update
if the callback/webhook url is not responding - OPAL will log it, but continue as normal - i.e: error in the application callback should not stop or interfere with OPAL - callback message will have some optional version/id string of the update and the hash of the update
Readiness policy
- OPAL will load OPA with a hardcoded policy called
opal.opa.readyor something like this - by default will befalse - OPAL will automatically change
opal.opa.readyto betrueupon successful completion of fetching all the default data sources entries and at least one policy bundle being loaded. - The application can query
opal.opa.readyif it wants to via OPAs normal data API to make sure OPA is ready
full discussion on slack: https://opal-access.slack.com/archives/C01RY1X9SBC/p1618920616054400
ERROR | Critical: OPA health-check policy is enabled, but cannot find policy
I followed How to use data update callbacks and OPA healthcheck policy and succeed running the docker-compose demo.
Now I am trying it in a Kubernetes cluster.
My opal-server has set up JWT, and I generated one and added to opal-client. They work well and can communicate with each other.
First here is the opal-client log when it works well:
[2021-10-18 23:26:16 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2021-10-18 23:26:16 +0000] [1] [INFO] Listening at: http://0.0.0.0:7000 (1)
[2021-10-18 23:26:16 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-10-18 23:26:16 +0000] [9] [INFO] Booting worker with pid: 9
2021-10-18T23:26:17.493894+0000 | opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'FastApiRpcFetchProvider' found at: <class 'opal_common.fetcher.providers.fastapi_rpc_fetch_provider.FastApiRpcFetchProvider'>
2021-10-18T23:26:17.494433+0000 | opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'HttpFetchProvider' found at: <class 'opal_common.fetcher.providers.http_fetch_provider.HttpFetchProvider'>
2021-10-18T23:26:17.546170+0000 | opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'PostgresFetchProvider' found at: <class 'opal_fetcher_postgres.provider.PostgresFetchProvider'>
2021-10-18T23:26:17.546534+0000 | opal_common.fetcher.fetcher_register | INFO | Fetcher Register loaded
2021-10-18T23:26:17.548360+0000 | opal_client.callbacks.register | INFO | Callbacks register loaded
2021-10-18T23:26:17.548847+0000 | opal_client.client | INFO | API authentication disabled (public encryption key was not provided)
2021-10-18T23:26:17.633390+0000 | uvicorn.server | INFO | Started server process [9]
2021-10-18T23:26:17.633652+0000 | uvicorn.lifespan.on | INFO | Waiting for application startup.
2021-10-18T23:26:17.634044+0000 | opal_client.opa.runner | INFO | Launching opa runner
2021-10-18T23:26:17.634298+0000 | uvicorn.lifespan.on | INFO | Application startup complete.
2021-10-18T23:26:17.637493+0000 | opal_client.opa.runner | INFO | Running OPA inline: opa run --server --addr=:8181 --authentication=off --authorization=off --config-file=/data/opa/config.yaml --log-level=info
2021-10-18T23:26:17.661563+0000 | opal_client.opa.logger | INFO | Initializing server. {'addrs': [':8181'], 'diagnostic-addrs': [], 'time': '2021-10-18T23:26:17Z'}
2021-10-18T23:26:17.663061+0000 | opal_client.opa.logger | INFO | Starting decision logger. {'plugin': 'decision_logs', 'time': '2021-10-18T23:26:17Z'}
2021-10-18T23:26:18.646088+0000 | opal_client.opa.runner | INFO | Running OPA initial start callbacks
2021-10-18T23:26:18.646692+0000 | opal_client.policy.updater | INFO | Launching policy updater
2021-10-18T23:26:18.647141+0000 | opal_client.data.updater | INFO | Launching data updater
2021-10-18T23:26:18.647495+0000 | opal_client.policy.updater | INFO | Subscribing to topics: ['policy:.']
2021-10-18T23:26:18.647911+0000 | opal_client.data.updater | INFO | Subscribing to topics: ['policy_data']
2021-10-18T23:26:18.648326+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO | Trying to connect to Pub/Sub server - ws://opal-server-service.hm-opa:7002/ws
2021-10-18T23:26:18.649096+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal-server-service.hm-opa:7002/ws
2021-10-18T23:26:18.652890+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO | Trying to connect to Pub/Sub server - ws://opal-server-service.hm-opa:7002/ws
2021-10-18T23:26:18.653460+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal-server-service.hm-opa:7002/ws
2021-10-18T23:26:18.747103+0000 | opal_client.data.updater | INFO | Connected to server
2021-10-18T23:26:18.747249+0000 | opal_client.data.updater | INFO | Performing data configuration, reason: Initial load
2021-10-18T23:26:18.747568+0000 | opal_client.data.updater | INFO | Getting data-sources configuration from 'http://opal-server-service.hm-opa:7002/data/config'
2021-10-18T23:26:18.754995+0000 | opal_client.data.updater | INFO | Triggering data update with id: 6f46820d41454c3886b22780f150de3a
2021-10-18T23:26:18.755274+0000 | opal_client.data.updater | INFO | Fetching policy data
2021-10-18T23:26:18.756081+0000 | opal_client.data.fetcher | INFO | Fetching data from url: postgresql://[email protected]:40072/opa_db
2021-10-18T23:26:18.778370+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO | Connected to PubSub server ws://opal-server-service.hm-opa:7002/ws
2021-10-18T23:26:18.779592+0000 | opal_client.policy.updater | INFO | Connected to server
2021-10-18T23:26:18.779807+0000 | opal_client.policy.updater | INFO | Refetching policy code (full bundle)
2021-10-18T23:26:18.799877+0000 | opal_client.data.updater | INFO | Saving fetched data to policy-store: source url='postgresql://[email protected]:40072/opa_db', destination path='/roles'
2021-10-18T23:26:18.803579+0000 | opal_client.opa.logger | INFO | Received request. {'client_addr': '[::1]:45030', 'req_id': 1, 'req_method': 'PUT', 'req_path': '/v1/data/roles', 'time': '2021-10-18T23:26:18Z'}
2021-10-18T23:26:18.803980+0000 | opal_client.opa.logger | INFO | Sent response. {'client_addr': '[::1]:45030', 'req_id': 1, 'req_method': 'PUT', 'req_path': '/v1/data/roles', 'resp_bytes': 0, 'resp_duration': 0.411255, 'resp_status': 204, 'time': '2021-10-18T23:26:18Z'}
2021-10-18T23:26:18.815959+0000 | opal_client.policy.updater | INFO | got policy bundle, commit hash: '910d43e7e4da262d838c562ce3ae9910600fe445'
2021-10-18T23:26:18.821297+0000 | opal_client.opa.logger | INFO | Received request. {'client_addr': '[::1]:45034', 'req_id': 2, 'req_method': 'GET', 'req_path': '/v1/policies', 'time': '2021-10-18T23:26:18Z'}
2021-10-18T23:26:18.822307+0000 | opal_client.opa.logger | INFO | Sent response. {'client_addr': '[::1]:45034', 'req_id': 2, 'req_method': 'GET', 'req_path': '/v1/policies', 'resp_bytes': 13, 'resp_duration': 0.459031, 'resp_status': 200, 'time': '2021-10-18T23:26:18Z'}
2021-10-18T23:26:18.828864+0000 | opal_client.opa.logger | INFO | Received request. {'client_addr': '[::1]:45036', 'req_id': 3, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'time': '2021-10-18T23:26:18Z'}
2021-10-18T23:26:18.831522+0000 | opal_client.opa.logger | INFO | Sent response. {'client_addr': '[::1]:45036', 'req_id': 3, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'resp_bytes': 2, 'resp_duration': 2.207848, 'resp_status': 200, 'time': '2021-10-18T23:26:18Z'}
2021-10-18T23:26:18.832607+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO | Connected to PubSub server ws://opal-server-service.hm-opa:7002/wsNow I tried to set up health check.
When I added
- OPAL_SHOULD_REPORT_ON_DATA_UPDATES=True
# we configure a callback to go to a default location in the OPAL server, but you can configure
# a callback to any url you'd like. Each callback is either the url alone, or a tuple of
# (url, HttpFetcherConfig).
# We show here both ways to configure the same endpoint, one of them demonstrate how to
# add extra HTTP headers (the header shown is ignored, only here for example).
- OPAL_DEFAULT_UPDATE_CALLBACKS={"callbacks":["http://opal_server:7002/data/callback_report"]}
# - OPAL_DEFAULT_UPDATE_CALLBACKS={"callbacks":[("http://opal_server:7002/data/callback_report",{"headers":{"X-My-Token":"token"}})]}
# OPAL can load a special policy into OPA that acts as a healthcheck policy (Not directly related to the callback feature).
# This policy defines two opa rules you can query:
# ready rule (POST http://localhost:8181/data/system/opal/ready): signals that OPA is ready to accept authorization queries.
# healthy rule (POST http://localhost:8181/data/system/opal/ready): signals that the last policy and data updates succeeded.
- OPAL_OPA_HEALTH_CHECK_POLICY_ENABLED=TrueIn my case, it is
OPAL_SHOULD_REPORT_ON_DATA_UPDATES: "True"
OPAL_DEFAULT_UPDATE_CALLBACKS: '{"callbacks":["http://opal-server-service.hm-opa:7002/data/callback_report"]}'
OPAL_OPA_HEALTH_CHECK_POLICY_ENABLED: "True"However, I got this error in my opal-client log:
[2021-10-18 23:24:07 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2021-10-18 23:24:07 +0000] [1] [INFO] Listening at: http://0.0.0.0:7000 (1)
[2021-10-18 23:24:07 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-10-18 23:24:07 +0000] [9] [INFO] Booting worker with pid: 9
2021-10-18T23:24:07.965891+0000 | opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'FastApiRpcFetchProvider' found at: <class 'opal_common.fetcher.providers.fastapi_rpc_fetch_provider.FastApiRpcFetchProvider'>
2021-10-18T23:24:07.966694+0000 | opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'HttpFetchProvider' found at: <class 'opal_common.fetcher.providers.http_fetch_provider.HttpFetchProvider'>
2021-10-18T23:24:08.012902+0000 | opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'PostgresFetchProvider' found at: <class 'opal_fetcher_postgres.provider.PostgresFetchProvider'>
2021-10-18T23:24:08.013219+0000 | opal_common.fetcher.fetcher_register | INFO | Fetcher Register loaded
2021-10-18T23:24:08.015348+0000 | opal_client.callbacks.register | INFO | Callbacks register loaded
2021-10-18T23:24:08.015979+0000 | opal_client.client | INFO | API authentication disabled (public encryption key was not provided)
2021-10-18T23:24:08.087297+0000 | uvicorn.server | INFO | Started server process [9]
2021-10-18T23:24:08.088012+0000 | uvicorn.lifespan.on | INFO | Waiting for application startup.
2021-10-18T23:24:08.088728+0000 | opal_client.opa.runner | INFO | Launching opa runner
2021-10-18T23:24:08.089187+0000 | uvicorn.lifespan.on | INFO | Application startup complete.
2021-10-18T23:24:08.091001+0000 | opal_client.opa.runner | INFO | Running OPA inline: opa run --server --addr=:8181 --authentication=off --authorization=off --config-file=/data/opa/config.yaml --log-level=info
2021-10-18T23:24:08.109728+0000 | opal_client.opa.logger | INFO | Initializing server. {'addrs': [':8181'], 'diagnostic-addrs': [], 'time': '2021-10-18T23:24:08Z'}
2021-10-18T23:24:08.110709+0000 | opal_client.opa.logger | INFO | Starting decision logger. {'plugin': 'decision_logs', 'time': '2021-10-18T23:24:08Z'}
2021-10-18T23:24:09.098182+0000 | opal_client.opa.runner | INFO | Running OPA initial start callbacks
2021-10-18T23:24:09.098529+0000 | opal_client.client |ERROR | Critical: OPA health-check policy is enabled, but cannot find policy at /usr/local/lib/python3.8/site-packages/opal_client-0.1.15-py3.8.egg/opal_client/opa/healthcheck/opal.rego
2021-10-18T23:24:09.098722+0000 | opal_client.client |CRITICAL | healthcheck policy enabled but could not be initialized!
2021-10-18T23:24:09.098798+0000 | opal_client.client | INFO | triggering shutdown with SIGTERM...
2021-10-18T23:24:09.197616+0000 | uvicorn.server | INFO | Shutting down
[2021-10-18 23:24:09 +0000] [9] [INFO] Error while closing socket [Errno 9] Bad file descriptor
2021-10-18T23:24:09.298780+0000 | uvicorn.lifespan.on | INFO | Waiting for application shutdown.
2021-10-18T23:24:09.299108+0000 | opal_client.client | INFO | stopping background tasks...
2021-10-18T23:24:09.299252+0000 | opal_client.opa.runner | INFO | Stopping opa runner
2021-10-18T23:24:09.299325+0000 | opal_client.opa.runner | INFO | Stopping OPA
2021-10-18T23:24:09.299944+0000 | opal_client.opa.logger | INFO | Shutting down... {'time': '2021-10-18T23:24:09Z'}
2021-10-18T23:24:09.301328+0000 | opal_client.opa.logger | INFO | Server shutdown. {'time': '2021-10-18T23:24:09Z'}
2021-10-18T23:24:09.308254+0000 | opal_client.opa.logger | INFO | Stopping decision logger. {'plugin': 'decision_logs', 'time': '2021-10-18T23:24:09Z'}
2021-10-18T23:24:09.308705+0000 | opal_client.opa.logger | INFO | Flushing decision logs. {'plugin': 'decision_logs', 'time': '2021-10-18T23:24:09Z'}
2021-10-18T23:24:09.309221+0000 | opal_client.opa.logger | INFO | All decisions in buffer uploaded. {'plugin': 'decision_logs', 'time': '2021-10-18T23:24:09Z'}
2021-10-18T23:24:09.311407+0000 | opal_client.opa.runner | INFO | OPA exited with return code: 0
2021-10-18T23:24:10.300139+0000 | opal_client.client | INFO | trying to shutdown DataUpdater and PolicyUpdater gracefully...
2021-10-18T23:24:10.300612+0000 | opal_client.data.updater | INFO | Stopping data updater
2021-10-18T23:24:10.300781+0000 | opal_client.policy.updater | INFO | Stopping policy updater
2021-10-18T23:24:10.301351+0000 | uvicorn.lifespan.on | INFO | Application shutdown complete.
2021-10-18T23:24:10.301698+0000 | uvicorn.server | INFO | Finished server process [9]
[2021-10-18 23:24:10 +0000] [9] [INFO] Worker exiting (pid: 9)
<green>{time}</green> | <blue>{name: <40}</blue>|<level>{level:^6} | {message}</level>
{exception}I also tried setting up headers, as my opal-server has set up the JWT. However, still no luck.
OPAL_DEFAULT_UPDATE_CALLBACKS: '{"callbacks":["http://opal-server-service.hm-opa:7002/data/callback_report",{"headers":{"secret-key":"looooogJWTToken"}})]}'Any clue? Thanks!
Data updater should log less stuff
Discussion in slack:
https://opal-access.slack.com/archives/C01RY1X9SBC/p1630351244003600
Allow skipping `OPAL_DEFAULT_UPDATE_CALLBACKS` if have `OPAL_SHOULD_REPORT_ON_DATA_UPDATES=True`
Following up with https://github.com/authorizon/opal/issues/166
Just a UX improvement idea. Not urgent.
If we only provide OPAL_SHOULD_REPORT_ON_DATA_UPDATES=True without OPAL_DEFAULT_UPDATE_CALLBACKS, then let opal-client can point to opal-server by default.
Error when requesting client access token from server API
I'm running the OPAL server using the docker image, and I'm getting the following error from the /token endpoint
The request seems to match the swagger definition, so I'm not sure what's missing that's causing the error.
Any help or tips would be appreciated.
ValueError: Reference at 'refs/heads/master' does not exist
Hi @asafc based on
https://github.com/authorizon/opal/discussions/139#discussioncomment-1303934
it works well, it can pull the pricate repo successfully if you search Clone succeeded in the log.
However, it also has some error logs inside like
ValueError: Reference at 'refs/heads/master' does not exist
Here is the full log:
Click to expand!
➜ docker compose up --force-recreate
[+] Running 4/4
⠿ Container opal-fetcher-postgres_example_db_1 Recreated 0.2s
⠿ Container opal-fetcher-postgres_broadcast_channel_1 Recreated 0.2s
⠿ Container opal-fetcher-postgres_opal_server_1 Recreated 0.1s
⠿ Container opal-fetcher-postgres_opal_client_1 Recreated 0.1s
Attaching to broadcast_channel_1, example_db_1, opal_client_1, opal_server_1
broadcast_channel_1 |
broadcast_channel_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
broadcast_channel_1 |
broadcast_channel_1 | 2021-09-10 20:12:20.222 UTC [1] LOG: starting PostgreSQL 13.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.3.1_git20210424) 10.3.1 20210424, 64-bit
broadcast_channel_1 | 2021-09-10 20:12:20.222 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
broadcast_channel_1 | 2021-09-10 20:12:20.222 UTC [1] LOG: listening on IPv6 address "::", port 5432
broadcast_channel_1 | 2021-09-10 20:12:20.231 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
broadcast_channel_1 | 2021-09-10 20:12:20.237 UTC [22] LOG: database system was shut down at 2021-09-10 20:11:56 UTC
broadcast_channel_1 | 2021-09-10 20:12:20.242 UTC [1] LOG: database system is ready to accept connections
example_db_1 |
example_db_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
example_db_1 |
example_db_1 | 2021-09-10 20:12:20.377 UTC [1] LOG: starting PostgreSQL 13.4 (Debian 13.4-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
example_db_1 | 2021-09-10 20:12:20.377 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
example_db_1 | 2021-09-10 20:12:20.377 UTC [1] LOG: listening on IPv6 address "::", port 5432
example_db_1 | 2021-09-10 20:12:20.382 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
example_db_1 | 2021-09-10 20:12:20.389 UTC [28] LOG: database system was shut down at 2021-09-10 20:11:55 UTC
example_db_1 | 2021-09-10 20:12:20.396 UTC [1] LOG: database system is ready to accept connections
opal_server_1 | [2021-09-10 20:12:21 +0000] [1] [INFO] Starting gunicorn 20.1.0
opal_server_1 | [2021-09-10 20:12:21 +0000] [1] [INFO] Listening at: http://0.0.0.0:7002 (1)
opal_server_1 | [2021-09-10 20:12:21 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
opal_server_1 | [2021-09-10 20:12:21 +0000] [9] [INFO] Booting worker with pid: 9
opal_server_1 | [2021-09-10 20:12:21 +0000] [10] [INFO] Booting worker with pid: 10
opal_server_1 | [2021-09-10 20:12:21 +0000] [11] [INFO] Booting worker with pid: 11
opal_server_1 | [2021-09-10 20:12:21 +0000] [12] [INFO] Booting worker with pid: 12
opal_server_1 | 2021-09-10T20:12:22.469124+0000 |opal_common.authentication.signer | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!
opal_server_1 | 2021-09-10T20:12:22.469185+0000 |opal_common.authentication.signer | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!
opal_server_1 | 2021-09-10T20:12:22.469414+0000 |opal_common.authentication.signer | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!
opal_server_1 | 2021-09-10T20:12:22.475916+0000 |opal_common.authentication.signer | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!
opal_server_1 | 2021-09-10T20:12:22.531102+0000 |opal_server.server | INFO | triggered startup event
opal_server_1 | 2021-09-10T20:12:22.531362+0000 |opal_common.topics.publisher | INFO | started topic publisher
opal_server_1 | 2021-09-10T20:12:22.531265+0000 |opal_server.server | INFO | triggered startup event
opal_server_1 | 2021-09-10T20:12:22.531640+0000 |opal_server.server | INFO | leadership lock acquired, leader pid: 11
opal_server_1 | 2021-09-10T20:12:22.531774+0000 |opal_server.server | INFO | listening on webhook topic: 'webhook'
opal_server_1 | 2021-09-10T20:12:22.531738+0000 |opal_common.topics.publisher | INFO | started topic publisher
opal_server_1 | 2021-09-10T20:12:22.531933+0000 |opal_server.server | INFO | triggered startup event
opal_server_1 | 2021-09-10T20:12:22.532309+0000 |opal_common.topics.publisher | INFO | started topic publisher
opal_server_1 | 2021-09-10T20:12:22.532634+0000 |fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': '598b3711bd6b458392bd973d3ac3b41d', 'subscriber_id': 'd8043eb8ebd046b186a03558f3407340', 'topic': 'webhook', 'callback': functools.partial(<function trigger_repo_watcher_pull at 0x7fb143f22040>, <opal_server.policy.watcher.task.RepoWatcherTask object at 0x7fb143ea6a90>), 'notifier_id': None}
opal_server_1 | 2021-09-10T20:12:22.532820+0000 |opal_server.policy.watcher.task | INFO | Launching repo watcher
opal_server_1 | 2021-09-10T20:12:22.533394+0000 |opal_common.git.repo_cloner | INFO | Cloning repo from '[email protected]:Hongbo-Miao/test-opa-policy.git' to '/regoclone'
opal_server_1 | 2021-09-10T20:12:22.535781+0000 |opal_server.server | INFO | triggered startup event
opal_server_1 | 2021-09-10T20:12:22.535975+0000 |opal_common.topics.publisher | INFO | started topic publisher
opal_client_1 | [2021-09-10 20:12:22 +0000] [11] [INFO] Starting gunicorn 20.1.0
opal_client_1 | [2021-09-10 20:12:22 +0000] [11] [INFO] Listening at: http://0.0.0.0:7000 (11)
opal_client_1 | [2021-09-10 20:12:22 +0000] [11] [INFO] Using worker: uvicorn.workers.UvicornWorker
opal_client_1 | [2021-09-10 20:12:22 +0000] [13] [INFO] Booting worker with pid: 13
opal_client_1 | 2021-09-10T20:12:23.331080+0000 |opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'FastApiRpcFetchProvider' found at: <class 'opal_common.fetcher.providers.fastapi_rpc_fetch_provider.FastApiRpcFetchProvider'>
opal_client_1 | 2021-09-10T20:12:23.331600+0000 |opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'HttpFetchProvider' found at: <class 'opal_common.fetcher.providers.http_fetch_provider.HttpFetchProvider'>
opal_client_1 | 2021-09-10T20:12:23.358514+0000 |opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'PostgresFetchProvider' found at: <class 'opal_fetcher_postgres.provider.PostgresFetchProvider'>
opal_client_1 | 2021-09-10T20:12:23.358800+0000 |opal_common.fetcher.fetcher_register | INFO | Fetcher Register loaded
opal_client_1 | 2021-09-10T20:12:23.387910+0000 |opal_client.opa.runner | INFO | Launching opa runner
opal_client_1 | 2021-09-10T20:12:23.389526+0000 |opal_client.opa.runner | INFO | Running OPA inline: opa run --server --addr=:8181 --authentication=off --authorization=off --log-level=info
opal_client_1 | 2021-09-10T20:12:23.406463+0000 |opal_client.opa.logger | INFO | Initializing server. {'addrs': [':8181'], 'diagnostic-addrs': [], 'time': '2021-09-10T20:12:23Z'}
opal_client_1 | 2021-09-10T20:12:24.396792+0000 |opal_client.opa.runner | INFO | Running OPA initial start callbacks
opal_client_1 | 2021-09-10T20:12:24.397314+0000 |opal_client.data.updater | INFO | Launching data updater
opal_client_1 | 2021-09-10T20:12:24.397560+0000 |opal_client.policy.updater | INFO | Launching policy updater
opal_client_1 | 2021-09-10T20:12:24.397728+0000 |opal_client.data.updater | INFO | Subscribing to topics: ['policy_data']
opal_client_1 | 2021-09-10T20:12:24.397961+0000 |opal_client.policy.updater | INFO | Subscribing to topics: ['policy:.']
opal_client_1 | 2021-09-10T20:12:24.398360+0000 |fastapi_websocket_pubsub.pub_sub_client | INFO | Trying to connect to Pub/Sub server - ws://opal_server:7002/ws
opal_client_1 | 2021-09-10T20:12:24.399210+0000 |fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal_server:7002/ws
opal_client_1 | 2021-09-10T20:12:24.403194+0000 |fastapi_websocket_pubsub.pub_sub_client | INFO | Trying to connect to Pub/Sub server - ws://opal_server:7002/ws
opal_client_1 | 2021-09-10T20:12:24.403589+0000 |fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal_server:7002/ws
opal_server_1 | 2021-09-10T20:12:24.411522+0000 |fastapi_websocket_pubsub.event_broadc...| INFO | Listening for incoming events from broadcast channel (first listener started)
opal_server_1 | 2021-09-10T20:12:24.412121+0000 |fastapi_websocket_pubsub.event_broadc...| INFO | Spawning broadcast listen task
opal_server_1 | 2021-09-10T20:12:24.413270+0000 |fastapi_websocket_pubsub.event_broadc...| INFO | Listening for incoming events from broadcast channel (first listener started)
opal_server_1 | 2021-09-10T20:12:24.413571+0000 |fastapi_websocket_pubsub.event_broadc...| INFO | Spawning broadcast listen task
opal_server_1 | 2021-09-10T20:12:24.446157+0000 |fastapi_websocket_pubsub.event_broadc...| INFO | Subscribing to ALL TOPICS, and sharing messages with broadcast channel
opal_server_1 | 2021-09-10T20:12:24.447245+0000 |fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': 'ffb64af922b3454f97c741c3d8d50190', 'subscriber_id': 'ef97ec8a954f4fcb9c270e234afcb49b', 'topic': '__EventNotifier_ALL_TOPICS__', 'callback': <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_websocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7fb143e544c0>>, 'notifier_id': None}
opal_server_1 | 2021-09-10T20:12:24.447584+0000 |fastapi_websocket_pubsub.event_broadc...| INFO | Subscribing to ALL TOPICS, and sharing messages with broadcast channel
opal_server_1 | 2021-09-10T20:12:24.447863+0000 |fastapi_websocket_rpc.websocket_rpc_e...| INFO | Client connected
opal_server_1 | 2021-09-10T20:12:24.448284+0000 |fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': '0c55f3933ba54872b9491d756608c300', 'subscriber_id': 'd900f3bc5cb14831bd619fb5df4c687e', 'topic': '__EventNotifier_ALL_TOPICS__', 'callback': <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_websocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7fb143e9b910>>, 'notifier_id': None}
opal_server_1 | 2021-09-10T20:12:24.448205+0000 |fastapi_websocket_pubsub.event_broadc...| INFO | Starting broadcaster listener
opal_server_1 | 2021-09-10T20:12:24.448578+0000 |fastapi_websocket_rpc.websocket_rpc_e...| INFO | Client connected
opal_server_1 | 2021-09-10T20:12:24.448857+0000 |fastapi_websocket_pubsub.event_broadc...| INFO | Starting broadcaster listener
opal_server_1 | 2021-09-10T20:12:24.482818+0000 |fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': 'c6b6ec203e7444c483aa02f4ae618253', 'subscriber_id': '9b62c07247884b85aae0d70bfda6e46d', 'topic': 'policy:.', 'callback': <function RpcEventServerMethods.subscribe.<locals>.callback at 0x7fb14392e700>, 'notifier_id': None}
opal_server_1 | 2021-09-10T20:12:24.485300+0000 |fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': 'c2f49cc15b2443cdac0704ccf75bd493', 'subscriber_id': '34c0d20eeaf049d1b799643ef07b54bc', 'topic': 'policy_data', 'callback': <function RpcEventServerMethods.subscribe.<locals>.callback at 0x7fb143923700>, 'notifier_id': None}
opal_client_1 | 2021-09-10T20:12:24.487855+0000 |opal_client.policy.updater | INFO | Connected to server
opal_client_1 | 2021-09-10T20:12:24.488128+0000 |opal_client.policy.updater | INFO | Refetching policy code (full bundle)
opal_client_1 | 2021-09-10T20:12:24.492840+0000 |opal_client.data.updater | INFO | Connected to server
opal_client_1 | 2021-09-10T20:12:24.493042+0000 |opal_client.data.updater | INFO | Performing data configuration, reason: Initial load
opal_client_1 | 2021-09-10T20:12:24.493137+0000 |opal_client.data.updater | INFO | Getting data-sources configuration from 'http://opal_server:7002/data/config'
opal_server_1 | 2021-09-10T20:12:24.497346+0000 |opal_server.data.api | INFO | Serving source configuration
opal_server_1 | 2021-09-10T20:12:24.498632+0000 |uvicorn.protocols.http.httptools_impl | INFO | 172.25.0.5:49640 - "GET /data/config HTTP/1.1" 200
opal_client_1 | 2021-09-10T20:12:24.500526+0000 |opal_client.data.updater | INFO | Triggering data update with id: 915b4daab96f465f9338f5d84663faf5
opal_client_1 | 2021-09-10T20:12:24.500758+0000 |opal_client.data.updater | INFO | Fetching policy data
opal_client_1 | 2021-09-10T20:12:24.500949+0000 |opal_client.data.fetcher | INFO | Fetching data from url: postgresql://postgres@example_db:5432/postgres
opal_server_1 | 2021-09-10T20:12:24.496140+0000 |opal_common.middleware |ERROR | Uncaught server exception: Reference at 'refs/heads/master' does not exist
opal_server_1 | Traceback (most recent call last):
opal_server_1 |
opal_server_1 | File "/root/.local/bin/gunicorn", line 8, in <module>
opal_server_1 | sys.exit(run())
opal_server_1 | │ │ └ <function run at 0x7fb1461173a0>
opal_server_1 | │ └ <built-in function exit>
opal_server_1 | └ <module 'sys' (built-in)>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
opal_server_1 | WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
opal_server_1 | └ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
opal_server_1 | super().run()
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
opal_server_1 | Arbiter(self).run()
opal_server_1 | │ └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x7fb146124850>
opal_server_1 | └ <class 'gunicorn.arbiter.Arbiter'>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 202, in run
opal_server_1 | self.manage_workers()
opal_server_1 | │ └ <function Arbiter.manage_workers at 0x7fb145a63ee0>
opal_server_1 | └ <gunicorn.arbiter.Arbiter object at 0x7fb146124130>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
opal_server_1 | self.spawn_workers()
opal_server_1 | │ └ <function Arbiter.spawn_workers at 0x7fb145a64040>
opal_server_1 | └ <gunicorn.arbiter.Arbiter object at 0x7fb146124130>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
opal_server_1 | self.spawn_worker()
opal_server_1 | │ └ <function Arbiter.spawn_worker at 0x7fb145a63f70>
opal_server_1 | └ <gunicorn.arbiter.Arbiter object at 0x7fb146124130>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
opal_server_1 | worker.init_process()
opal_server_1 | │ └ <function UvicornWorker.init_process at 0x7fb1453bb940>
opal_server_1 | └ <uvicorn.workers.UvicornWorker object at 0x7fb145664cd0>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 64, in init_process
opal_server_1 | super(UvicornWorker, self).init_process()
opal_server_1 | │ └ <uvicorn.workers.UvicornWorker object at 0x7fb145664cd0>
opal_server_1 | └ <class 'uvicorn.workers.UvicornWorker'>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 142, in init_process
opal_server_1 | self.run()
opal_server_1 | │ └ <function UvicornWorker.run at 0x7fb1453bba60>
opal_server_1 | └ <uvicorn.workers.UvicornWorker object at 0x7fb145664cd0>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 77, in run
opal_server_1 | loop.run_until_complete(server.serve(sockets=self.sockets))
opal_server_1 | │ │ │ │ │ └ [<gunicorn.sock.TCPSocket object at 0x7fb1451218e0>]
opal_server_1 | │ │ │ │ └ <uvicorn.workers.UvicornWorker object at 0x7fb145664cd0>
opal_server_1 | │ │ │ └ <function Server.serve at 0x7fb145286940>
opal_server_1 | │ │ └ <uvicorn.server.Server object at 0x7fb1454defd0>
opal_server_1 | │ └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
opal_server_1 | └ <uvloop.Loop running=True closed=False debug=False>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 371, in run_asgi
opal_server_1 | result = await app(self.scope, self.receive, self.send)
opal_server_1 | │ │ │ │ │ │ └ <function RequestResponseCycle.send at 0x7fb143ce5b80>
opal_server_1 | │ │ │ │ │ └ <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb143949610>
opal_server_1 | │ │ │ │ └ <function RequestResponseCycle.receive at 0x7fb143ce5c10>
opal_server_1 | │ │ │ └ <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb143949610>
opal_server_1 | │ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.25.0.4', 7002), 'c...
opal_server_1 | │ └ <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb143949610>
opal_server_1 | └ <uvicorn.middleware.proxy_headers.ProxyHeadersMiddleware object at 0x7fb143e2e460>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 59, in __call__
opal_server_1 | return await self.app(scope, receive, send)
opal_server_1 | │ │ │ │ └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb1439496...
opal_server_1 | │ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb1439...
opal_server_1 | │ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.25.0.4', 7002), 'c...
opal_server_1 | │ └ <fastapi.applications.FastAPI object at 0x7fb143e9bd00>
opal_server_1 | └ <uvicorn.middleware.proxy_headers.ProxyHeadersMiddleware object at 0x7fb143e2e460>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
opal_server_1 | await super().__call__(scope, receive, send)
opal_server_1 | │ │ └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb1439496...
opal_server_1 | │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb1439...
opal_server_1 | └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.25.0.4', 7002), 'c...
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
opal_server_1 | await self.middleware_stack(scope, receive, send)
opal_server_1 | │ │ │ │ └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb1439496...
opal_server_1 | │ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb1439...
opal_server_1 | │ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.25.0.4', 7002), 'c...
opal_server_1 | │ └ <starlette.middleware.errors.ServerErrorMiddleware object at 0x7fb143eb0280>
opal_server_1 | └ <fastapi.applications.FastAPI object at 0x7fb143e9bd00>
opal_server_1 | > File "/root/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
opal_server_1 | await self.app(scope, receive, _send)
opal_server_1 | │ │ │ │ └ <function ServerErrorMiddleware.__call__.<locals>._send at 0x7fb14392ec10>
opal_server_1 | │ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb1439...
opal_server_1 | │ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.25.0.4', 7002), 'c...
opal_server_1 | │ └ <starlette.middleware.cors.CORSMiddleware object at 0x7fb143eb00d0>
opal_server_1 | └ <starlette.middleware.errors.ServerErrorMiddleware object at 0x7fb143eb0280>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 78, in __call__
opal_server_1 | await self.app(scope, receive, send)
opal_server_1 | │ │ │ │ └ <function ServerErrorMiddleware.__call__.<locals>._send at 0x7fb14392ec10>
opal_server_1 | │ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb1439...
opal_server_1 | │ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.25.0.4', 7002), 'c...
opal_server_1 | │ └ <starlette.exceptions.ExceptionMiddleware object at 0x7fb143eb01f0>
opal_server_1 | └ <starlette.middleware.cors.CORSMiddleware object at 0x7fb143eb00d0>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
opal_server_1 | raise exc from None
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
opal_server_1 | await self.app(scope, receive, sender)
opal_server_1 | │ │ │ │ └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x7fb14392eca0>
opal_server_1 | │ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb1439...
opal_server_1 | │ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.25.0.4', 7002), 'c...
opal_server_1 | │ └ <fastapi.routing.APIRouter object at 0x7fb143e9bcd0>
opal_server_1 | └ <starlette.exceptions.ExceptionMiddleware object at 0x7fb143eb01f0>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/routing.py", line 580, in __call__
opal_server_1 | await route.handle(scope, receive, send)
opal_server_1 | │ │ │ │ └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x7fb14392eca0>
opal_server_1 | │ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb1439...
opal_server_1 | │ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.25.0.4', 7002), 'c...
opal_server_1 | │ └ <function Route.handle at 0x7fb1448fde50>
opal_server_1 | └ <fastapi.routing.APIRoute object at 0x7fb143eb0640>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/routing.py", line 241, in handle
opal_server_1 | await self.app(scope, receive, send)
opal_server_1 | │ │ │ │ └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x7fb14392eca0>
opal_server_1 | │ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb1439...
opal_server_1 | │ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.25.0.4', 7002), 'c...
opal_server_1 | │ └ <function request_response.<locals>.app at 0x7fb143ec2ca0>
opal_server_1 | └ <fastapi.routing.APIRoute object at 0x7fb143eb0640>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/routing.py", line 52, in app
opal_server_1 | response = await func(request)
opal_server_1 | │ └ <starlette.requests.Request object at 0x7fb143949820>
opal_server_1 | └ <function get_request_handler.<locals>.app at 0x7fb143d680d0>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/fastapi/routing.py", line 204, in app
opal_server_1 | solved_result = await solve_dependencies(
opal_server_1 | └ <function solve_dependencies at 0x7fb14496d4c0>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/fastapi/dependencies/utils.py", line 548, in solve_dependencies
opal_server_1 | solved = await call(**sub_values)
opal_server_1 | │ └ {'repo': <git.repo.base.Repo '/regoclone/.git'>, 'paths': ['.']}
opal_server_1 | └ <function get_input_paths_or_throw at 0x7fb143eed040>
opal_server_1 |
opal_server_1 | File "/opal_server/policy/bundles/api.py", line 43, in get_input_paths_or_throw
opal_server_1 | with CommitViewer(repo.head.commit) as viewer:
opal_server_1 | │ │ └ <property object at 0x7fb143e80810>
opal_server_1 | │ └ <git.repo.base.Repo '/regoclone/.git'>
opal_server_1 | └ <class 'opal_common.git.commit_viewer.CommitViewer'>
opal_server_1 |
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 197, in _get_commit
opal_server_1 | obj = self._get_object()
opal_server_1 | │ └ <function SymbolicReference._get_object at 0x7fb143f26160>
opal_server_1 | └ <git.HEAD "HEAD">
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 190, in _get_object
opal_server_1 | return Object.new_from_sha(self.repo, hex_to_bin(self.dereference_recursive(self.repo, self.path)))
opal_server_1 | │ │ │ │ │ │ │ │ │ │ └ <member 'path' of 'SymbolicReference' objects>
opal_server_1 | │ │ │ │ │ │ │ │ │ └ <git.HEAD "HEAD">
opal_server_1 | │ │ │ │ │ │ │ │ └ <member 'repo' of 'SymbolicReference' objects>
opal_server_1 | │ │ │ │ │ │ │ └ <git.HEAD "HEAD">
opal_server_1 | │ │ │ │ │ │ └ <classmethod object at 0x7fb143f99700>
opal_server_1 | │ │ │ │ │ └ <git.HEAD "HEAD">
opal_server_1 | │ │ │ │ └ <built-in function a2b_hex>
opal_server_1 | │ │ │ └ <member 'repo' of 'SymbolicReference' objects>
opal_server_1 | │ │ └ <git.HEAD "HEAD">
opal_server_1 | │ └ <classmethod object at 0x7fb143d6ff70>
opal_server_1 | └ <class 'git.objects.base.Object'>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 132, in dereference_recursive
opal_server_1 | hexsha, ref_path = cls._get_ref_info(repo, ref_path)
opal_client_1 | 2021-09-10T20:12:24.518549+0000 |fastapi_websocket_pubsub.pub_sub_client | INFO | Connected to PubSub server ws://opal_server:7002/ws
opal_server_1 | │ │ │ │ └ 'refs/heads/master'
opal_server_1 | │ │ │ └ <git.repo.base.Repo '/regoclone/.git'>
opal_client_1 | 2021-09-10T20:12:24.519709+0000 |opal_client.policy.fetcher |WARNING | Unexpected response code 500: {'error': 'Uncaught server exception'}
opal_server_1 | │ │ └ <classmethod object at 0x7fb143f99730>
opal_server_1 | │ └ <class 'git.refs.head.HEAD'>
opal_server_1 | └ None
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 181, in _get_ref_info
opal_server_1 | return cls._get_ref_info_helper(repo, ref_path)
opal_server_1 | │ │ │ └ 'refs/heads/master'
opal_server_1 | │ │ └ <git.repo.base.Repo '/regoclone/.git'>
opal_server_1 | │ └ <classmethod object at 0x7fb143f99760>
opal_server_1 | └ <class 'git.refs.head.HEAD'>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 164, in _get_ref_info_helper
opal_server_1 | raise ValueError("Reference at %r does not exist" % ref_path)
opal_server_1 | └ 'refs/heads/master'
opal_server_1 |
opal_server_1 | ValueError: Reference at 'refs/heads/master' does not exist
opal_server_1 | 2021-09-10T20:12:24.511752+0000 |uvicorn.protocols.http.httptools_impl | INFO | 172.25.0.5:49638 - "GET /policy?path=. HTTP/1.1" 500
opal_server_1 | Exception in ASGI application
opal_server_1 | Traceback (most recent call last):
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 371, in run_asgi
opal_server_1 | result = await app(self.scope, self.receive, self.send)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 59, in __call__
opal_server_1 | return await self.app(scope, receive, send)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
opal_server_1 | await super().__call__(scope, receive, send)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
opal_server_1 | await self.middleware_stack(scope, receive, send)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
opal_server_1 | raise exc from None
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
opal_server_1 | await self.app(scope, receive, _send)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 78, in __call__
opal_server_1 | await self.app(scope, receive, send)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
opal_server_1 | raise exc from None
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
opal_server_1 | await self.app(scope, receive, sender)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/routing.py", line 580, in __call__
opal_server_1 | await route.handle(scope, receive, send)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/routing.py", line 241, in handle
opal_server_1 | await self.app(scope, receive, send)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/routing.py", line 52, in app
opal_server_1 | response = await func(request)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/fastapi/routing.py", line 204, in app
opal_server_1 | solved_result = await solve_dependencies(
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/fastapi/dependencies/utils.py", line 548, in solve_dependencies
opal_server_1 | solved = await call(**sub_values)
opal_server_1 | File "/opal_server/policy/bundles/api.py", line 43, in get_input_paths_or_throw
opal_server_1 | with CommitViewer(repo.head.commit) as viewer:
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 197, in _get_commit
opal_server_1 | obj = self._get_object()
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 190, in _get_object
opal_server_1 | return Object.new_from_sha(self.repo, hex_to_bin(self.dereference_recursive(self.repo, self.path)))
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 132, in dereference_recursive
opal_server_1 | hexsha, ref_path = cls._get_ref_info(repo, ref_path)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 181, in _get_ref_info
opal_server_1 | return cls._get_ref_info_helper(repo, ref_path)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 164, in _get_ref_info_helper
opal_server_1 | raise ValueError("Reference at %r does not exist" % ref_path)
opal_server_1 | ValueError: Reference at 'refs/heads/master' does not exist
opal_client_1 | 2021-09-10T20:12:24.535320+0000 |opal_client.data.updater | INFO | Saving fetched data to policy-store: source url='postgresql://postgres@example_db:5432/postgres', destination path='/cities'
opal_client_1 | 2021-09-10T20:12:24.540133+0000 |opal_client.opa.logger | INFO | Received request. PUT /v1/data/cities
opal_client_1 | 2021-09-10T20:12:24.540594+0000 |opal_client.opa.logger | INFO | Sent response. PUT /v1/data/cities -> 204
opal_server_1 | 2021-09-10T20:12:25.521378+0000 |opal_common.middleware |ERROR | Uncaught server exception: Reference at 'refs/heads/master' does not exist
opal_server_1 | Traceback (most recent call last):
opal_server_1 |
opal_server_1 | File "/root/.local/bin/gunicorn", line 8, in <module>
opal_server_1 | sys.exit(run())
opal_server_1 | │ │ └ <function run at 0x7fb1461173a0>
opal_server_1 | │ └ <built-in function exit>
opal_server_1 | └ <module 'sys' (built-in)>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
opal_server_1 | WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
opal_server_1 | └ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
opal_server_1 | super().run()
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
opal_server_1 | Arbiter(self).run()
opal_server_1 | │ └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x7fb146124850>
opal_server_1 | └ <class 'gunicorn.arbiter.Arbiter'>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 202, in run
opal_server_1 | self.manage_workers()
opal_server_1 | │ └ <function Arbiter.manage_workers at 0x7fb145a63ee0>
opal_server_1 | └ <gunicorn.arbiter.Arbiter object at 0x7fb146124130>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
opal_server_1 | self.spawn_workers()
opal_server_1 | │ └ <function Arbiter.spawn_workers at 0x7fb145a64040>
opal_server_1 | └ <gunicorn.arbiter.Arbiter object at 0x7fb146124130>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
opal_server_1 | self.spawn_worker()
opal_server_1 | │ └ <function Arbiter.spawn_worker at 0x7fb145a63f70>
opal_server_1 | └ <gunicorn.arbiter.Arbiter object at 0x7fb146124130>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
opal_server_1 | worker.init_process()
opal_server_1 | │ └ <function UvicornWorker.init_process at 0x7fb1453bb940>
opal_server_1 | └ <uvicorn.workers.UvicornWorker object at 0x7fb1454e5790>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 64, in init_process
opal_server_1 | super(UvicornWorker, self).init_process()
opal_server_1 | │ └ <uvicorn.workers.UvicornWorker object at 0x7fb1454e5790>
opal_server_1 | └ <class 'uvicorn.workers.UvicornWorker'>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 142, in init_process
opal_server_1 | self.run()
opal_server_1 | │ └ <function UvicornWorker.run at 0x7fb1453bba60>
opal_server_1 | └ <uvicorn.workers.UvicornWorker object at 0x7fb1454e5790>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 77, in run
opal_server_1 | loop.run_until_complete(server.serve(sockets=self.sockets))
opal_server_1 | │ │ │ │ │ └ [<gunicorn.sock.TCPSocket object at 0x7fb1451218e0>]
opal_server_1 | │ │ │ │ └ <uvicorn.workers.UvicornWorker object at 0x7fb1454e5790>
opal_server_1 | │ │ │ └ <function Server.serve at 0x7fb145286940>
opal_server_1 | │ │ └ <uvicorn.server.Server object at 0x7fb1453af4c0>
opal_server_1 | │ └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
opal_server_1 | └ <uvloop.Loop running=True closed=False debug=False>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 371, in run_asgi
opal_server_1 | result = await app(self.scope, self.receive, self.send)
opal_server_1 | │ │ │ │ │ │ └ <function RequestResponseCycle.send at 0x7fb143e85c10>
opal_server_1 | │ │ │ │ │ └ <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb143cc8040>
opal_server_1 | │ │ │ │ └ <function RequestResponseCycle.receive at 0x7fb143e85ca0>
opal_server_1 | │ │ │ └ <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb143cc8040>
opal_server_1 | │ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.25.0.4', 7002), 'c...
opal_server_1 | │ └ <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb143cc8040>
opal_server_1 | └ <uvicorn.middleware.proxy_headers.ProxyHeadersMiddleware object at 0x7fb143e700d0>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 59, in __call__
opal_server_1 | return await self.app(scope, receive, send)
opal_server_1 | │ │ │ │ └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb143cc80...
opal_server_1 | │ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb143c...
opal_server_1 | │ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.25.0.4', 7002), 'c...
opal_server_1 | │ └ <fastapi.applications.FastAPI object at 0x7fb143ed6c70>
opal_server_1 | └ <uvicorn.middleware.proxy_headers.ProxyHeadersMiddleware object at 0x7fb143e700d0>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
opal_server_1 | await super().__call__(scope, receive, send)
opal_server_1 | │ │ └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb143cc80...
opal_server_1 | │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb143c...
opal_server_1 | └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.25.0.4', 7002), 'c...
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
opal_server_1 | await self.middleware_stack(scope, receive, send)
opal_server_1 | │ │ │ │ └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb143cc80...
opal_server_1 | │ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb143c...
opal_server_1 | │ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.25.0.4', 7002), 'c...
opal_server_1 | │ └ <starlette.middleware.errors.ServerErrorMiddleware object at 0x7fb143ee91c0>
opal_server_1 | └ <fastapi.applications.FastAPI object at 0x7fb143ed6c70>
opal_server_1 | > File "/root/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
opal_server_1 | await self.app(scope, receive, _send)
opal_client_1 | 2021-09-10T20:12:25.537716+0000 |opal_client.policy.fetcher |WARNING | Unexpected response code 500: {'error': 'Uncaught server exception'}
opal_server_1 | │ │ │ │ └ <function ServerErrorMiddleware.__call__.<locals>._send at 0x7fb143cadc10>
opal_server_1 | │ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb143c...
opal_server_1 | │ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.25.0.4', 7002), 'c...
opal_server_1 | │ └ <starlette.middleware.cors.CORSMiddleware object at 0x7fb143ee90a0>
opal_server_1 | └ <starlette.middleware.errors.ServerErrorMiddleware object at 0x7fb143ee91c0>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 78, in __call__
opal_server_1 | await self.app(scope, receive, send)
opal_server_1 | │ │ │ │ └ <function ServerErrorMiddleware.__call__.<locals>._send at 0x7fb143cadc10>
opal_server_1 | │ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb143c...
opal_server_1 | │ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.25.0.4', 7002), 'c...
opal_server_1 | │ └ <starlette.exceptions.ExceptionMiddleware object at 0x7fb143ee9130>
opal_server_1 | └ <starlette.middleware.cors.CORSMiddleware object at 0x7fb143ee90a0>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
opal_server_1 | raise exc from None
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
opal_server_1 | await self.app(scope, receive, sender)
opal_server_1 | │ │ │ │ └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x7fb143cade50>
opal_server_1 | │ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb143c...
opal_server_1 | │ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.25.0.4', 7002), 'c...
opal_server_1 | │ └ <fastapi.routing.APIRouter object at 0x7fb143ed6c40>
opal_server_1 | └ <starlette.exceptions.ExceptionMiddleware object at 0x7fb143ee9130>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/routing.py", line 580, in __call__
opal_server_1 | await route.handle(scope, receive, send)
opal_server_1 | │ │ │ │ └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x7fb143cade50>
opal_server_1 | │ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb143c...
opal_server_1 | │ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.25.0.4', 7002), 'c...
opal_server_1 | │ └ <function Route.handle at 0x7fb144774ee0>
opal_server_1 | └ <fastapi.routing.APIRoute object at 0x7fb143ee9580>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/routing.py", line 241, in handle
opal_server_1 | await self.app(scope, receive, send)
opal_server_1 | │ │ │ │ └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x7fb143cade50>
opal_server_1 | │ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb143c...
opal_server_1 | │ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('172.25.0.4', 7002), 'c...
opal_server_1 | │ └ <function request_response.<locals>.app at 0x7fb143f00d30>
opal_server_1 | └ <fastapi.routing.APIRoute object at 0x7fb143ee9580>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/routing.py", line 52, in app
opal_server_1 | response = await func(request)
opal_server_1 | │ └ <starlette.requests.Request object at 0x7fb143cc8250>
opal_server_1 | └ <function get_request_handler.<locals>.app at 0x7fb143efc940>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/fastapi/routing.py", line 204, in app
opal_server_1 | solved_result = await solve_dependencies(
opal_server_1 | └ <function solve_dependencies at 0x7fb144764550>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/fastapi/dependencies/utils.py", line 548, in solve_dependencies
opal_server_1 | solved = await call(**sub_values)
opal_server_1 | │ └ {'repo': <git.repo.base.Repo '/regoclone/.git'>, 'paths': ['.']}
opal_server_1 | └ <function get_input_paths_or_throw at 0x7fb143eab040>
opal_server_1 |
opal_server_1 | File "/opal_server/policy/bundles/api.py", line 43, in get_input_paths_or_throw
opal_server_1 | with CommitViewer(repo.head.commit) as viewer:
opal_server_1 | │ │ └ <property object at 0x7fb143f3e590>
opal_server_1 | │ └ <git.repo.base.Repo '/regoclone/.git'>
opal_server_1 | └ <class 'opal_common.git.commit_viewer.CommitViewer'>
opal_server_1 |
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 197, in _get_commit
opal_server_1 | obj = self._get_object()
opal_server_1 | │ └ <function SymbolicReference._get_object at 0x7fb143ce8160>
opal_server_1 | └ <git.HEAD "HEAD">
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 190, in _get_object
opal_server_1 | return Object.new_from_sha(self.repo, hex_to_bin(self.dereference_recursive(self.repo, self.path)))
opal_server_1 | │ │ │ │ │ │ │ │ │ │ └ <member 'path' of 'SymbolicReference' objects>
opal_server_1 | │ │ │ │ │ │ │ │ │ └ <git.HEAD "HEAD">
opal_server_1 | │ │ │ │ │ │ │ │ └ <member 'repo' of 'SymbolicReference' objects>
opal_server_1 | │ │ │ │ │ │ │ └ <git.HEAD "HEAD">
opal_server_1 | │ │ │ │ │ │ └ <classmethod object at 0x7fb143f99790>
opal_server_1 | │ │ │ │ │ └ <git.HEAD "HEAD">
opal_server_1 | │ │ │ │ └ <built-in function a2b_hex>
opal_server_1 | │ │ │ └ <member 'repo' of 'SymbolicReference' objects>
opal_server_1 | │ │ └ <git.HEAD "HEAD">
opal_server_1 | │ └ <classmethod object at 0x7fb143d760a0>
opal_server_1 | └ <class 'git.objects.base.Object'>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 132, in dereference_recursive
opal_server_1 | hexsha, ref_path = cls._get_ref_info(repo, ref_path)
opal_server_1 | │ │ │ │ └ 'refs/heads/master'
opal_server_1 | │ │ │ └ <git.repo.base.Repo '/regoclone/.git'>
opal_server_1 | │ │ └ <classmethod object at 0x7fb143f997c0>
opal_server_1 | │ └ <class 'git.refs.head.HEAD'>
opal_server_1 | └ None
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 181, in _get_ref_info
opal_server_1 | return cls._get_ref_info_helper(repo, ref_path)
opal_server_1 | │ │ │ └ 'refs/heads/master'
opal_server_1 | │ │ └ <git.repo.base.Repo '/regoclone/.git'>
opal_server_1 | │ └ <classmethod object at 0x7fb143f997f0>
opal_server_1 | └ <class 'git.refs.head.HEAD'>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 164, in _get_ref_info_helper
opal_server_1 | raise ValueError("Reference at %r does not exist" % ref_path)
opal_server_1 | └ 'refs/heads/master'
opal_server_1 |
opal_server_1 | ValueError: Reference at 'refs/heads/master' does not exist
opal_server_1 | 2021-09-10T20:12:25.536392+0000 |uvicorn.protocols.http.httptools_impl | INFO | 172.25.0.5:49646 - "GET /policy?path=. HTTP/1.1" 500
opal_server_1 | Exception in ASGI application
opal_server_1 | Traceback (most recent call last):
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 371, in run_asgi
opal_server_1 | result = await app(self.scope, self.receive, self.send)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 59, in __call__
opal_server_1 | return await self.app(scope, receive, send)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
opal_server_1 | await super().__call__(scope, receive, send)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
opal_server_1 | await self.middleware_stack(scope, receive, send)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
opal_server_1 | raise exc from None
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
opal_server_1 | await self.app(scope, receive, _send)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 78, in __call__
opal_server_1 | await self.app(scope, receive, send)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
opal_server_1 | raise exc from None
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
opal_server_1 | await self.app(scope, receive, sender)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/routing.py", line 580, in __call__
opal_server_1 | await route.handle(scope, receive, send)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/routing.py", line 241, in handle
opal_server_1 | await self.app(scope, receive, send)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/starlette/routing.py", line 52, in app
opal_server_1 | response = await func(request)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/fastapi/routing.py", line 204, in app
opal_server_1 | solved_result = await solve_dependencies(
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/fastapi/dependencies/utils.py", line 548, in solve_dependencies
opal_server_1 | solved = await call(**sub_values)
opal_server_1 | File "/opal_server/policy/bundles/api.py", line 43, in get_input_paths_or_throw
opal_server_1 | with CommitViewer(repo.head.commit) as viewer:
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 197, in _get_commit
opal_server_1 | obj = self._get_object()
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 190, in _get_object
opal_server_1 | return Object.new_from_sha(self.repo, hex_to_bin(self.dereference_recursive(self.repo, self.path)))
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 132, in dereference_recursive
opal_server_1 | hexsha, ref_path = cls._get_ref_info(repo, ref_path)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 181, in _get_ref_info
opal_server_1 | return cls._get_ref_info_helper(repo, ref_path)
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 164, in _get_ref_info_helper
opal_server_1 | raise ValueError("Reference at %r does not exist" % ref_path)
opal_server_1 | ValueError: Reference at 'refs/heads/master' does not exist
opal_server_1 | 2021-09-10T20:12:26.376926+0000 |opal_common.git.repo_cloner | INFO | Clone succeeded
opal_server_1 | 2021-09-10T20:12:26.384872+0000 |opal_common.git.repo_watcher | INFO | Launching polling task, interval: 30 seconds
opal_server_1 | 2021-09-10T20:12:27.399577+0000 |uvicorn.protocols.http.httptools_impl | INFO | 172.25.0.5:49648 - "GET /policy?path=. HTTP/1.1" 200
opal_client_1 | 2021-09-10T20:12:27.401184+0000 |opal_client.policy.updater | INFO | got policy bundle, commit hash: '6d849b1ce92bae05c31146470e66e07c3c41b164'
opal_client_1 | 2021-09-10T20:12:27.404390+0000 |opal_client.opa.logger | INFO | Received request. GET /v1/policies
opal_client_1 | 2021-09-10T20:12:27.404678+0000 |opal_client.opa.logger | INFO | Sent response. GET /v1/policies -> 200
opal_client_1 | 2021-09-10T20:12:27.407538+0000 |opal_client.opa.logger | INFO | Received request. PUT /v1/data
opal_client_1 | 2021-09-10T20:12:27.408493+0000 |opal_client.opa.logger | INFO | Sent response. PUT /v1/data -> 204
opal_client_1 | 2021-09-10T20:12:27.410862+0000 |opal_client.opa.logger | INFO | Received request. PUT /v1/policies/rbac.rego
opal_client_1 | 2021-09-10T20:12:27.415721+0000 |opal_client.opa.logger | INFO | Sent response. PUT /v1/policies/rbac.rego -> 200
opal_client_1 | 2021-09-10T20:12:27.418303+0000 |opal_client.opa.logger | INFO | Received request. PUT /v1/policies/utils.rego
opal_client_1 | 2021-09-10T20:12:27.420669+0000 |opal_client.opa.logger | INFO | Sent response. PUT /v1/policies/utils.rego -> 200
opal_client_1 | 2021-09-10T20:12:27.421790+0000 |fastapi_websocket_pubsub.pub_sub_client | INFO | Connected to PubSub server ws://opal_server:7002/wsConnecting several OPAL clients to one server
Hello, I read in your documentation that you can connect several clients to one OPAL server, but I did not understand how this can be done. Could you please suggest how to do this or send a link to the documentation?
Check whether the opal-client has latest data
Discussed in https://github.com/authorizon/opal/discussions/152
(^ Please go to the discussion there to check more info)
Originally posted by Hongbo-Miao October 14, 2021
A similar question to Check whether the opal-client has latest policy, but for data instead of policy.
As for data, we send POST http://localhost:7002/data/config command to opal-server with HTTP body telling where it can pull the data. Once opal-client receives from opal-server, it will start to pull the data.
Is it possible for us to know
- When is the last time opal-client succeed pulling the data.
- If continues failing, when is the last time it tries.
I think this can solves by parsing opal-client logs. However, does it make sense to have some health APIs just like policy?
How to disable data fetching and only use policy syncing
Hello,
I'm interested in disabling the data fetching portion of OPAL and only using the policy syncing feature at this time. I've tried setting OPAL_DATA_CONFIG_SOURCES to {"entries":[]} on the server and OPAL_DATA_TOPICS to an empty string on the client, however the client continues attempting to sync data from the server, which fails due to some networking issues.
Is it possible to disable data fetching/syncing and only use OPAL's policy syncing feature?
Support nested manifest files
Support for dependency management using manifest files was added with https://github.com/authorizon/opal/pull/127
Currently, only a root level manifest is supported, and the exact file paths must be provided in that manifest file. This can be difficult to manage when policy data grows complex with many different files and folders.
I would like to suggest adding support for nested manifest files.
EX:
manifest
rules
|> manifest
|> rbac.rego
|> abac.rego
utils
|> manifest
|> utils.rego
|> moreutils
|> manifest
|> anotherUtil.rego
Where, in the above example, abac.rego depends on rbac.rego, everything in rules is dependent on the utils folder, and the utils folder is dependent on the utils/moreutils folder
Sample manifest contents can be seen below:
manifest:
utils
rules
utils/manifest:
moreutils
utils.rego
utils/moreutils/manifest:
anotherUtil.rego
rules/manifest:
rbac.rego
abac.rego
information request about : Repo already exists ... moving previous clone to
Hi
Thanks for your job. I start to use opal, it's look great.
(Sorry for my english in advance).
I see a strange behavior. RepoCloner._discard_previous_local_clone create a new repo clone each time server is run. If server reboot 10 time, there are 9 directories "regoclone.oldX".
- There are any reason to not update in place ?
- There ara any reason why it's never deleted, (max history or something like that) ?
- There are any reason why this not configurable ?
If it could be a new feature, may be i could make a pull request.
Another thing: it's coold be great (but not essential) to support the « --single-branch» for clone repo
Sincerely
Add option to limit directories in git repo on the server side
how it works now
At the moment OPAL server tracks a git repo for sourcing policy code (rego and static data) to its clients. The entire repo is exposed by default to the clients, and the clients decide what directories that want to subscribe to.
potential improvement
The opal server should potentially accept a config option to only track some directories on the source repo (i,e: have different roots) and only tracks them.
Enhancements to docs (Planned)
I thought i will open this issue to let whomever interested what are the pending improvements to docs we are planning to add this week :)
If you have more suggestions, please comment on this issue and i will take this into account.
Planned improvements
- complete the docker tutorial with a full example of how to deploy to production with all security parameters, currently this is only documented in the cli/library tutorial.
- "refactor" getting started tutorials, to better explain the different options and route the reader to the proper tutorial for him/her (i.e: getting to learn about opal and play around, deploying a container, using the library to potentially extend opal for your needs, etc).
- document for options to track policy git repo: using polling, using webhooks, tutorial with github.
- improve architecture tutorial: add diagrams or more detailed descriptions of the messages that goes between opal server and client, e.g: policy bundle format, etc.
- expand the core modules docs
POLICY_REPO_CLONE_PATH should default look up current working directory
$ opal-server print-config
...
"POLICY_REPO_CLONE_PATH": "/Users/furkan/regoclone"
My working dir: ~/src/poc/opal
$ OPAL_POLICY_REPO_URL=https://github.com/open-policy-agent/gatekeeper-library.git opal-server --policy-repo-polling-interval 60 run
...
Cloning repo from 'https://github.com/open-policy-agent/gatekeeper-library.git' to '/Users/furkan/regoclone'
It had better clone repo into current directory in order to keep clean out $HOME directory. What do you thing? 🤔
More data fetcher providers wanted
What are fetch providers?
FetchProviders are the components OPAL uses to fetch data from sources on demand.
Fetch providers are designed to be extendable, and you can easily create more fetch providers to enable OPAL to fetch data from your own unique sources (e.g. a SaaS service, a new DB, your own proprietary solution, ...)
We have a guide on how to write new fetch providers
We have a fully-working example fetch provider for Postgres
Help wanted
More data fetch providers are planned, but we would love the community to help and take an active part by developing them.
Ideas for missing providers: DBs from any kind (i.e: postgres, mongo, etc), popular 3rd party SaaS providers (i.e: stripe, salesforce, etc)
Reach out to us, we are happy to guide you in the process.
Default log configuration does not show client unable to connect
details here: #81
Need to change default log filters to include fastapi_websocket_pubsub connection errors.
thanks @hongbo-miao for floating this up.
access token was not provided when running opal-client
I run opal-server and opal-client.
This is the error I got on opal-client
2021-08-02T15:37:52.223415+0800 |opal_client.data.fetcher | INFO | [<ClientResponse(http://localhost:7002/policy-data) [401 Unauthorized]>
<CIMultiDictProxy('Date': 'Mon, 02 Aug 2021 07:37:52 GMT', 'Server': 'uvicorn', 'WWW-Authenticate': 'Bearer', 'Content-Length': '65', 'Content-Type': 'application/json')>
]
2021-08-02T15:37:52.223593+0800 |opal_client.data.updater |ERROR | Failed to fetch url http://localhost:7002/policy-data, got response code 401 with error: {'detail': {'error': 'access token was not provided', 'token': None}}
2021-08-02T15:37:52.953338+0800 |opal_client.client | INFO | stopping background tasks...
I did some checking on the code and edit it, the error went away.
opal_client/data/fetcher.py
async def handle_urls(self, urls: List[Tuple[str, FetcherConfig]] = None) -> List[Tuple[str, FetcherConfig, Any]]:
"""
Fetch data for each given url with the (optional) fetching configuration; return the resulting data mapped to each URL
Args:
urls (List[Tuple[str, FetcherConfig]], optional): Urls (and fetching configuration) to fetch from.
Defaults to None - init data_url with HttpFetcherConfig (loaded with the provided auth token).
Returns:
List[Tuple[str,FetcherConfig, Any]]: urls mapped to their resulting fetched data
"""
# tasks
tasks = []
# if no url provided - default to the builtin route
if urls is None:
urls = [(self._data_url, self._default_fetcher_config)]
# create a task for each url
for url, config in urls:
if not config:
config = self._default_fetcher_config
tasks.append(self.handle_url(url, config))
# wait for all data fetches to complete
results = await asyncio.gather(*tasks, return_exceptions=True)
logging.info(results)Please let me know if this is the intended behavior of the function in order for it to work?
OPAL_DEFAULT_UPDATE_CALLBACKS is not parsed correctly when provided with config
I have successfully set up the JWT token. This is my opal-client log which shows it works well.
[2021-10-21 20:30:06 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2021-10-21 20:30:06 +0000] [1] [INFO] Listening at: http://0.0.0.0:7000 (1)
[2021-10-21 20:30:06 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-10-21 20:30:06 +0000] [10] [INFO] Booting worker with pid: 10
2021-10-21T20:30:06.997648+0000 | opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'FastApiRpcFetchProvider' found at: <class 'opal_common.fetcher.providers.fastapi_rpc_fetch_provider.FastApiRpcFetchProvider'>
2021-10-21T20:30:06.998161+0000 | opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'HttpFetchProvider' found at: <class 'opal_common.fetcher.providers.http_fetch_provider.HttpFetchProvider'>
2021-10-21T20:30:07.028551+0000 | opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'PostgresFetchProvider' found at: <class 'opal_fetcher_postgres.provider.PostgresFetchProvider'>
2021-10-21T20:30:07.028762+0000 | opal_common.fetcher.fetcher_register | INFO | Fetcher Register loaded
2021-10-21T20:30:07.029683+0000 | opal_client.callbacks.register | INFO | Callbacks register loaded
2021-10-21T20:30:07.030053+0000 | opal_client.client | INFO | API authentication disabled (public encryption key was not provided)
2021-10-21T20:30:07.097858+0000 | uvicorn.server | INFO | Started server process [10]
2021-10-21T20:30:07.098212+0000 | uvicorn.lifespan.on | INFO | Waiting for application startup.
2021-10-21T20:30:07.098566+0000 | opal_client.opa.runner | INFO | Launching opa runner
2021-10-21T20:30:07.098808+0000 | uvicorn.lifespan.on | INFO | Application startup complete.
2021-10-21T20:30:07.101344+0000 | opal_client.opa.runner | INFO | Running OPA inline: opa run --server --addr=:8181 --authentication=off --authorization=off --config-file=/data/opa/config.yaml --log-level=info
2021-10-21T20:30:07.130883+0000 | opal_client.opa.logger | INFO | Initializing server. {'addrs': [':8181'], 'diagnostic-addrs': [], 'time': '2021-10-21T20:30:07Z'}
2021-10-21T20:30:07.132577+0000 | opal_client.opa.logger | INFO | Starting decision logger. {'plugin': 'decision_logs', 'time': '2021-10-21T20:30:07Z'}
2021-10-21T20:30:08.114086+0000 | opal_client.opa.runner | INFO | Running OPA initial start callbacks
2021-10-21T20:30:08.114794+0000 | opal_client.policy_store.opa_client | INFO | persisting health check policy: ready=false, healthy=false
2021-10-21T20:30:08.118031+0000 | opal_client.opa.logger | INFO | Received request. {'client_addr': '[::1]:60792', 'req_id': 1, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.119414+0000 | opal_client.opa.logger | INFO | Sent response. {'client_addr': '[::1]:60792', 'req_id': 1, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'resp_bytes': 2, 'resp_duration': 1.4131, 'resp_status': 200, 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.120468+0000 | opal_client.data.updater | INFO | Launching data updater
2021-10-21T20:30:08.120688+0000 | opal_client.policy.updater | INFO | Launching policy updater
2021-10-21T20:30:08.120808+0000 | opal_client.data.updater | INFO | Subscribing to topics: ['policy_data']
2021-10-21T20:30:08.121188+0000 | opal_client.policy.updater | INFO | Subscribing to topics: ['policy:.']
2021-10-21T20:30:08.121432+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO | Trying to connect to Pub/Sub server - ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:30:08.121708+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:30:08.124114+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO | Trying to connect to Pub/Sub server - ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:30:08.124399+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:30:08.157712+0000 | opal_client.data.updater | INFO | Connected to server
2021-10-21T20:30:08.157862+0000 | opal_client.data.updater | INFO | Performing data configuration, reason: Initial load
2021-10-21T20:30:08.158014+0000 | opal_client.data.updater | INFO | Getting data-sources configuration from 'http://opal-server-service.hm-opa:7002/data/config'
2021-10-21T20:30:08.161874+0000 | opal_client.policy.updater | INFO | Connected to server
2021-10-21T20:30:08.162482+0000 | opal_client.policy.updater | INFO | Refetching policy code (full bundle)
2021-10-21T20:30:08.169424+0000 | opal_client.data.updater | INFO | Triggering data update with id: 5bd1936ae65146648788dd6081c79b2b
2021-10-21T20:30:08.170344+0000 | opal_client.data.updater | INFO | Fetching policy data
2021-10-21T20:30:08.170627+0000 | opal_client.data.fetcher | INFO | Fetching data from url: postgresql://[email protected]:40072/opa_db
2021-10-21T20:30:08.193387+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO | Connected to PubSub server ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:30:08.225411+0000 | opal_client.policy.updater | INFO | Got policy bundle with 1 rego files, 0 data files, commit hash: '910d43e7e4da262d838c562ce3ae9910600fe445'
2021-10-21T20:30:08.228156+0000 | opal_client.data.updater | INFO | Saving fetched data to policy-store: source url='postgresql://[email protected]:40072/opa_db', destination path='/roles'
2021-10-21T20:30:08.234055+0000 | opal_client.opa.logger | INFO | Received request. {'client_addr': '[::1]:60808', 'req_id': 2, 'req_method': 'GET', 'req_path': '/v1/policies', 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.234757+0000 | opal_client.opa.logger | INFO | Received request. {'client_addr': '[::1]:60810', 'req_id': 3, 'req_method': 'PUT', 'req_path': '/v1/data/roles', 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.239404+0000 | opal_client.opa.logger | INFO | Sent response. {'client_addr': '[::1]:60808', 'req_id': 2, 'req_method': 'GET', 'req_path': '/v1/policies', 'resp_bytes': 4603, 'resp_duration': 1.7899, 'resp_status': 200, 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.239731+0000 | opal_client.opa.logger | INFO | Sent response. {'client_addr': '[::1]:60810', 'req_id': 3, 'req_method': 'PUT', 'req_path': '/v1/data/roles', 'resp_bytes': 0, 'resp_duration': 1.3754, 'resp_status': 204, 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.240629+0000 | opal_client.policy_store.opa_client | INFO | processing store transaction: {'id': '5bd1936ae65146648788dd6081c79b2b', 'actions': ['set_policy_data'], 'success': True, 'error': ''}
2021-10-21T20:30:08.240791+0000 | opal_client.policy_store.opa_client | INFO | persisting health check policy: ready=false, healthy=false
2021-10-21T20:30:08.243687+0000 | opal_client.opa.logger | INFO | Received request. {'client_addr': '[::1]:60812', 'req_id': 4, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.244179+0000 | opal_client.opa.logger | INFO | Received request. {'client_addr': '[::1]:60814', 'req_id': 5, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.250523+0000 | opal_client.opa.logger | INFO | Sent response. {'client_addr': '[::1]:60812', 'req_id': 4, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'resp_bytes': 2, 'resp_duration': 6.3852, 'resp_status': 200, 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.251354+0000 | opal_client.policy_store.opa_client | INFO | processing store transaction: {'id': '910d43e7e4da262d838c562ce3ae9910600fe445', 'actions': ['set_policies'], 'success': True, 'error': ''}
2021-10-21T20:30:08.251502+0000 | opal_client.policy_store.opa_client | INFO | persisting health check policy: ready=true, healthy=true
2021-10-21T20:30:08.256960+0000 | opal_client.opa.logger | INFO | Received request. {'client_addr': '[::1]:60816', 'req_id': 6, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.268133+0000 | opal_client.opa.logger | INFO | Sent response. {'client_addr': '[::1]:60814', 'req_id': 5, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'resp_bytes': 2, 'resp_duration': 23.318, 'resp_status': 200, 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.273519+0000 | opal_client.opa.logger | INFO | Sent response. {'client_addr': '[::1]:60816', 'req_id': 6, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'resp_bytes': 2, 'resp_duration': 14.152, 'resp_status': 200, 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.276124+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO | Connected to PubSub server ws://opal-server-service.hm-opa:7002/wsNow I added
OPAL_SHOULD_REPORT_ON_DATA_UPDATES: "True"
OPAL_DEFAULT_UPDATE_CALLBACKS: '{"callbacks":["http://opal-server-service.hm-opa:7002/data/callback_report"]}'However, my opal-client failed to report the update to opal-server.
[2021-10-21 20:25:33 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2021-10-21 20:25:33 +0000] [1] [INFO] Listening at: http://0.0.0.0:7000 (1)
[2021-10-21 20:25:33 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-10-21 20:25:33 +0000] [9] [INFO] Booting worker with pid: 9
2021-10-21T20:25:34.312137+0000 | opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'FastApiRpcFetchProvider' found at: <class 'opal_common.fetcher.providers.fastapi_rpc_fetch_provider.FastApiRpcFetchProvider'>
2021-10-21T20:25:34.312482+0000 | opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'HttpFetchProvider' found at: <class 'opal_common.fetcher.providers.http_fetch_provider.HttpFetchProvider'>
2021-10-21T20:25:34.346065+0000 | opal_common.fetcher.fetcher_register | INFO | Loading FetcherProvider 'PostgresFetchProvider' found at: <class 'opal_fetcher_postgres.provider.PostgresFetchProvider'>
2021-10-21T20:25:34.346267+0000 | opal_common.fetcher.fetcher_register | INFO | Fetcher Register loaded
2021-10-21T20:25:34.347387+0000 | opal_client.callbacks.register | INFO | Callbacks register loaded
2021-10-21T20:25:34.347677+0000 | opal_client.client | INFO | API authentication disabled (public encryption key was not provided)
2021-10-21T20:25:34.416326+0000 | uvicorn.server | INFO | Started server process [9]
2021-10-21T20:25:34.416639+0000 | uvicorn.lifespan.on | INFO | Waiting for application startup.
2021-10-21T20:25:34.417208+0000 | opal_client.opa.runner | INFO | Launching opa runner
2021-10-21T20:25:34.417500+0000 | uvicorn.lifespan.on | INFO | Application startup complete.
2021-10-21T20:25:34.419152+0000 | opal_client.opa.runner | INFO | Running OPA inline: opa run --server --addr=:8181 --authentication=off --authorization=off --config-file=/data/opa/config.yaml --log-level=info
2021-10-21T20:25:34.472966+0000 | opal_client.opa.logger | INFO | Initializing server. {'addrs': [':8181'], 'diagnostic-addrs': [], 'time': '2021-10-21T20:25:34Z'}
2021-10-21T20:25:34.474321+0000 | opal_client.opa.logger | INFO | Starting decision logger. {'plugin': 'decision_logs', 'time': '2021-10-21T20:25:34Z'}
2021-10-21T20:25:35.425433+0000 | opal_client.opa.runner | INFO | Running OPA initial start callbacks
2021-10-21T20:25:35.426772+0000 | opal_client.policy_store.opa_client | INFO | persisting health check policy: ready=false, healthy=false
2021-10-21T20:25:35.432836+0000 | opal_client.opa.logger | INFO | Received request. {'client_addr': '[::1]:57464', 'req_id': 1, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.435393+0000 | opal_client.opa.logger | INFO | Sent response. {'client_addr': '[::1]:57464', 'req_id': 1, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'resp_bytes': 2, 'resp_duration': 2.7916, 'resp_status': 200, 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.437263+0000 | opal_client.data.updater | INFO | Launching data updater
2021-10-21T20:25:35.437548+0000 | opal_client.policy.updater | INFO | Launching policy updater
2021-10-21T20:25:35.437701+0000 | opal_client.data.updater | INFO | Subscribing to topics: ['policy_data']
2021-10-21T20:25:35.438353+0000 | opal_client.policy.updater | INFO | Subscribing to topics: ['policy:.']
2021-10-21T20:25:35.438681+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO | Trying to connect to Pub/Sub server - ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:25:35.439513+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:25:35.443346+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO | Trying to connect to Pub/Sub server - ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:25:35.445246+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:25:35.513493+0000 | opal_client.data.updater | INFO | Connected to server
2021-10-21T20:25:35.513641+0000 | opal_client.data.updater | INFO | Performing data configuration, reason: Initial load
2021-10-21T20:25:35.513738+0000 | opal_client.data.updater | INFO | Getting data-sources configuration from 'http://opal-server-service.hm-opa:7002/data/config'
2021-10-21T20:25:35.517492+0000 | opal_client.policy.updater | INFO | Connected to server
2021-10-21T20:25:35.517724+0000 | opal_client.policy.updater | INFO | Refetching policy code (full bundle)
2021-10-21T20:25:35.521039+0000 | opal_client.data.updater | INFO | Triggering data update with id: b950fa27e74743c39c8dd01d7cc53999
2021-10-21T20:25:35.521225+0000 | opal_client.data.updater | INFO | Fetching policy data
2021-10-21T20:25:35.521581+0000 | opal_client.data.fetcher | INFO | Fetching data from url: postgresql://[email protected]:40072/opa_db
2021-10-21T20:25:35.538200+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO | Connected to PubSub server ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:25:35.548979+0000 | opal_client.policy.updater | INFO | Got policy bundle with 1 rego files, 0 data files, commit hash: '910d43e7e4da262d838c562ce3ae9910600fe445'
2021-10-21T20:25:35.555292+0000 | opal_client.opa.logger | INFO | Received request. {'client_addr': '[::1]:57478', 'req_id': 2, 'req_method': 'GET', 'req_path': '/v1/policies', 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.555595+0000 | opal_client.opa.logger | INFO | Sent response. {'client_addr': '[::1]:57478', 'req_id': 2, 'req_method': 'GET', 'req_path': '/v1/policies', 'resp_bytes': 4603, 'resp_duration': 0.6619, 'resp_status': 200, 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.560291+0000 | opal_client.opa.logger | INFO | Received request. {'client_addr': '[::1]:57480', 'req_id': 3, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.563343+0000 | opal_client.opa.logger | INFO | Sent response. {'client_addr': '[::1]:57480', 'req_id': 3, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'resp_bytes': 2, 'resp_duration': 4.317, 'resp_status': 200, 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.566660+0000 | opal_client.policy_store.opa_client | INFO | processing store transaction: {'id': '910d43e7e4da262d838c562ce3ae9910600fe445', 'actions': ['set_policies'], 'success': True, 'error': ''}
2021-10-21T20:25:35.567034+0000 | opal_client.policy_store.opa_client | INFO | persisting health check policy: ready=false, healthy=false
2021-10-21T20:25:35.569385+0000 | opal_client.callbacks.reporter | INFO | Reporting the update to requested callbacks
2021-10-21T20:25:35.570242+0000 | opal_client.data.fetcher | INFO | Fetching data from url: http://opal-server-service.hm-opa:7002/data/callback_report
2021-10-21T20:25:35.575745+0000 | opal_client.data.updater | INFO | Saving fetched data to policy-store: source url='postgresql://[email protected]:40072/opa_db', destination path='/roles'
2021-10-21T20:25:35.578385+0000 | opal_client.opa.logger | INFO | Received request. {'client_addr': '[::1]:57482', 'req_id': 4, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.580662+0000 | opal_client.opa.logger | INFO | Sent response. {'client_addr': '[::1]:57482', 'req_id': 4, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'resp_bytes': 2, 'resp_duration': 3.8314, 'resp_status': 200, 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.584730+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO | Connected to PubSub server ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:25:35.585993+0000 | opal_client.opa.logger | INFO | Received request. {'client_addr': '[::1]:57486', 'req_id': 5, 'req_method': 'PUT', 'req_path': '/v1/data/roles', 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.586574+0000 | opal_client.opa.logger | INFO | Sent response. {'client_addr': '[::1]:57486', 'req_id': 5, 'req_method': 'PUT', 'req_path': '/v1/data/roles', 'resp_bytes': 0, 'resp_duration': 1.4706, 'resp_status': 204, 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.588882+0000 | opal_client.policy_store.opa_client | INFO | processing store transaction: {'id': 'b950fa27e74743c39c8dd01d7cc53999', 'actions': ['set_policy_data'], 'success': True, 'error': ''}
2021-10-21T20:25:35.589174+0000 | opal_client.policy_store.opa_client | INFO | persisting health check policy: ready=true, healthy=true
2021-10-21T20:25:35.592062+0000 | opal_client.callbacks.reporter |ERROR | Failed to send report to http://opal-server-service.hm-opa:7002/data/callback_report, got response code 401 with error: {'detail': {'error': 'access token was not provided', 'token': None}}
2021-10-21T20:25:35.592760+0000 | opal_client.opa.logger | INFO | Received request. {'client_addr': '[::1]:57488', 'req_id': 6, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.595426+0000 | opal_client.opa.logger | INFO | Sent response. {'client_addr': '[::1]:57488', 'req_id': 6, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'resp_bytes': 2, 'resp_duration': 2.4091, 'resp_status': 200, 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.597263+0000 | opal_client.callbacks.reporter | INFO | Reporting the update to requested callbacks
2021-10-21T20:25:35.597523+0000 | opal_client.data.fetcher | INFO | Fetching data from url: http://opal-server-service.hm-opa:7002/data/callback_report
2021-10-21T20:25:35.607322+0000 | opal_client.callbacks.reporter |ERROR | Failed to send report to http://opal-server-service.hm-opa:7002/data/callback_report, got response code 401 with error: {'detail': {'error': 'access token was not provided', 'token': None}}Both my opal-client and opal-server versions are 0.1.16.
Possible memory leak with OPAL client
I'm running an OPAL client container in Kubernetes. I recently looked at metrics on the memory usage of this container and noticed that memory is climbing and climbing until it hits 100%. This indicates a possible memory leak with OPAL client.
Here is some more info:
I'm running the image authorizon/opal-client-standalone:0.1.12.
Here's my env var config:
- name: OPAL_POLICY_STORE_URL
value: http://localhost:8181/v1
- name: OPAL_SERVER_URL
value: http://localhost:7002
- name: OPAL_DATA_UPDATER_ENABLED
value: "0"
- name: OPAL_KEEP_ALIVE_INTERVAL
value: "10"And finally the command to run the container:
command:
- "sh"
- "-c"
- "/usr/wait-for.sh http://localhost:7002 --timeout=20 -- /start.sh"Fix official docker images to use non-root user
1 validation error for DataSourceConfig entries field required (type=value_error.missing)
Based on the How to configure external data sources, I created my config server.
When I hit my config server by
curl --location --request GET 'http://localhost:26660/config?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpYXQiOjE2MzUzNjY2NjYsImV4cCI6MTY2NjkwMjY2NiwiYXVkIjoiaHR0cHM6Ly9hcGkuYXV0aG9yaXpvbi5jb20vdjEvIiwiaXNzIjoiaHR0cHM6Ly9hdXRob3Jpem9uLmNvbS8iLCJzdWIiOiJlMGU5NmU2YjZmMTE0MmM0YTc0N2RlMmVhNTYzNWZjNSIsInBlZXJfdHlwZSI6ImNsaWVudCIsImNsaWVudF9pZCI6ImhtLW9wYWwtY2xpZW50In0.B_TmttUj8AZl_wX3LJ0v-0DAUE45YmNqEOpJutEOmaVvPdmOlFqkKMiQHm_98uP1dewiSRKV4GskNG2VPtI4evn_obV9To_q3W1K86pw8f701zAlDpLjlWA1JcG4vqQcxHzBuN0xJ2yekQTUtEJpTz33eKnt4FPIUBVXppH6xUdgmRxyfR6Hg_XUVwXE7eTQbkPEmsGu2fx_VPUzLqWz9y3jOB-1q8LDeObIPeR7g5YyJAHiUvkP2HtZ2uh6micDIDexF2tm2on7WOuSwCA2ITaheHeyaHXCN7qZ2H86X9kwjNqihWkJMbzO646pNgdqhkNkKGmD0BdeoDwjSYZ4Q4T0tulWEV7TIzbiVTq1sWAaKf7P8tcY66MhqNjXVXkdJPkQMXbtDqOZ3Rz51rypgGx1l4AM5LIlamMrCJiZjAlpCdatGep7Jknht6mSA2n11wY5UgXnFQt4vzcT0-x0YzfrwwzrpWa2UM-4a3OSvx9SEoc2c5GfctFjGQNDFfEflKjBZgzjUsmQ9TCpkeRxJr1YIGGO-6jh0HWNykxA3GNBayzhWOvYb6f3aZBVJAYk7QS12nASg8-zQ2DmSYzf8Sy4BVDtxx4uoHzoM9UZI61Xsp0Dg9OVoaMT3o1OpsYCV4CB0YXr9gMYhXMQGepxdJbdPujsVHRrfJBvksTTXbc'I got response:
{
"entries": [
{
"url": "postgresql://[email protected]:40072/opa_db",
"config": {
"fetcher": "PostgresFetchProvider",
"query": "select role, allow from roles;",
"connection_params": {
"password": "passw0rd"
},
"dict_key": "role"
},
"topics": [
"policy_data"
],
"dst_path": "roles"
}
]
}However, my opal client log shows entries field is missing:
2021-10-27T21:04:52.369278+0000 | opal_client.opa.logger | INFO | Received request. {'client_addr': '[::1]:60498', 'req_id': 3, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'time': '2021-10-27T21:04:52Z'}
2021-10-27T21:04:52.373770+0000 | opal_client.opa.logger | INFO | Sent response. {'client_addr': '[::1]:60498', 'req_id': 3, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'resp_bytes': 2, 'resp_duration': 4.1672, 'resp_status': 200, 'time': '2021-10-27T21:04:52Z'}
2021-10-27T21:04:52.374628+0000 | opal_client.policy_store.opa_client | INFO | processing store transaction: {'id': '910d43e7e4da262d838c562ce3ae9910600fe445', 'actions': ['set_policies'], 'success': True, 'error': ''}
2021-10-27T21:04:52.374789+0000 | opal_client.policy_store.opa_client | INFO | persisting health check policy: ready=false, healthy=false
2021-10-27T21:04:52.376963+0000 | opal_client.opa.logger | INFO | Received request. {'client_addr': '[::1]:60500', 'req_id': 4, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'time': '2021-10-27T21:04:52Z'}
2021-10-27T21:04:52.380915+0000 | opal_client.opa.logger | INFO | Sent response. {'client_addr': '[::1]:60500', 'req_id': 4, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'resp_bytes': 2, 'resp_duration': 3.8851, 'resp_status': 200, 'time': '2021-10-27T21:04:52Z'}
2021-10-27T21:04:52.382173+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO | Connected to PubSub server ws://opal-server-service.hm-opa:7002/ws
2021-10-27T21:04:52.775804+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal-server-service.hm-opa:7002/ws
2021-10-27T21:04:52.793599+0000 | opal_client.data.updater | INFO | Connected to server
2021-10-27T21:04:52.793742+0000 | opal_client.data.updater | INFO | Performing data configuration, reason: Initial load
2021-10-27T21:04:52.793820+0000 | opal_client.data.updater | INFO | Getting data-sources configuration from 'http://opal-server-service.hm-opa:7002/data/config'
2021-10-27T21:04:52.802597+0000 | opal_client.data.updater |ERROR | Failed to load data sources config
Traceback (most recent call last):
File "/root/.local/bin/gunicorn", line 8, in <module>
sys.exit(run())
│ │ └ <function run at 0x7f8dc88eab80>
│ └ <built-in function exit>
└ <module 'sys' (built-in)>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
└ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
super().run()
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
│ └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x7f8dc89df6d0>
└ <class 'gunicorn.arbiter.Arbiter'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 202, in run
self.manage_workers()
│ └ <function Arbiter.manage_workers at 0x7f8dc826a310>
└ <gunicorn.arbiter.Arbiter object at 0x7f8dc7de25b0>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
self.spawn_workers()
│ └ <function Arbiter.spawn_workers at 0x7f8dc826a430>
└ <gunicorn.arbiter.Arbiter object at 0x7f8dc7de25b0>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
self.spawn_worker()
│ └ <function Arbiter.spawn_worker at 0x7f8dc826a3a0>
└ <gunicorn.arbiter.Arbiter object at 0x7f8dc7de25b0>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
│ └ <function UvicornWorker.init_process at 0x7f8dc76cfee0>
└ <uvicorn.workers.UvicornWorker object at 0x7f8dc748bc40>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 66, in init_process
super(UvicornWorker, self).init_process()
│ └ <uvicorn.workers.UvicornWorker object at 0x7f8dc748bc40>
└ <class 'uvicorn.workers.UvicornWorker'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 142, in init_process
self.run()
│ └ <function UvicornWorker.run at 0x7f8dc76b40d0>
└ <uvicorn.workers.UvicornWorker object at 0x7f8dc748bc40>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 84, in run
return asyncio.run(self._serve())
│ │ │ └ <function UvicornWorker._serve at 0x7f8dc76b4040>
│ │ └ <uvicorn.workers.UvicornWorker object at 0x7f8dc748bc40>
│ └ <function run at 0x7f8dc7cd48b0>
└ <module 'asyncio' from '/usr/local/lib/python3.8/asyncio/__init__.py'>
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
│ │ └ <coroutine object UvicornWorker._serve at 0x7f8dc6494dc0>
│ └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
└ <uvloop.Loop running=True closed=False debug=False>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.16-py3.8.egg/opal_client/data/updater.py", line 168, in on_connect
await self.get_base_policy_data()
│ └ <function DataUpdater.get_base_policy_data at 0x7f8dc688ec10>
└ <opal_client.data.updater.DataUpdater object at 0x7f8dc6484c40>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.16-py3.8.egg/opal_client/data/updater.py", line 152, in get_base_policy_data
sources_config = await self.get_policy_data_config(url=config_url)
│ │ └ None
│ └ <function DataUpdater.get_policy_data_config at 0x7f8dc688eb80>
└ <opal_client.data.updater.DataUpdater object at 0x7f8dc6484c40>
> File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.16-py3.8.egg/opal_client/data/updater.py", line 135, in get_policy_data_config
return DataSourceConfig.parse_obj(await response.json())
│ │ │ └ <function ClientResponse.json at 0x7f8dc6e261f0>
│ │ └ <ClientResponse(http://config-server-service.hm:26660/?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpYXQiOjE2MzUzNjY2NjYsImV...
│ └ <classmethod object at 0x7f8dc7c165b0>
└ <class 'opal_common.schemas.data.DataSourceConfig'>
File "/root/.local/lib/python3.8/site-packages/pydantic/main.py", line 578, in parse_obj
return cls(**obj)
│ └ {'status': 'ok'}
└ <class 'opal_common.schemas.data.DataSourceConfig'>
File "/root/.local/lib/python3.8/site-packages/pydantic/main.py", line 406, in __init__
raise validation_error
└ ValidationError(model='DataSourceConfig', errors=[{'loc': ('entries',), 'msg': 'field required', 'type': 'value_error.missing...
pydantic.error_wrappers.ValidationError: 1 validation error for DataSourceConfig
entries
field required (type=value_error.missing)
2021-10-27T21:04:52.806754+0000 | fastapi_websocket_rpc.websocket_rpc_c...|ERROR | RPC Error
Traceback (most recent call last):
File "/root/.local/bin/gunicorn", line 8, in <module>
sys.exit(run())
│ │ └ <function run at 0x7f8dc88eab80>
│ └ <built-in function exit>
└ <module 'sys' (built-in)>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
└ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
super().run()
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
│ └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x7f8dc89df6d0>
└ <class 'gunicorn.arbiter.Arbiter'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 202, in run
self.manage_workers()
│ └ <function Arbiter.manage_workers at 0x7f8dc826a310>
└ <gunicorn.arbiter.Arbiter object at 0x7f8dc7de25b0>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
self.spawn_workers()
│ └ <function Arbiter.spawn_workers at 0x7f8dc826a430>
└ <gunicorn.arbiter.Arbiter object at 0x7f8dc7de25b0>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
self.spawn_worker()
│ └ <function Arbiter.spawn_worker at 0x7f8dc826a3a0>
└ <gunicorn.arbiter.Arbiter object at 0x7f8dc7de25b0>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
│ └ <function UvicornWorker.init_process at 0x7f8dc76cfee0>
└ <uvicorn.workers.UvicornWorker object at 0x7f8dc748bc40>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 66, in init_process
super(UvicornWorker, self).init_process()
│ └ <uvicorn.workers.UvicornWorker object at 0x7f8dc748bc40>
└ <class 'uvicorn.workers.UvicornWorker'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 142, in init_process
self.run()
│ └ <function UvicornWorker.run at 0x7f8dc76b40d0>
└ <uvicorn.workers.UvicornWorker object at 0x7f8dc748bc40>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 84, in run
return asyncio.run(self._serve())
│ │ │ └ <function UvicornWorker._serve at 0x7f8dc76b4040>
│ │ └ <uvicorn.workers.UvicornWorker object at 0x7f8dc748bc40>
│ └ <function run at 0x7f8dc7cd48b0>
└ <module 'asyncio' from '/usr/local/lib/python3.8/asyncio/__init__.py'>
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
│ │ └ <coroutine object UvicornWorker._serve at 0x7f8dc6494dc0>
│ └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
└ <uvloop.Loop running=True closed=False debug=False>
> File "/root/.local/lib/python3.8/site-packages/fastapi_websocket_rpc/websocket_rpc_client.py", line 133, in __connect__
await self.channel.on_connect()
│ │ └ <function RpcChannel.on_connect at 0x7f8dc686a160>
│ └ <fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7f8dc6312820>
└ <fastapi_websocket_rpc.websocket_rpc_client.WebSocketRpcClient object at 0x7f8dc64550a0>
File "/root/.local/lib/python3.8/site-packages/fastapi_websocket_rpc/rpc_channel.py", line 237, in on_connect
await self.on_handler_event(self._connect_handlers, self)
│ │ │ │ └ <fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7f8dc6312820>
│ │ │ └ [<bound method PubSubClient._primary_on_connect of <fastapi_websocket_pubsub.pub_sub_client.PubSubClient object at 0x7f8dc644...
│ │ └ <fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7f8dc6312820>
│ └ <function RpcChannel.on_handler_event at 0x7f8dc686a0d0>
└ <fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7f8dc6312820>
File "/root/.local/lib/python3.8/site-packages/fastapi_websocket_rpc/rpc_channel.py", line 234, in on_handler_event
await asyncio.gather(*(callback(*args, **kwargs) for callback in handlers))
│ │ │ │ └ [<bound method PubSubClient._primary_on_connect of <fastapi_websocket_pubsub.pub_sub_client.PubSubClient object at 0x7f8dc644...
│ │ │ └ {}
│ │ └ (<fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7f8dc6312820>,)
│ └ <function gather at 0x7f8dc7a81700>
└ <module 'asyncio' from '/usr/local/lib/python3.8/asyncio/__init__.py'>
File "/root/.local/lib/python3.8/site-packages/fastapi_websocket_pubsub/pub_sub_client.py", line 217, in _primary_on_connect
await asyncio.gather(*(callback(self, channel) for callback in self._on_connect))
│ │ │ │ │ └ [<bound method DataUpdater.on_connect of <opal_client.data.updater.DataUpdater object at 0x7f8dc6484c40>>]
│ │ │ │ └ <fastapi_websocket_pubsub.pub_sub_client.PubSubClient object at 0x7f8dc644ae50>
│ │ │ └ <fastapi_websocket_rpc.rpc_channel.RpcChannel object at 0x7f8dc6312820>
│ │ └ <fastapi_websocket_pubsub.pub_sub_client.PubSubClient object at 0x7f8dc644ae50>
│ └ <function gather at 0x7f8dc7a81700>
└ <module 'asyncio' from '/usr/local/lib/python3.8/asyncio/__init__.py'>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.16-py3.8.egg/opal_client/data/updater.py", line 168, in on_connect
await self.get_base_policy_data()
│ └ <function DataUpdater.get_base_policy_data at 0x7f8dc688ec10>
└ <opal_client.data.updater.DataUpdater object at 0x7f8dc6484c40>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.16-py3.8.egg/opal_client/data/updater.py", line 152, in get_base_policy_data
sources_config = await self.get_policy_data_config(url=config_url)
│ │ └ None
│ └ <function DataUpdater.get_policy_data_config at 0x7f8dc688eb80>
└ <opal_client.data.updater.DataUpdater object at 0x7f8dc6484c40>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.16-py3.8.egg/opal_client/data/updater.py", line 135, in get_policy_data_config
return DataSourceConfig.parse_obj(await response.json())
│ │ │ └ <function ClientResponse.json at 0x7f8dc6e261f0>
│ │ └ <ClientResponse(http://config-server-service.hm:26660/?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpYXQiOjE2MzUzNjY2NjYsImV...
│ └ <classmethod object at 0x7f8dc7c165b0>
└ <class 'opal_common.schemas.data.DataSourceConfig'>
File "/root/.local/lib/python3.8/site-packages/pydantic/main.py", line 578, in parse_obj
return cls(**obj)
│ └ {'status': 'ok'}
└ <class 'opal_common.schemas.data.DataSourceConfig'>
File "/root/.local/lib/python3.8/site-packages/pydantic/main.py", line 406, in __init__
raise validation_error
└ ValidationError(model='DataSourceConfig', errors=[{'loc': ('entries',), 'msg': 'field required', 'type': 'value_error.missing...
pydantic.error_wrappers.ValidationError: 1 validation error for DataSourceConfig
entries
field required (type=value_error.missing)
2021-10-27T21:04:53.440492+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal-server-service.hm-opa:7002/ws
2021-10-27T21:04:53.453091+0000 | opal_client.data.updater | INFO | Connected to server
2021-10-27T21:04:53.453254+0000 | opal_client.data.updater | INFO | Performing data configuration, reason: Initial load
2021-10-27T21:04:53.453400+0000 | opal_client.data.updater | INFO | Getting data-sources configuration from 'http://opal-server-service.hm-opa:7002/data/config'
2021-10-27T21:04:53.465428+0000 | opal_client.data.updater |ERROR | Failed to load data sources config
Traceback (most recent call last):
File "/root/.local/bin/gunicorn", line 8, in <module>
sys.exit(run())
│ │ └ <function run at 0x7f8dc88eab80>
│ └ <built-in function exit>
└ <module 'sys' (built-in)>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
└ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
super().run()
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
│ └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x7f8dc89df6d0>
└ <class 'gunicorn.arbiter.Arbiter'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 202, in run
self.manage_workers()
│ └ <function Arbiter.manage_workers at 0x7f8dc826a310>
└ <gunicorn.arbiter.Arbiter object at 0x7f8dc7de25b0>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
self.spawn_workers()
│ └ <function Arbiter.spawn_workers at 0x7f8dc826a430>
└ <gunicorn.arbiter.Arbiter object at 0x7f8dc7de25b0>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
self.spawn_worker()
│ └ <function Arbiter.spawn_worker at 0x7f8dc826a3a0>
└ <gunicorn.arbiter.Arbiter object at 0x7f8dc7de25b0>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
│ └ <function UvicornWorker.init_process at 0x7f8dc76cfee0>
└ <uvicorn.workers.UvicornWorker object at 0x7f8dc748bc40>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 66, in init_process
super(UvicornWorker, self).init_process()
│ └ <uvicorn.workers.UvicornWorker object at 0x7f8dc748bc40>
└ <class 'uvicorn.workers.UvicornWorker'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 142, in init_process
self.run()
│ └ <function UvicornWorker.run at 0x7f8dc76b40d0>
└ <uvicorn.workers.UvicornWorker object at 0x7f8dc748bc40>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 84, in run
return asyncio.run(self._serve())
│ │ │ └ <function UvicornWorker._serve at 0x7f8dc76b4040>
│ │ └ <uvicorn.workers.UvicornWorker object at 0x7f8dc748bc40>
│ └ <function run at 0x7f8dc7cd48b0>
└ <module 'asyncio' from '/usr/local/lib/python3.8/asyncio/__init__.py'>
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
│ │ └ <coroutine object UvicornWorker._serve at 0x7f8dc6494dc0>
│ └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
└ <uvloop.Loop running=True closed=False debug=False>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.16-py3.8.egg/opal_client/data/updater.py", line 168, in on_connect
await self.get_base_policy_data()
│ └ <function DataUpdater.get_base_policy_data at 0x7f8dc688ec10>
└ <opal_client.data.updater.DataUpdater object at 0x7f8dc6484c40>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.16-py3.8.egg/opal_client/data/updater.py", line 152, in get_base_policy_data
sources_config = await self.get_policy_data_config(url=config_url)
│ │ └ None
│ └ <function DataUpdater.get_policy_data_config at 0x7f8dc688eb80>
└ <opal_client.data.updater.DataUpdater object at 0x7f8dc6484c40>
> File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.16-py3.8.egg/opal_client/data/updater.py", line 135, in get_policy_data_config
return DataSourceConfig.parse_obj(await response.json())
│ │ │ └ <function ClientResponse.json at 0x7f8dc6e261f0>
│ │ └ <ClientResponse(http://config-server-service.hm:26660/?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpYXQiOjE2MzUzNjY2NjYsImV...
│ └ <classmethod object at 0x7f8dc7c165b0>
└ <class 'opal_common.schemas.data.DataSourceConfig'>
File "/root/.local/lib/python3.8/site-packages/pydantic/main.py", line 578, in parse_obj
return cls(**obj)
│ └ {'status': 'ok'}
└ <class 'opal_common.schemas.data.DataSourceConfig'>
File "/root/.local/lib/python3.8/site-packages/pydantic/main.py", line 406, in __init__
raise validation_error
└ ValidationError(model='DataSourceConfig', errors=[{'loc': ('entries',), 'msg': 'field required', 'type': 'value_error.missing...
pydantic.error_wrappers.ValidationError: 1 validation error for DataSourceConfig
entries
field required (type=value_error.missing)Any idea? Thanks!
support more git providers (for webhooks)
e.g: gitlab, bitbucket, etc
work started in #64, will complete it in a separate pull :)
fix test_repo_cloner_clone_remote_repo_ssh_url
Background
The test is checking that opal common can clone a private repo.
we sometimes need to clone a private repo when the policy code synced to opal is stored on a private git repo.
the only way to authenticate to a private github repo is with SSH, and not with HTTPS.
cloning with HTTPS is done with a clone url like this: https://github.com/authorizon/fastapi_websocket_pubsub.git
cloning with SSH is done with a clone url like this: [email protected]:authorizon/fastapi_websocket_pubsub.git
why the test is failing
when running in CI (github actions) if the repo is private or (currently) when running locally on a machine of a developer who does not have the private ssh key to this repo: https://github.com/authorizon/fastapi_websocket_pubsub
The test is failing on line: https://github.com/authorizon/opal/blob/380eed6812a5452935bb93f37f733690280550b2/opal_common/git/tests/repo_cloner_test.py#L111
in CI we do this, and although the clone is failing, the test expects to fail:
with pytest.raises(GitFailed):
# result =
RepoCloner(
repo_url=VALID_REPO_REMOTE_URL_SSH,
clone_path=target_path
).clone()
what we should do instead:
- open disposable private github repo
- create an ssh key pair with this command:
ssh-keygen -t rsa -b 4096 -m pem -C "<your email>"
creates to files (one with .pub ext which is the public key and one without ext which is the private key)
-
add the public key you creates as a deploy key on github (on the disposable repo settings)
-
the test needs to get the ssh key
RepoCloner(
repo_url=VALID_REPO_REMOTE_URL_SSH,
clone_path=target_path,
ssh_key=PRIVATE_SSH_KEY_FOR_TEST. <- can get from env var (default empty string)
).clone()
can test that works with
PRIVATE_SSH_KEY_FOR_TEST= pytest -s path/to/test.py
key contents should be the contents of the private ssh key file, but with newlines encoded as _
in ipython
x = """ <contents> """
x.replace('\n', '_')
Policy fetcher should know how to deal with 503 (and maybe retry)
What's need to be fixed?
-
We need to add another if statement here when the
/policybundle api (docs here) returns 503. -
We need to fix the server openapi schema of the route (example how it's done is here) to show the 503 in possible responses
When does the policy bundle api return 503
if the OPAL server did not finish to clone the repo yet, the endpoint can return 503.
This race condition that rarely happens if at all, since you generally are supposed to deploy OPAL server before the client anyways.
Help wanted :)
This can be a great first issue for someone to get into the codebase, we are willing to work with you and guide you :)
Change external_source_url to send token in body as a POST request
Currently when opal client gets redirected to an external_source_url, the client token is sent as a URL parameter.
Although this is encrypted with SSL encryption, there may still be a lot of infrastructure between the client and the actual API server that returns the configs, which can log or otherwise reveal the token.
Any sensitive info like this should either be in the headers or body.
Since OPAL server just performs a 307 redirect, and authorization header is not maintained on redirect, I would suggest to send the token in the API body as form-data or form-urlencoded, which is good for such sensitive info. Although the API method would need to be changed from GET to POST.
Fix flaky data updater test
For some reason the data updater test is flaky (sometime does not pass due to timeout on this line).
We will get to it soon, but if someone wants to pick this up we are more than happy to guide you and work with you :)
Marketing site is a bit wonky
Looking at the marketing site and I can't scroll or see stuff on the page.
opal-server stops pulling new updates if the policy repo master branch has a force push
opal-server stops pulling new updates if the policy repo master branch has a force push (replace the last one commit with a new one).
Here is the log, it just stops further pulling without any error. And current pulling will be pending.
Logs
[2021-07-12 19:21:33 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2021-07-12 19:21:33 +0000] [1] [INFO] Listening at: http://0.0.0.0:7002 (1)
[2021-07-12 19:21:33 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-07-12 19:21:33 +0000] [10] [INFO] Booting worker with pid: 10
[2021-07-12 19:21:33 +0000] [11] [INFO] Booting worker with pid: 11
[2021-07-12 19:21:33 +0000] [12] [INFO] Booting worker with pid: 12
[2021-07-12 19:21:33 +0000] [13] [INFO] Booting worker with pid: 13
2021-07-12T19:21:34.654948+0000 |opal_common.authentication.signer | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!
2021-07-12T19:21:34.655596+0000 |opal_common.authentication.signer | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!
2021-07-12T19:21:34.661812+0000 |opal_common.authentication.signer | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!
2021-07-12T19:21:34.668544+0000 |opal_common.authentication.signer | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!
2021-07-12T19:21:34.726044+0000 |opal_server.server | INFO | triggered startup event
2021-07-12T19:21:34.726328+0000 |opal_common.topics.publisher | INFO | started topic publisher
2021-07-12T19:21:34.727299+0000 |opal_server.server | INFO | leadership lock acquired, leader pid: 11
2021-07-12T19:21:34.727466+0000 |opal_server.server | INFO | listening on webhook topic: 'webhook'
2021-07-12T19:21:34.728867+0000 |fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': '5d84fef30f3f4f0a8f712adee48a1bfd', 'subscriber_id': '8e5cdf788008461e9a0d900294df08ce', 'topic': 'webhook', 'callback': functools.partial(<function trigger_repo_watcher_pull at 0x7fd96c9d4e50>, <opal_server.policy.watcher.task.RepoWatcherTask object at 0x7fd96c9e5400>), 'notifier_id': None}
2021-07-12T19:21:34.729054+0000 |opal_server.policy.watcher.task | INFO | Launching repo watcher
2021-07-12T19:21:34.729679+0000 |opal_common.git.repo_cloner | INFO | Cloning repo from 'https://github.com/Hongbo-Miao/opal-example-policy-repo' to '/regoclone'
2021-07-12T19:21:34.746472+0000 |opal_server.server | INFO | triggered startup event
2021-07-12T19:21:34.752982+0000 |opal_common.topics.publisher | INFO | started topic publisher
2021-07-12T19:21:34.764244+0000 |opal_server.server | INFO | triggered startup event
2021-07-12T19:21:34.764488+0000 |opal_common.topics.publisher | INFO | started topic publisher
2021-07-12T19:21:34.765975+0000 |opal_server.server | INFO | triggered startup event
2021-07-12T19:21:34.766253+0000 |opal_common.topics.publisher | INFO | started topic publisher
2021-07-12T19:21:37.484660+0000 |opal_common.git.repo_cloner | INFO | Clone succeeded
2021-07-12T19:21:37.491395+0000 |opal_common.git.repo_watcher | INFO | Launching polling task, interval: 30 seconds
2021-07-12T19:21:59.811384+0000 |fastapi_websocket_pubsub.event_broadc...| INFO | Listening for incoming events from broadcast channel (first listener started)
2021-07-12T19:21:59.812194+0000 |fastapi_websocket_pubsub.event_broadc...| INFO | Spawning broadcast listen task
2021-07-12T19:21:59.877097+0000 |fastapi_websocket_pubsub.event_broadc...| INFO | Subscribing to ALL TOPICS, and sharing messages with broadcast channel
2021-07-12T19:21:59.878474+0000 |fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': '450b2e2b53934279991697fee7738013', 'subscriber_id': '16eb38578f7446728d09c53f9adbf579', 'topic': '__EventNotifier_ALL_TOPICS__', 'callback': <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_websocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7fd96ca2ab20>>, 'notifier_id': None}
2021-07-12T19:21:59.880866+0000 |fastapi_websocket_rpc.websocket_rpc_e...| INFO | Client connected
2021-07-12T19:21:59.881216+0000 |fastapi_websocket_pubsub.event_broadc...| INFO | Starting broadcaster listener
2021-07-12T19:21:59.949840+0000 |fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': 'e08d36109cda402c8860d1cdf755e057', 'subscriber_id': '25b6419659f54c1696d9781a8cbaa0de', 'topic': 'policy_data', 'callback': <function RpcEventServerMethods.subscribe.<locals>.callback at 0x7fd96c47f310>, 'notifier_id': None}
2021-07-12T19:22:07.458130+0000 |opal_common.git.repo_watcher | INFO | Pulling changes from remote: 'origin'
2021-07-12T19:22:09.299170+0000 |opal_common.git.repo_watcher | INFO | No new commits: HEAD is at 'b7480e393d5a21086a07d8da6c99ff2516f270f4'
2021-07-12T19:22:39.266851+0000 |opal_common.git.repo_watcher | INFO | Pulling changes from remote: 'origin'
2021-07-12T19:22:41.104471+0000 |opal_common.git.repo_watcher | INFO | No new commits: HEAD is at 'b7480e393d5a21086a07d8da6c99ff2516f270f4'
2021-07-12T19:23:11.071210+0000 |opal_common.git.repo_watcher | INFO | Pulling changes from remote: 'origin'
2021-07-12T19:23:12.905227+0000 |opal_common.git.repo_watcher | INFO | No new commits: HEAD is at 'b7480e393d5a21086a07d8da6c99ff2516f270f4'
2021-07-12T19:23:42.872009+0000 |opal_common.git.repo_watcher | INFO | Pulling changes from remote: 'origin'
2021-07-12T19:23:44.737825+0000 |opal_common.git.repo_watcher | INFO | No new commits: HEAD is at 'b7480e393d5a21086a07d8da6c99ff2516f270f4'
2021-07-12T19:24:14.702912+0000 |opal_common.git.repo_watcher | INFO | Pulling changes from remote: 'origin'
2021-07-12T19:24:16.494131+0000 |opal_common.git.repo_watcher | INFO | No new commits: HEAD is at 'b7480e393d5a21086a07d8da6c99ff2516f270f4'
2021-07-12T19:24:46.461508+0000 |opal_common.git.repo_watcher | INFO | Pulling changes from remote: 'origin'
2021-07-12T19:24:48.373691+0000 |opal_common.git.repo_watcher | INFO | No new commits: HEAD is at 'b7480e393d5a21086a07d8da6c99ff2516f270f4'
2021-07-12T19:25:18.342024+0000 |opal_common.git.repo_watcher | INFO | Pulling changes from remote: 'origin'
2021-07-12T19:25:20.176742+0000 |opal_common.git.repo_watcher | INFO | No new commits: HEAD is at 'b7480e393d5a21086a07d8da6c99ff2516f270f4'
2021-07-12T19:25:50.142963+0000 |opal_common.git.repo_watcher | INFO | Pulling changes from remote: 'origin'
2021-07-12T19:25:51.972437+0000 |opal_common.git.repo_watcher | INFO | No new commits: HEAD is at 'b7480e393d5a21086a07d8da6c99ff2516f270f4'
2021-07-12T19:26:21.938838+0000 |opal_common.git.repo_watcher | INFO | Pulling changes from remote: 'origin'
2021-07-12T19:26:23.796473+0000 |opal_common.git.repo_watcher | INFO | No new commits: HEAD is at 'b7480e393d5a21086a07d8da6c99ff2516f270f4'
2021-07-12T19:26:53.764178+0000 |opal_common.git.repo_watcher | INFO | Pulling changes from remote: 'origin'
2021-07-12T19:26:55.618927+0000 |opal_common.git.repo_watcher | INFO | No new commits: HEAD is at 'b7480e393d5a21086a07d8da6c99ff2516f270f4'
2021-07-12T19:27:25.584704+0000 |opal_common.git.repo_watcher | INFO | Pulling changes from remote: 'origin'
2021-07-12T19:27:27.492180+0000 |opal_common.git.repo_watcher | INFO | No new commits: HEAD is at 'b7480e393d5a21086a07d8da6c99ff2516f270f4'
2021-07-12T19:27:57.458852+0000 |opal_common.git.repo_watcher | INFO | Pulling changes from remote: 'origin'This is a very rare case as usually we don't force push master branch so definitely a low priority bug.
However, just leave this here as I was curious what gonna happen if I do a force push. 😅
OPAL does not consider dependencies when loading policy files into OPA
When loading policy files into OPA one at a time, OPAL does not consider the dependencies of those policies in the order that they are loaded, and instead loads the policies in alphabetical order of the file name.
Imagine the following structure:
.
├── authorization.rego
├── utils.rego
package authorization
import data.utils
allow {
utils.hasPermission(input.grants, input.roles)
}
package utils
hasPermission(grants, roles) {
grants[_] == roles[_]
}
authorization has a dependency on utils, so if authorization is loaded before utils is loaded, there would be a type error when compiling.
Because, alphabetically, authorization < utils, OPAL does indeed first send the authorization policy to OPA, which OPA cannot compile:
opal_client_1 | 2021-08-18T20:39:20.169031+0000 |opal_client.policy.updater | INFO | got policy bundle, commit hash: '4fcc7d3aaa646b869c072874540ec61e38210144'
opal_client_1 | 2021-08-18T20:39:20.172057+0000 |opal_client.opa.logger | INFO | Received request. {'client_addr': '127.0.0.1:60262', 'req_id': 8, 'req_method': 'GET', 'req_path': '/v1/policies', 'time': '2021-08-18T20:39:20Z'}
opal_client_1 | 2021-08-18T20:39:20.172976+0000 |opal_client.opa.logger | INFO | Sent response. {'client_addr': '127.0.0.1:60262', 'req_id': 8, 'req_method': 'GET', 'req_path': '/v1/policies', 'resp_bytes': 13, 'resp_duration': 0.2695, 'resp_status': 200, 'time': '2021-08-18T20:39:20Z'}
opal_client_1 | 2021-08-18T20:39:20.177236+0000 |opal_client.opa.logger | INFO | Received request. {'client_addr': '127.0.0.1:60264', 'req_id': 9, 'req_method': 'PUT', 'req_path': '/v1/policies/authorization.rego', 'time': '2021-08-18T20:39:20Z'}
opal_client_1 | 2021-08-18T20:39:20.179623+0000 |opal_client.opa.logger | INFO | Sent response. {'client_addr': '127.0.0.1:60264', 'req_id': 9, 'req_method': 'PUT', 'req_path': '/v1/policies/authorization.rego', 'resp_bytes': 753, 'resp_duration': 2.2022, 'resp_status': 400, 'time': '2021-08-18T20:39:20Z'}
opal_client_1 | 2021-08-18T20:39:22.184929+0000 |opal_client.opa.logger | INFO | Received request. {'client_addr': '127.0.0.1:60266', 'req_id': 10, 'req_method': 'PUT', 'req_path': '/v1/policies/authorization.rego', 'time': '2021-08-18T20:39:22Z'}
opal_client_1 | 2021-08-18T20:39:22.186592+0000 |opal_client.opa.logger | INFO | Sent response. {'client_addr': '127.0.0.1:60266', 'req_id': 10, 'req_method': 'PUT', 'req_path': '/v1/policies/authorization.rego', 'resp_bytes': 753, 'resp_duration': 1.5418, 'resp_status': 400, 'time': '2021-08-18T20:39:22Z'}
opal_client_1 | 2021-08-18T20:39:22.187957+0000 |opal_client...base_policy_store_client |WARNING | OPA transaction failed, transaction id=4fcc7d3aaa646b869c072874540ec61e38210144, actions=['set_policies'], error=RetryError(<Future at 0x7f2f3b7e1fd0 state=finished raised ValueError>)
ValueError: OPA Client: unexpected status code: 400, error: {'code': 'invalid_parameter', 'message': 'error(s) occurred while compiling module(s)', 'errors': [{'code': 'rego_type_error', 'message': 'undefined function data.utils.hasPermission', 'location': {'file': 'authorization.rego', 'row': 12, 'col': 2}}]}
If we rename the rego files so that dependencies are in alphabetical order, there is no issue:
.
├── a_utils.rego
├── b_authorization.rego
opal_client_1 | 2021-08-18T20:41:30.282751+0000 |opal_client.policy.updater | INFO | got policy bundle, commit hash: '1e13e803d5bde6a6627ac0a90dceb188e68a63ae'
opal_client_1 | 2021-08-18T20:41:30.286355+0000 |opal_client.opa.logger | INFO | Received request. {'client_addr': '127.0.0.1:60310', 'req_id': 1, 'req_method': 'PUT', 'req_path': '/v1/data', 'time': '2021-08-18T20:41:30Z'}
opal_client_1 | 2021-08-18T20:41:30.288170+0000 |opal_client.opa.logger | INFO | Sent response. {'client_addr': '127.0.0.1:60310', 'req_id': 1, 'req_method': 'PUT', 'req_path': '/v1/data', 'resp_bytes': 0, 'resp_duration': 0.3158, 'resp_status': 204, 'time': '2021-08-18T20:41:30Z'}
opal_client_1 | 2021-08-18T20:41:30.289615+0000 |opal_client.opa.logger | INFO | Received request. {'client_addr': '127.0.0.1:60312', 'req_id': 2, 'req_method': 'GET', 'req_path': '/v1/policies', 'time': '2021-08-18T20:41:30Z'}
opal_client_1 | 2021-08-18T20:41:30.290652+0000 |opal_client.opa.logger | INFO | Sent response. {'client_addr': '127.0.0.1:60312', 'req_id': 2, 'req_method': 'GET', 'req_path': '/v1/policies', 'resp_bytes': 13, 'resp_duration': 0.5773, 'resp_status': 200, 'time': '2021-08-18T20:41:30Z'}
opal_client_1 | 2021-08-18T20:41:30.295297+0000 |opal_client.opa.logger | INFO | Received request. {'client_addr': '127.0.0.1:60314', 'req_id': 3, 'req_method': 'PUT', 'req_path': '/v1/policies/a_utils.rego', 'time': '2021-08-18T20:41:30Z'}
opal_client_1 | 2021-08-18T20:41:30.300373+0000 |opal_client.opa.logger | INFO | Sent response. {'client_addr': '127.0.0.1:60314', 'req_id': 3, 'req_method': 'PUT', 'req_path': '/v1/policies/a_utils.rego', 'resp_bytes': 2, 'resp_duration': 4.944, 'resp_status': 200, 'time': '2021-08-18T20:41:30Z'}
opal_client_1 | 2021-08-18T20:41:30.304157+0000 |opal_client.opa.logger | INFO | Received request. {'client_addr': '127.0.0.1:60316', 'req_id': 4, 'req_method': 'PUT', 'req_path': '/v1/policies/b_authorization.rego', 'time': '2021-08-18T20:41:30Z'}
opal_client_1 | 2021-08-18T20:41:30.310509+0000 |opal_client.opa.logger | INFO | Sent response. {'client_addr': '127.0.0.1:60316', 'req_id': 4, 'req_method': 'PUT', 'req_path': '/v1/policies/b_authorization.rego', 'resp_bytes': 2, 'resp_duration': 6.1766, 'resp_status': 200, 'time': '2021-08-18T20:41:30Z'}
Healthcheck always returns false with option `OPAL_DATA_UPDATER_ENABLED=False`
I was looking at running the healthcheck policy as it is useful configuring this for a Kubernetes environment but there is also no requirement currently to update data, only policy. It turns out that by disabling data updates using the OPAL_DATA_UPDATER_ENABLED option it causes the healthcheck to never return true.
Steps to reproduce
- Update
docker-compose-with-callbacks.ymlto disable data updates (gdmarsh/opal@44e6db9) - Start docker-compose
- Observe client logs and/or curl
v1/data/system/opalto show that ready and healthy both return false and will never end up being true
opal_client.opa.runner | INFO | Running OPA initial start callbacks
opal_client.policy_store.opa_client | INFO | persisting health check policy: ready=false, healthy=false
opal_client.policy_store.opa_client | INFO | Policy and data statistics: policy: (successful 0, failed 0); data: (successful 0, failed 0)
opal_client.opa.logger | INFO | Received request. PUT /v1/policies/opa/healthcheck/opal.rego
opal_client.opa.logger | INFO | Sent response. PUT /v1/policies/opa/healthcheck/opal.rego -> 200
opal_client.policy.updater | INFO | Launching policy updater
opal_client.policy.updater | INFO | Subscribing to topics: ['policy:.']
fastapi_websocket_pubsub.pub_sub_client | INFO | Trying to connect to Pub/Sub server - ws://opal_server:7002/ws
fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal_server:7002/ws
opal_client.policy.updater | INFO | Connected to server
opal_client.policy.updater | INFO | Refetching policy code (full bundle)
opal_client.policy.updater | INFO | Got policy bundle with 2 rego files, 1 data files, commit hash: '6d849b1ce92bae05c31146470e66e07c3c41b164'
opal_client.opa.logger | INFO | Received request. GET /v1/policies
opal_client.opa.logger | INFO | Sent response. GET /v1/policies -> 200
opal_client.opa.logger | INFO | Received request. PUT /v1/data
opal_client.opa.logger | INFO | Sent response. PUT /v1/data -> 204
opal_client.opa.logger | INFO | Received request. PUT /v1/policies/utils.rego
opal_client.opa.logger | INFO | Sent response. PUT /v1/policies/utils.rego -> 200
opal_client.opa.logger | INFO | Received request. PUT /v1/policies/rbac.rego
opal_client.opa.logger | INFO | Sent response. PUT /v1/policies/rbac.rego -> 200
opal_client.policy_store.opa_client | INFO | processing store transaction: {'id': '6d849b1ce92bae05c31146470e66e07c3c41b164', 'actions': ['set_policies'], 'transaction_type': <TransactionType.policy: 'policy'>, 'success': True, 'error': '', 'creation_time': '2021-11-10T13:44:07.012526', 'end_time': '2021-11-10T13:44:07.031347', 'remotes_status': [{'remote_url': 'http://opal_server:7002/policy', 'succeed': True, 'error': None}]}
opal_client.policy_store.opa_client | INFO | persisting health check policy: ready=false, healthy=false
opal_client.policy_store.opa_client | INFO | Policy and data statistics: policy: (successful 1, failed 0); data: (successful 0, failed 0)
opal_client.opa.logger | INFO | Received request. PUT /v1/policies/opa/healthcheck/opal.rego
opal_client.opa.logger | INFO | Sent response. PUT /v1/policies/opa/healthcheck/opal.rego -> 200
fastapi_websocket_pubsub.pub_sub_client | INFO | Connected to PubSub server ws://opal_server:7002/ws
Restrict topic subscription from OPAL Client and topic events from OPAL Server
Started as a discussion in https://github.com/authorizon/opal/discussions/159
Originally posted by jyoussefzadeh October 19, 2021
From the documentation & previous discussions, I understand that a JWT token can be issued to authenticate the OPAL Clients connection to OPAL Server.
Is it also possible to restrict which OPAL_DATA_TOPICS can be subscribed to from that client?
For instance, if there is a single shared pool of OPAL Servers with hundreds of different OPAL Clients connected to it, we do not want one client to be able to subscribe to & fetch unauthorized information that belongs to another client.
On a similar vein, given some bearer token for calling OPAL Server APIs, is it possible to restrict which topics that token can publish an event to, using the /data/config API?
Also, is it possible to restrict access to different OPAL Server endpoints entirely?
Publish update api should enforce JWT claim type to be datasource
About OPAL Security model
Note: We will soon add the doc about OPAL security in depth, in the meanwhile i am willing to work closely with whoever wants to tackle this
- OPAL security is based on JWTs (Json web tokens).
- Only the OPAL server has a private key and can cryptographically sign on the JWTs.
- OPAL server exposes an API to generate new tokens. This API is protected by a "master token", known only to the person/team deploying opal (and should be stored in a secrets vault, obviously).
- The OPAL JWT has a peer type claim that can either be a client (which means an OPAL client) or a datasource (which is usually your own backend (micro)service that pushes data updates via OPAL server).
The change
the change we need to do is to make sure this API endpoint (see docs here) can only be accessed by a JWT with a datasource claim.
Allow to use new version of tenacity
Currently opal-common pins tenacity to version 6.3.1.
When we build the Docker image of our own fetcher based on opal-fetcher-postgres, it will show this error if we use new tenacity version
8.0.1 is installed but tenacity==6.3.1 is required by {'opal-common'}
It would be great to allow us to use new version in future. Thanks! 😃
When OPAL is not provided a valid passphrase - the private key is set to None instead of throwing
Not sure if it's a bug in OPAL casting method or in the crypto library, but if a private key generated with a passphrase incorrectly or without passing the passphrase env var, the following code will not throw:
if passphrase is None:
password = None
else:
password = passphrase.encode('utf-8')
...
if key_format == EncryptionKeyFormat.pem:
return serialization.load_pem_private_key(raw_key, password=password, backend=default_backend())
if key_format == EncryptionKeyFormat.ssh:
return serialization.load_ssh_private_key(raw_key, password=password, backend=default_backend())
if key_format == EncryptionKeyFormat.der:
return serialization.load_der_private_key(raw_key, password=password, backend=default_backend())
i expect this kind of line to throw if password is invalid or None
return serialization.load_pem_private_key(raw_key, password=password, backend=default_backend())
instead it returns None
as a result - OPAL thinks it got a public key without a private key
Go over all OPAL's config variables and add descriptions
Confi (our configuration lib) allows each config var to have a description string. All strings should be well explained and documented.
Policy from APIs instead of git?
I was wondering if there is a way to synch policies from an API rather than from git.
OPAL cannot clone non-base branch on private repo
reported by Jake on slack: https://opal-access.slack.com/archives/C01RUUYV3TP/p1634067725023100
config:
- OPAL_POLICY_REPO_MAIN_BRANCH=dev
- OPAL_POLICY_REPO_MAIN_REMOTE=origin
- OPAL_POLICY_REPO_SSH_KEY=${OPAL_POLICY_REPO_SSH_KEY}
stack trace:
opal_server_1 | 2021-10-12T19:40:53.555089+0000 |opal_common.git.branch_tracker |ERROR | did not find main branch: 'IterableList' object has no attribute 'dev', instead found: [{'name': 'main', 'path': 'refs/heads/main'}]
opal_server_1 | Traceback (most recent call last):
opal_server_1 |
opal_server_1 | File "/root/.local/bin/gunicorn", line 8, in <module>
opal_server_1 | sys.exit(run())
opal_server_1 | │ │ └ <function run at 0x7f275202b3a0>
opal_server_1 | │ └ <built-in function exit>
opal_server_1 | └ <module 'sys' (built-in)>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
opal_server_1 | WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
opal_server_1 | └ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
opal_server_1 | super().run()
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
opal_server_1 | Arbiter(self).run()
opal_server_1 | │ └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x7f27520368b0>
opal_server_1 | └ <class 'gunicorn.arbiter.Arbiter'>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 202, in run
opal_server_1 | self.manage_workers()
opal_server_1 | │ └ <function Arbiter.manage_workers at 0x7f2751978ee0>
opal_server_1 | └ <gunicorn.arbiter.Arbiter object at 0x7f2752036190>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
opal_server_1 | self.spawn_workers()
opal_server_1 | │ └ <function Arbiter.spawn_workers at 0x7f2751979040>
opal_server_1 | └ <gunicorn.arbiter.Arbiter object at 0x7f2752036190>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
opal_server_1 | self.spawn_worker()
opal_server_1 | │ └ <function Arbiter.spawn_worker at 0x7f2751978f70>
opal_server_1 | └ <gunicorn.arbiter.Arbiter object at 0x7f2752036190>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
opal_server_1 | worker.init_process()
opal_server_1 | │ └ <function UvicornWorker.init_process at 0x7f2750f07940>
opal_server_1 | └ <uvicorn.workers.UvicornWorker object at 0x7f27511b4a30>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 64, in init_process
opal_server_1 | super(UvicornWorker, self).init_process()
opal_server_1 | │ └ <uvicorn.workers.UvicornWorker object at 0x7f27511b4a30>
opal_server_1 | └ <class 'uvicorn.workers.UvicornWorker'>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 142, in init_process
opal_server_1 | self.run()
opal_server_1 | │ └ <function UvicornWorker.run at 0x7f2750f07a60>
opal_server_1 | └ <uvicorn.workers.UvicornWorker object at 0x7f27511b4a30>
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 77, in run
opal_server_1 | loop.run_until_complete(server.serve(sockets=self.sockets))
opal_server_1 | │ │ │ │ │ └ [<gunicorn.sock.TCPSocket object at 0x7f275114d340>]
opal_server_1 | │ │ │ │ └ <uvicorn.workers.UvicornWorker object at 0x7f27511b4a30>
opal_server_1 | │ │ │ └ <function Server.serve at 0x7f275129b940>
opal_server_1 | │ │ └ <uvicorn.server.Server object at 0x7f2750f030d0>
opal_server_1 | │ └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
opal_server_1 | └ <uvloop.Loop running=True closed=False debug=False>
opal_server_1 |
opal_server_1 | File "/opal_common/git/repo_watcher.py", line 68, in run
opal_server_1 | self._tracker = BranchTracker(
opal_server_1 | │ │ └ <class 'opal_common.git.branch_tracker.BranchTracker'>
opal_server_1 | │ └ None
opal_server_1 | └ <opal_common.git.repo_watcher.RepoWatcher object at 0x7f274fe42bb0>
opal_server_1 |
opal_server_1 | File "/opal_common/git/branch_tracker.py", line 42, in __init__
opal_server_1 | self._save_latest_commit_as_prev_commit()
opal_server_1 | │ └ <function BranchTracker._save_latest_commit_as_prev_commit at 0x7f274fe2b790>
opal_server_1 | └ <opal_common.git.branch_tracker.BranchTracker object at 0x7f274fbf2400>
opal_server_1 |
opal_server_1 | File "/opal_common/git/branch_tracker.py", line 82, in _save_latest_commit_as_prev_commit
opal_server_1 | self._prev_commit = self.latest_commit
opal_server_1 | │ │ └ <property object at 0x7f274fe37360>
opal_server_1 | │ └ <opal_common.git.branch_tracker.BranchTracker object at 0x7f274fbf2400>
opal_server_1 | └ <opal_common.git.branch_tracker.BranchTracker object at 0x7f274fbf2400>
opal_server_1 |
opal_server_1 | File "/opal_common/git/branch_tracker.py", line 89, in latest_commit
opal_server_1 | return self.tracked_branch.commit
opal_server_1 | │ └ <property object at 0x7f274fe377c0>
opal_server_1 | └ <opal_common.git.branch_tracker.BranchTracker object at 0x7f274fbf2400>
opal_server_1 |
opal_server_1 | > File "/opal_common/git/branch_tracker.py", line 105, in tracked_branch
opal_server_1 | return getattr(self._repo.heads, self._branch_name)
opal_server_1 | │ │ │ │ └ 'dev'
opal_server_1 | │ │ │ └ <opal_common.git.branch_tracker.BranchTracker object at 0x7f274fbf2400>
opal_server_1 | │ │ └ <property object at 0x7f274feae2c0>
opal_server_1 | │ └ <git.repo.base.Repo '/regoclone/.git'>
opal_server_1 | └ <opal_common.git.branch_tracker.BranchTracker object at 0x7f274fbf2400>
opal_server_1 |
opal_server_1 | File "/root/.local/lib/python3.8/site-packages/git/util.py", line 972, in __getattr__
opal_server_1 | return list.__getattribute__(self, attr)
opal_server_1 | │ └ 'dev'
opal_server_1 | └ [<git.Head "refs/heads/main">]
opal_server_1 |
opal_server_1 | AttributeError: 'IterableList' object has no attribute 'dev'
OPA version inside opal-client
(Not urgent, just some ideas)
OPA published 0.30.1 ten days ago.
https://hub.docker.com/r/openpolicyagent/opa/tags?page=1&ordering=last_updated&name=0.30.1
Currently opal-client 0.1.9 is using OPA 0.29.4 inside.
OPA is out of date. {'current_version': '0.29.4', 'download_opa': 'https://openpolicyagent.org/downloads/v0.30.1/opa_linux_amd64', 'latest_version': '0.30.1', 'release_notes': 'https://github.com/open-policy-agent/opa/releases/tag/v0.30.1', 'time': '2021-07-12T15:47:45Z'}
Ideally, if can implement some automation way (definitely needs a lot of tests) to bump version and keep OPA inside opal-client update, that will be awesome!
Information request about store type
What do you thing to add a new disk store type (an addition of mock and opa) ?
(sorry for my english in advance)
context
- I found simple/and much powerful to use the watch file option of opa than pass by api/put request for store data.
- Our (old) bundler create a bundle (archive with rego policy and json data).
- When we found your projet i start to implement a custom fetcher
- but, unless I missed something, watch feature seem's more easier because there is less temporal coupling: no matter in what order opa/opa-client (re) start .(And there is some other property, like no network, bundle properties).
problem
- Solve (re) start order problem opa/opal-client
- Easier initial state
inference
- Actually there only two store types that is opa server it self and the mock.
- Add new store type for that will allow new integration opa/opal-client: disk
- disk can be mounted/prefil/etc: initial state natural/natif/interinsec
- opa watch option is really great for update opa with new policy data
- bundle have a nice property of location decoupling: (a) the archive can be move easly everywhere, (b) single artifact that contain policy an data, (c) immutable
solution
for now, our old code is invoked into a custom fetcher, but it could be great to have something more clean
According my actual investigation, i suppose the code is to be located into opal_client/policy_store/. i think, a new store, disk_store, can be added.
Conclusion
If and/or about 3 assertions :
- you are interested about this,
- my suggestion is not to bad
- i missed nothing
then
i can try to make a P.R
PS:
- in our world, we use mainly kubernetes, and policy is put in config map , but with your project i start investigate a git repo synchronisation (instead of config map).
- For the bundle creation, in opal-client, i think (? naively ?) when data change we can get last policies, and when policies change, we can last data so bundle is every time up to date.
- the old bundler: get policy from configmap and get data from db, then put in bundle.tgz
Thanks in advance.
Subscription to multiple policy directories does not fully update on new commit
Hi
I have noticed that when opal-client subscribes to multiple policy directories and a new commit is added to the policy branch that not all files across the subscribed directories are pulled from opal-server. This has been tested on 0.1.17.
Steps to reproduce
- Fork or branch authorizon/opal-example-policy-repo and add two directories with some basic rego files (gdmarsh/opal-example-policy-repo@c2fce0a)
- Create or adjust docker-compose file to point to new branch and subscribe to directories (gdmarsh/opal@a3a2b63)
- Start docker-compose
- Curl
http://localhost:8181/v1/data, should return both rules with them being true or undefined result if set to false - Push update to rego files in directories, changing from false to true or vice versa. Must be in the same commit.
- Check opal-client logs, should be able to observe that there is only a single PUT request for a single directory
- Curl
http://localhost:8181/v1/dataagain, only one of the rules should have changed
Logs observed
fastapi_websocket_pubsub.rpc_event_me...| INFO | Received notification of event
opal_client.policy.updater | INFO | Received policy update: affected directories=['topic1'], new commit hash='2c9dee973800a689ab5e848b6408a7c1f26ff1ab'
opal_client.policy.updater | INFO | Refetching policy code (delta bundle), base hash: 'bfd481b5f70b8fa6f1dff1838ea4914f4e937868'
opal_client.policy.updater | INFO | got policy bundle (delta): 'bfd481b5f70b8fa6f1dff1838ea4914f4e937868' -> '2c9dee973800a689ab5e848b6408a7c1f26ff1ab'
opal_client.opa.logger | INFO | Received request. PUT /v1/policies/topic1/policy.rego
opal_client.opa.logger | INFO | Sent response. PUT /v1/policies/topic1/policy.rego -> 200
fastapi_websocket_pubsub.rpc_event_me...| INFO | Received notification of event
opal_client.policy.updater | INFO | Received policy update: affected directories=['topic2'], new commit hash='2c9dee973800a689ab5e848b6408a7c1f26ff1ab'
opal_client.policy.updater | INFO | Refetching policy code (delta bundle), base hash: '2c9dee973800a689ab5e848b6408a7c1f26ff1ab'
opal_client.policy.updater | INFO | got policy bundle (delta): '2c9dee973800a689ab5e848b6408a7c1f26ff1ab' -> '2c9dee973800a689ab5e848b6408a7c1f26ff1ab'
Curl
# initial (allow = true)
curl localhost:8181/v1/data
{"result":{"topic1":{"bug":{"allow":true}},"topic2":{"bug":{"allow":true}}}}
# after push (allow = false)
curl localhost:8181/v1/data
{"result":{"topic1":{"bug":{}},"topic2":{"bug":{"allow":true}}}}
opal-server stops the pulling if failed to connect
When I restart my local Kubernetes today, opal-server seems having a connection issue, then it fails with the error below.
I think when the Kubernetes restarts, it might have some connection issue in the beginning, however, I expect opal-server should not crash.
opal-server logs
[2021-07-22 08:50:57 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2021-07-22 08:50:57 +0000] [1] [INFO] Listening at: http://0.0.0.0:7002 (1)
[2021-07-22 08:50:57 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-07-22 08:50:57 +0000] [9] [INFO] Booting worker with pid: 9
[2021-07-22 08:50:57 +0000] [10] [INFO] Booting worker with pid: 10
[2021-07-22 08:50:57 +0000] [11] [INFO] Booting worker with pid: 11
[2021-07-22 08:50:57 +0000] [12] [INFO] Booting worker with pid: 12
2021-07-22T08:51:03.962012+0000 |opal_common.authentication.signer | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!
2021-07-22T08:51:04.097899+0000 |opal_common.authentication.signer | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!
2021-07-22T08:51:04.211168+0000 |opal_server.server | INFO | triggered startup event
2021-07-22T08:51:04.211353+0000 |opal_common.topics.publisher | INFO | started topic publisher
2021-07-22T08:51:04.213199+0000 |opal_server.server | INFO | leadership lock acquired, leader pid: 12
2021-07-22T08:51:04.213388+0000 |opal_server.server | INFO | listening on webhook topic: 'webhook'
2021-07-22T08:51:04.214861+0000 |fastapi_websocket_pubsub.event_notifier | INFO | New subscription {'id': 'd992603b8bff43ffa767d7afc46c7a05', 'subscriber_id': '75ee773441b14d1fbc5c474332e55b97', 'topic': 'webhook', 'callback': functools.partial(<function trigger_repo_watcher_pull at 0x7f3051fe1e50>, <opal_server.policy.watcher.task.RepoWatcherTask object at 0x7f3051f744f0>), 'notifier_id': None}
2021-07-22T08:51:04.215018+0000 |opal_server.policy.watcher.task | INFO | Launching repo watcher
2021-07-22T08:51:04.216898+0000 |opal_common.git.repo_cloner | INFO | Cloning repo from 'https://github.com/Hongbo-Miao/opal-example-policy-repo' to '/regoclone'
2021-07-22T08:51:04.401109+0000 |opal_common.authentication.signer | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!
2021-07-22T08:51:04.581012+0000 |opal_server.server | INFO | triggered startup event
2021-07-22T08:51:04.582026+0000 |opal_common.topics.publisher | INFO | started topic publisher
2021-07-22T08:51:04.599344+0000 |opal_server.server | INFO | triggered startup event
2021-07-22T08:51:04.599745+0000 |opal_common.topics.publisher | INFO | started topic publisher
2021-07-22T08:51:04.700600+0000 |opal_common.authentication.signer | INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!
2021-07-22T08:51:04.813917+0000 |opal_server.server | INFO | triggered startup event
2021-07-22T08:51:04.817148+0000 |opal_common.topics.publisher | INFO | started topic publisher
2021-07-22T08:51:19.312913+0000 |opal_common.git.repo_cloner |ERROR | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
cmdline: git clone -v https://github.com/Hongbo-Miao/opal-example-policy-repo /regoclone
stderr: 'Cloning into '/regoclone'...
fatal: unable to access 'https://github.com/Hongbo-Miao/opal-example-policy-repo/': Could not resolve host: github.com
'
Traceback (most recent call last):
File "/root/.local/bin/gunicorn", line 8, in <module>
sys.exit(run())
│ │ └ <function run at 0x7f30541e9430>
│ └ <built-in function exit>
└ <module 'sys' (built-in)>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
└ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
super().run()
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
│ └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x7f30541ed790>
└ <class 'gunicorn.arbiter.Arbiter'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 202, in run
self.manage_workers()
│ └ <function Arbiter.manage_workers at 0x7f3053b34f70>
└ <gunicorn.arbiter.Arbiter object at 0x7f30541ed760>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
self.spawn_workers()
│ └ <function Arbiter.spawn_workers at 0x7f3053b350d0>
└ <gunicorn.arbiter.Arbiter object at 0x7f30541ed760>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
self.spawn_worker()
│ └ <function Arbiter.spawn_worker at 0x7f3053b35040>
└ <gunicorn.arbiter.Arbiter object at 0x7f30541ed760>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
│ └ <function UvicornWorker.init_process at 0x7f30535929d0>
└ <uvicorn.workers.UvicornWorker object at 0x7f30531c7250>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 64, in init_process
super(UvicornWorker, self).init_process()
│ └ <uvicorn.workers.UvicornWorker object at 0x7f30531c7250>
└ <class 'uvicorn.workers.UvicornWorker'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 142, in init_process
self.run()
│ └ <function UvicornWorker.run at 0x7f3053592af0>
└ <uvicorn.workers.UvicornWorker object at 0x7f30531c7250>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 77, in run
loop.run_until_complete(server.serve(sockets=self.sockets))
│ │ │ │ │ └ [<gunicorn.sock.TCPSocket object at 0x7f3053308cd0>]
│ │ │ │ └ <uvicorn.workers.UvicornWorker object at 0x7f30531c7250>
│ │ │ └ <function Server.serve at 0x7f30532da9d0>
│ │ └ <uvicorn.server.Server object at 0x7f30531f9e80>
│ └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
└ <uvloop.Loop running=True closed=False debug=False>
File "/opal_common/git/repo_watcher.py", line 63, in run
result = self._cloner.clone()
│ │ └ <function RepoCloner.clone at 0x7f3051fdcb80>
│ └ <opal_common.git.repo_cloner.RepoCloner object at 0x7f3051f742e0>
└ <opal_common.git.repo_watcher.RepoWatcher object at 0x7f3051f74220>
File "/opal_common/git/repo_cloner.py", line 113, in clone
return self._attempt_clone_from_url()
│ └ <function RepoCloner._attempt_clone_from_url at 0x7f3051fdcca0>
└ <opal_common.git.repo_cloner.RepoCloner object at 0x7f3051f742e0>
> File "/opal_common/git/repo_cloner.py", line 136, in _attempt_clone_from_url
repo = _clone_with_retries()
└ <function BaseRetrying.wraps.<locals>.wrapped_f at 0x7f3051d825e0>
File "/root/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 333, in wrapped_f
return self(f, *args, **kw)
│ │ │ └ {}
│ │ └ ()
│ └ functools.partial(<bound method Repo.clone_from of <class 'git.repo.base.Repo'>>, url='https://github.com/Hongbo-Miao/opal-ex...
└ <Retrying object at 0x7f3051d7d8e0 (stop=<tenacity.stop.stop_after_attempt object at 0x7f3051fe20d0>, wait=<tenacity.wait.wai...
File "/root/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 423, in __call__
do = self.iter(retry_state=retry_state)
│ │ └ <tenacity.RetryCallState object at 0x7f3051d7d460>
│ └ <function BaseRetrying.iter at 0x7f30528afca0>
└ <Retrying object at 0x7f3051d7d8e0 (stop=<tenacity.stop.stop_after_attempt object at 0x7f3051fe20d0>, wait=<tenacity.wait.wai...
File "/root/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 372, in iter
raise retry_exc.reraise()
│ └ <function RetryError.reraise at 0x7f30528af1f0>
└ RetryError(<Future at 0x7f3051d7d9a0 state=finished raised GitCommandError>)
File "/root/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 189, in reraise
raise self.last_attempt.result()
│ │ └ <function Future.result at 0x7f305331e280>
│ └ <Future at 0x7f3051d7d9a0 state=finished raised GitCommandError>
└ RetryError(<Future at 0x7f3051d7d9a0 state=finished raised GitCommandError>)
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
└ <Future at 0x7f3051d7d9a0 state=finished raised GitCommandError>
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
raise self._exception
│ └ GitCommandError(['git', 'clone', '-v', 'https://github.com/Hongbo-Miao/opal-example-policy-repo', '/regoclone'], 128, b"Cloni...
└ <Future at 0x7f3051d7d9a0 state=finished raised GitCommandError>
File "/root/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 426, in __call__
result = fn(*args, **kwargs)
│ │ └ {}
│ └ ()
└ functools.partial(<bound method Repo.clone_from of <class 'git.repo.base.Repo'>>, url='https://github.com/Hongbo-Miao/opal-ex...
File "/root/.local/lib/python3.8/site-packages/git/repo/base.py", line 1111, in clone_from
return cls._clone(git, url, to_path, GitCmdObjectDB, progress, multi_options, **kwargs)
│ │ │ │ │ │ │ │ └ {}
│ │ │ │ │ │ │ └ None
│ │ │ │ │ │ └ None
│ │ │ │ │ └ <class 'git.db.GitCmdObjectDB'>
│ │ │ │ └ '/regoclone'
│ │ │ └ 'https://github.com/Hongbo-Miao/opal-example-policy-repo'
│ │ └ <git.cmd.Git object at 0x7f3051d70fa0>
│ └ <classmethod object at 0x7f3051dda070>
└ <class 'git.repo.base.Repo'>
File "/root/.local/lib/python3.8/site-packages/git/repo/base.py", line 1049, in _clone
finalize_process(proc, stderr=stderr)
│ │ └ "Cloning into '/regoclone'...\nfatal: unable to access 'https://github.com/Hongbo-Miao/opal-example-policy-repo/': Could not ...
│ └ <git.cmd.Git.AutoInterrupt object at 0x7f3051d7dc40>
└ <function finalize_process at 0x7f3051e34700>
File "/root/.local/lib/python3.8/site-packages/git/util.py", line 370, in finalize_process
proc.wait(**kwargs)
│ │ └ {'stderr': "Cloning into '/regoclone'...\nfatal: unable to access 'https://github.com/Hongbo-Miao/opal-example-policy-repo/':...
│ └ <function Git.AutoInterrupt.wait at 0x7f3051e66af0>
└ <git.cmd.Git.AutoInterrupt object at 0x7f3051d7dc40>
File "/root/.local/lib/python3.8/site-packages/git/cmd.py", line 447, in wait
raise GitCommandError(remove_password_if_present(self.args), status, errstr)
│ │ │ │ │ └ b"Cloning into '/regoclone'...\nfatal: unable to access 'https://github.com/Hongbo-Miao/opal-example-policy-repo/': Could not...
│ │ │ │ └ 128
│ │ │ └ <member 'args' of 'AutoInterrupt' objects>
│ │ └ <git.cmd.Git.AutoInterrupt object at 0x7f3051d7dc40>
│ └ <function remove_password_if_present at 0x7f3051e34820>
└ <class 'git.exc.GitCommandError'>
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
cmdline: git clone -v https://github.com/Hongbo-Miao/opal-example-policy-repo /regoclone
stderr: 'Cloning into '/regoclone'...
fatal: unable to access 'https://github.com/Hongbo-Miao/opal-example-policy-repo/': Could not resolve host: github.com
'
2021-07-22T08:51:19.334662+0000 |opal_server.policy.watcher.task |ERROR | watcher failed with exception: GitFailed()
2021-07-22T08:51:19.334870+0000 |opal_server.policy.watcher.task | INFO | Stopping repo watcher
2021-07-22T08:51:19.337535+0000 |uvicorn.workers |ERROR | Exception in callback <TaskStepMethWrapper object at 0x7f3051d7ddc0>
handle: <Handle <TaskStepMethWrapper object at 0x7f3051d7ddc0>>
Traceback (most recent call last):
File "/root/.local/bin/gunicorn", line 8, in <module>
sys.exit(run())
│ │ └ <function run at 0x7f30541e9430>
│ └ <built-in function exit>
└ <module 'sys' (built-in)>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
└ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
super().run()
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
│ └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x7f30541ed790>
└ <class 'gunicorn.arbiter.Arbiter'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 202, in run
self.manage_workers()
│ └ <function Arbiter.manage_workers at 0x7f3053b34f70>
└ <gunicorn.arbiter.Arbiter object at 0x7f30541ed760>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
self.spawn_workers()
│ └ <function Arbiter.spawn_workers at 0x7f3053b350d0>
└ <gunicorn.arbiter.Arbiter object at 0x7f30541ed760>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
self.spawn_worker()
│ └ <function Arbiter.spawn_worker at 0x7f3053b35040>
└ <gunicorn.arbiter.Arbiter object at 0x7f30541ed760>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
│ └ <function UvicornWorker.init_process at 0x7f30535929d0>
└ <uvicorn.workers.UvicornWorker object at 0x7f30531c7250>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 64, in init_process
super(UvicornWorker, self).init_process()
│ └ <uvicorn.workers.UvicornWorker object at 0x7f30531c7250>
└ <class 'uvicorn.workers.UvicornWorker'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 142, in init_process
self.run()
│ └ <function UvicornWorker.run at 0x7f3053592af0>
└ <uvicorn.workers.UvicornWorker object at 0x7f30531c7250>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 77, in run
loop.run_until_complete(server.serve(sockets=self.sockets))
│ │ │ │ │ └ [<gunicorn.sock.TCPSocket object at 0x7f3053308cd0>]
│ │ │ │ └ <uvicorn.workers.UvicornWorker object at 0x7f30531c7250>
│ │ │ └ <function Server.serve at 0x7f30532da9d0>
│ │ └ <uvicorn.server.Server object at 0x7f30531f9e80>
│ └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
└ <uvloop.Loop running=True closed=False debug=False>
> File "uvloop/cbhandles.pyx", line 61, in uvloop.loop.Handle._run
callback()
File "/usr/local/lib/python3.8/asyncio/tasks.py", line 718, in cancel
if child.cancel():
│ └ <method 'cancel' of '_asyncio.Task' objects>
└ <Task pending name='Task-4' coro=<RepoWatcher.run() running at /opal_common/git/repo_watcher.py:65> wait_for=<_GatheringFutur...
File "/usr/local/lib/python3.8/asyncio/tasks.py", line 718, in cancel
if child.cancel():
│ └ <method 'cancel' of '_asyncio.Task' objects>
└ <Task cancelling name='Task-5' coro=<RepoWatcherTask._fail() running at /opal_server/policy/watcher/task.py:75> wait_for=<_Ga...
File "/usr/local/lib/python3.8/asyncio/tasks.py", line 718, in cancel
if child.cancel():
│ └ <method 'cancel' of '_asyncio.Task' objects>
└ <Task pending name='Task-4' coro=<RepoWatcher.run() running at /opal_common/git/repo_watcher.py:65> wait_for=<_GatheringFutur...
[Previous line repeated 490 more times]
RecursionError: maximum recursion depth exceeded
More tests wanted
OPAL is pretty well tested (we have over 40 tests) but more tests is always better. :)
If someone wants to contribute more tests, please reach out and we will be happy to guide you :)
OPAL need to alert the user if the policy bundle is invalid
For example - if multiple .rego files have the same package name - OPAL will not complain but OPA will not allow it.
So, invalid configuration is caught during runtime - it should be caught during startup.
e.g: in multi-tenant policy setup: /global/rbac.rego should have different package name than /tenants/tenant1/rbac.rego - otherwise OPA will return 400 when applying the bundle.
Worth mentioning `OPAL_LOG_MODULE_EXCLUDE_LIST='[]'`
Worth mentioning OPAL_LOG_MODULE_EXCLUDE_LIST='[]' in the doc, which is helpful for debugging.
Currently cannot find anything about it in this repo.
Context: https://github.com/authorizon/opal/discussions/81#discussioncomment-913721
Redundant Configuration / Polling for Data Fetchers
Hi,
I'm currently implementing a data fetcher for LDAP and testing it in my docker environment:
https://github.com/phi1010/opal-fetcher-postgres
I'd like to trigger updates, and have two ideas -- could you tell me whether they can be done with the current OPAL framework?
- Polling in Python
Is there an intended way to schedule a later update form a FetcherProvider from within the process() step?
- Lightweight Web Hooks
I could also imagine triggering updates from Cron&Curl, or a Webhook URL that I can trigger with some other existing service.
However, I am not a fan of encoding the source's configuration multiple times -- as far as I can see, I'd have to include the configuration including the URLs and secret credentials in both the docker config for the initial update, and all other webservices triggering further updates from the same source again. This would be quite a hazzle to keep up-to-date.
Also, considering that both trusted SQL- and LDAP-like databases are good at enforcing restrictions, but not that ideal for triggering REST API calls; the data update will likely be triggered by some other tool modifying SQL/LDAP, which is not necessarily trusted that well and should not be able to change the configuration to load data from some other server. Also, not everyone who is allowed to change parts of the database necessarily is allowed to read the complete database, which is also compromised if every service triggering updates requires access to the credentials of the configuration.
For these reasons, I'd like to trigger an update only specifying the key (URL) or topic of an existing data source, can the remaining configuration be taken from the actual configuration, as in https://github.com/authorizon/opal-fetcher-postgres/blob/master/docker-compose.yml#L54 ?
In my attempts, I have only been able to trigger updates successfully when copying and re-specifying the complete configuration, as in https://docs.opal.ac/tutorials/trigger_data_updates
github action: publish docker images to docker hub on push
We want to be able to release the docker images automatically to docker hub when we merge new features.
We have 3 image variations:
opal-server
opal-client
opal-client-standalone
Tags:
When publishing a new image, it should have both the latest and the version tag (i.e: 0.1.5)
github action
We use this action to publish another project, should be very similar
name: Build and publish to Docker Hub
on:
# push:
# branches:
# - master
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
jobs:
# ====== Todos Backend (build and deploy) ======
build-and-publish-image:
runs-on: ubuntu-latest
steps:
-
name: Checkout
uses: actions/checkout@v2
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
-
name: Login to DockerHub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
-
name: Build image and push
id: docker_build
uses: docker/build-push-action@v2
with:
push: true
tags: authorizon/sidecar:latest
context: .
file: ./Dockerfile
-
name: Image digest
run: echo ${{ steps.docker_build.outputs.digest }}
Testing the images (future addition)
we can use the example docker compose and the steps outlined in the playground tutorial to check the built images before pushing to docker hub (sanity check).
Sometimes, the opal-client will return the wrong decision
Sometimes, the opal-client will return the wrong decision (always false in my case which should be true) when opal-server crashes (?). I feel the crash happens most likely because of the internet connection issue because my internet is not stable based on the log below.
If this happens, opal-client will always return wrong decision until I redeploy them.
I can confirm the OPA server has correct data by calling GET http://localhost:8181/v1/data.
At this moment, even I send command POST http://localhost:7002/data/config to opal-server to let opal-client update by
(Although this step is not necessary, because OPA server already return correct data)
The opal-client still returns the wrong decision.
Here are logs:
OPAL server log
[2021-07-09 14:53:59 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2021-07-09 14:53:59 +0000] [1] [INFO] Listening at: http://0.0.0.0:7002 (1)
[2021-07-09 14:53:59 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-07-09 14:53:59 +0000] [10] [INFO] Booting worker with pid: 10
[2021-07-09 14:53:59 +0000] [11] [INFO] Booting worker with pid: 11
[2021-07-09 14:53:59 +0000] [12] [INFO] Booting worker with pid: 12
[2021-07-09 14:53:59 +0000] [13] [INFO] Booting worker with pid: 13
�[32m2021-07-09T14:54:07.286676+0000�[0m |�[34mopal_common.authentication.signer �[0m|�[1m INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!�[0m
�[32m2021-07-09T14:54:08.276578+0000�[0m |�[34mopal_common.authentication.signer �[0m|�[1m INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!�[0m
�[32m2021-07-09T14:54:08.300949+0000�[0m |�[34mopal_server.server �[0m|�[1m INFO | triggered startup event�[0m
�[32m2021-07-09T14:54:08.304489+0000�[0m |�[34mopal_common.topics.publisher �[0m|�[1m INFO | started topic publisher�[0m
�[32m2021-07-09T14:54:08.312564+0000�[0m |�[34mopal_server.server �[0m|�[1m INFO | leadership lock acquired, leader pid: 13�[0m
�[32m2021-07-09T14:54:08.317004+0000�[0m |�[34mopal_server.server �[0m|�[1m INFO | listening on webhook topic: 'webhook'�[0m
�[32m2021-07-09T14:54:08.334346+0000�[0m |�[34mfastapi_websocket_pubsub.event_notifier �[0m|�[1m INFO | New subscription {'id': '4fd3c952c07a4e76aa1d330d21f27f4e', 'subscriber_id': 'f52b2d198847449fbc954f18885b5a18', 'topic': 'webhook', 'callback': functools.partial(<function trigger_repo_watcher_pull at 0x7fb6e63cee50>, <opal_server.policy.watcher.task.RepoWatcherTask object at 0x7fb6e6361340>), 'notifier_id': None}�[0m
�[32m2021-07-09T14:54:08.334538+0000�[0m |�[34mopal_server.policy.watcher.task �[0m|�[1m INFO | Launching repo watcher�[0m
�[32m2021-07-09T14:54:08.343751+0000�[0m |�[34mopal_common.git.repo_cloner �[0m|�[1m INFO | Cloning repo from 'https://github.com/Hongbo-Miao/opal-example-policy-repo' to '/regoclone'�[0m
�[32m2021-07-09T14:54:08.434177+0000�[0m |�[34mopal_common.authentication.signer �[0m|�[1m INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!�[0m
�[32m2021-07-09T14:54:08.517614+0000�[0m |�[34mopal_server.server �[0m|�[1m INFO | triggered startup event�[0m
�[32m2021-07-09T14:54:08.524541+0000�[0m |�[34mopal_common.topics.publisher �[0m|�[1m INFO | started topic publisher�[0m
�[32m2021-07-09T14:54:08.823555+0000�[0m |�[34mopal_common.authentication.signer �[0m|�[1m INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!�[0m
�[32m2021-07-09T14:54:08.870694+0000�[0m |�[34mopal_server.server �[0m|�[1m INFO | triggered startup event�[0m
�[32m2021-07-09T14:54:08.870871+0000�[0m |�[34mopal_common.topics.publisher �[0m|�[1m INFO | started topic publisher�[0m
�[32m2021-07-09T14:54:08.925299+0000�[0m |�[34mopal_server.server �[0m|�[1m INFO | triggered startup event�[0m
�[32m2021-07-09T14:54:08.925504+0000�[0m |�[34mopal_common.topics.publisher �[0m|�[1m INFO | started topic publisher�[0m
[2021-07-09 14:54:38 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:13)
[2021-07-09 14:54:38 +0000] [1] [WARNING] Worker with pid 13 was terminated due to signal 6
[2021-07-09 14:54:38 +0000] [28] [INFO] Booting worker with pid: 28
�[32m2021-07-09T14:54:38.875252+0000�[0m |�[34mopal_server.server �[0m|�[1m INFO | leadership lock acquired, leader pid: 10�[0m
�[32m2021-07-09T14:54:38.875432+0000�[0m |�[34mopal_server.server �[0m|�[1m INFO | listening on webhook topic: 'webhook'�[0m
�[32m2021-07-09T14:54:38.876035+0000�[0m |�[34mfastapi_websocket_pubsub.event_notifier �[0m|�[1m INFO | New subscription {'id': 'e44e830cd9ab41dda7f66725285ff75d', 'subscriber_id': '14ceb257716147a6842061bcbdbe44df', 'topic': 'webhook', 'callback': functools.partial(<function trigger_repo_watcher_pull at 0x7fb6e6320d30>, <opal_server.policy.watcher.task.RepoWatcherTask object at 0x7fb6e6326d90>), 'notifier_id': None}�[0m
�[32m2021-07-09T14:54:38.876196+0000�[0m |�[34mopal_server.policy.watcher.task �[0m|�[1m INFO | Launching repo watcher�[0m
�[32m2021-07-09T14:54:38.897736+0000�[0m |�[34mopal_common.git.repo_cloner �[0m|�[1m INFO | Cloning repo from 'https://github.com/Hongbo-Miao/opal-example-policy-repo' to '/regoclone'�[0m
�[32m2021-07-09T14:54:38.897937+0000�[0m |�[34mopal_common.git.repo_cloner �[0m|�[1m INFO | Repo already exists in '/regoclone'�[0m
�[32m2021-07-09T14:54:38.904031+0000�[0m |�[34mopal_common.git.branch_tracker �[0m|�[31m�[1mERROR | did not find main branch: 'IterableList' object has no attribute 'master', instead found: []�[0m
�[33m�[1mTraceback (most recent call last):�[0m
File "/root/.local/bin/gunicorn", line 8, in <module>
sys.exit(run())
│ │ └ <function run at 0x7fb6e85d43a0>
│ └ <built-in function exit>
└ <module 'sys' (built-in)>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
└ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
super().run()
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
│ └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x7fb6e8643100>
└ <class 'gunicorn.arbiter.Arbiter'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 202, in run
self.manage_workers()
│ └ <function Arbiter.manage_workers at 0x7fb6e7f1fee0>
└ <gunicorn.arbiter.Arbiter object at 0x7fb6e8643160>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
self.spawn_workers()
│ └ <function Arbiter.spawn_workers at 0x7fb6e7f20040>
└ <gunicorn.arbiter.Arbiter object at 0x7fb6e8643160>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
self.spawn_worker()
│ └ <function Arbiter.spawn_worker at 0x7fb6e7f1ff70>
└ <gunicorn.arbiter.Arbiter object at 0x7fb6e8643160>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
│ └ <function UvicornWorker.init_process at 0x7fb6e75b3940>
└ <uvicorn.workers.UvicornWorker object at 0x7fb6e75daaf0>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 64, in init_process
super(UvicornWorker, self).init_process()
│ └ <uvicorn.workers.UvicornWorker object at 0x7fb6e75daaf0>
└ <class 'uvicorn.workers.UvicornWorker'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 142, in init_process
self.run()
│ └ <function UvicornWorker.run at 0x7fb6e75b3a60>
└ <uvicorn.workers.UvicornWorker object at 0x7fb6e75daaf0>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 77, in run
loop.run_until_complete(server.serve(sockets=self.sockets))
│ │ │ │ │ └ [<gunicorn.sock.TCPSocket object at 0x7fb6e76f6b50>]
│ │ │ │ └ <uvicorn.workers.UvicornWorker object at 0x7fb6e75daaf0>
│ │ │ └ <function Server.serve at 0x7fb6e7848940>
│ │ └ <uvicorn.server.Server object at 0x7fb6e796c070>
│ └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
└ <uvloop.Loop running=True closed=False debug=False>
File "�[32m/opal_common/git/�[0m�[32m�[1mrepo_watcher.py�[0m", line �[33m68�[0m, in �[35mrun�[0m
�[1mself�[0m�[35m�[1m.�[0m�[1m_tracker�[0m �[35m�[1m=�[0m �[1mBranchTracker�[0m�[1m(�[0m
�[36m│ │ └ �[0m�[36m�[1m<class 'opal_common.git.branch_tracker.BranchTracker'>�[0m
�[36m│ └ �[0m�[36m�[1mNone�[0m
�[36m└ �[0m�[36m�[1m<opal_common.git.repo_watcher.RepoWatcher object at 0x7fb6e6326ac0>�[0m
File "�[32m/opal_common/git/�[0m�[32m�[1mbranch_tracker.py�[0m", line �[33m42�[0m, in �[35m__init__�[0m
�[1mself�[0m�[35m�[1m.�[0m�[1m_save_latest_commit_as_prev_commit�[0m�[1m(�[0m�[1m)�[0m
�[36m│ └ �[0m�[36m�[1m<function BranchTracker._save_latest_commit_as_prev_commit at 0x7fb6e63163a0>�[0m
�[36m└ �[0m�[36m�[1m<opal_common.git.branch_tracker.BranchTracker object at 0x7fb6e61917f0>�[0m
File "�[32m/opal_common/git/�[0m�[32m�[1mbranch_tracker.py�[0m", line �[33m82�[0m, in �[35m_save_latest_commit_as_prev_commit�[0m
�[1mself�[0m�[35m�[1m.�[0m�[1m_prev_commit�[0m �[35m�[1m=�[0m �[1mself�[0m�[35m�[1m.�[0m�[1mlatest_commit�[0m
�[36m│ │ └ �[0m�[36m�[1m<property object at 0x7fb6e6317450>�[0m
�[36m│ └ �[0m�[36m�[1m<opal_common.git.branch_tracker.BranchTracker object at 0x7fb6e61917f0>�[0m
�[36m└ �[0m�[36m�[1m<opal_common.git.branch_tracker.BranchTracker object at 0x7fb6e61917f0>�[0m
File "�[32m/opal_common/git/�[0m�[32m�[1mbranch_tracker.py�[0m", line �[33m89�[0m, in �[35mlatest_commit�[0m
�[35m�[1mreturn�[0m �[1mself�[0m�[35m�[1m.�[0m�[1mtracked_branch�[0m�[35m�[1m.�[0m�[1mcommit�[0m
�[36m │ └ �[0m�[36m�[1m<property object at 0x7fb6e63175e0>�[0m
�[36m └ �[0m�[36m�[1m<opal_common.git.branch_tracker.BranchTracker object at 0x7fb6e61917f0>�[0m
> File "�[32m/opal_common/git/�[0m�[32m�[1mbranch_tracker.py�[0m", line �[33m105�[0m, in �[35mtracked_branch�[0m
�[35m�[1mreturn�[0m �[1mgetattr�[0m�[1m(�[0m�[1mself�[0m�[35m�[1m.�[0m�[1m_repo�[0m�[35m�[1m.�[0m�[1mheads�[0m�[1m,�[0m �[1mself�[0m�[35m�[1m.�[0m�[1m_branch_name�[0m�[1m)�[0m
�[36m │ │ │ │ └ �[0m�[36m�[1m'master'�[0m
�[36m │ │ │ └ �[0m�[36m�[1m<opal_common.git.branch_tracker.BranchTracker object at 0x7fb6e61917f0>�[0m
�[36m │ │ └ �[0m�[36m�[1m<property object at 0x7fb6e6351130>�[0m
�[36m │ └ �[0m�[36m�[1m<git.repo.base.Repo '/regoclone/.git'>�[0m
�[36m └ �[0m�[36m�[1m<opal_common.git.branch_tracker.BranchTracker object at 0x7fb6e61917f0>�[0m
File "/root/.local/lib/python3.8/site-packages/git/util.py", line 972, in __getattr__
return list.__getattribute__(self, attr)
│ └ 'master'
└ []
�[31m�[1mAttributeError�[0m:�[1m 'IterableList' object has no attribute 'master'�[0m
�[32m2021-07-09T14:54:39.993756+0000�[0m |�[34mopal_common.authentication.signer �[0m|�[1m INFO | OPAL was not provided with JWT encryption keys, cannot verify api requests!�[0m
�[32m2021-07-09T14:54:40.165527+0000�[0m |�[34mopal_server.server �[0m|�[1m INFO | triggered startup event�[0m
�[32m2021-07-09T14:54:40.167811+0000�[0m |�[34mopal_common.topics.publisher �[0m|�[1m INFO | started topic publisher�[0m
�[32m2021-07-09T14:55:28.952305+0000�[0m |�[34mfastapi_websocket_pubsub.event_broadc...�[0m|�[1m INFO | Listening for incoming events from broadcast channel (first listener started)�[0m
�[32m2021-07-09T14:55:28.953804+0000�[0m |�[34mfastapi_websocket_pubsub.event_broadc...�[0m|�[1m INFO | Spawning broadcast listen task�[0m
�[32m2021-07-09T14:55:28.978508+0000�[0m |�[34mfastapi_websocket_pubsub.event_broadc...�[0m|�[1m INFO | Subscribing to ALL TOPICS, and sharing messages with broadcast channel�[0m
�[32m2021-07-09T14:55:28.979728+0000�[0m |�[34mfastapi_websocket_pubsub.event_notifier �[0m|�[1m INFO | New subscription {'id': '9e03e5c118bc4c8c8573527806bc3f67', 'subscriber_id': 'd9d0935ebf934d299f099cf33a4b2b66', 'topic': '__EventNotifier_ALL_TOPICS__', 'callback': <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_websocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7fb6e63268b0>>, 'notifier_id': None}�[0m
�[32m2021-07-09T14:55:28.980905+0000�[0m |�[34mfastapi_websocket_rpc.websocket_rpc_e...�[0m|�[1m INFO | Client connected�[0m
�[32m2021-07-09T14:55:28.982024+0000�[0m |�[34mfastapi_websocket_pubsub.event_broadc...�[0m|�[1m INFO | Starting broadcaster listener�[0m
�[32m2021-07-09T14:55:29.029336+0000�[0m |�[34mfastapi_websocket_pubsub.event_notifier �[0m|�[1m INFO | New subscription {'id': '46ec6da94c0040dcaa2c83d9864947c7', 'subscriber_id': 'becfdb2c5ff94ca8add1531acb8e8dfc', 'topic': 'policy:.', 'callback': <function RpcEventServerMethods.subscribe.<locals>.callback at 0x7fb6e5dcec10>, 'notifier_id': None}�[0m
�[32m2021-07-09T14:55:41.157809+0000�[0m |�[34mopal_common.middleware �[0m|�[31m�[1mERROR | Uncaught server exception: Reference at 'refs/heads/master' does not exist�[0m
�[33m�[1mTraceback (most recent call last):�[0m
File "/root/.local/bin/gunicorn", line 8, in <module>
sys.exit(run())
│ │ └ <function run at 0x7fb6e85d43a0>
│ └ <built-in function exit>
└ <module 'sys' (built-in)>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
└ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
super().run()
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
│ └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x7fb6e8643100>
└ <class 'gunicorn.arbiter.Arbiter'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 202, in run
self.manage_workers()
│ └ <function Arbiter.manage_workers at 0x7fb6e7f1fee0>
└ <gunicorn.arbiter.Arbiter object at 0x7fb6e8643160>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
self.spawn_workers()
│ └ <function Arbiter.spawn_workers at 0x7fb6e7f20040>
└ <gunicorn.arbiter.Arbiter object at 0x7fb6e8643160>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
self.spawn_worker()
│ └ <function Arbiter.spawn_worker at 0x7fb6e7f1ff70>
└ <gunicorn.arbiter.Arbiter object at 0x7fb6e8643160>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
│ └ <function UvicornWorker.init_process at 0x7fb6e75b3940>
└ <uvicorn.workers.UvicornWorker object at 0x7fb6e7748220>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 64, in init_process
super(UvicornWorker, self).init_process()
│ └ <uvicorn.workers.UvicornWorker object at 0x7fb6e7748220>
└ <class 'uvicorn.workers.UvicornWorker'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 142, in init_process
self.run()
│ └ <function UvicornWorker.run at 0x7fb6e75b3a60>
└ <uvicorn.workers.UvicornWorker object at 0x7fb6e7748220>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 77, in run
loop.run_until_complete(server.serve(sockets=self.sockets))
│ │ │ │ │ └ [<gunicorn.sock.TCPSocket object at 0x7fb6e76f6b50>]
│ │ │ │ └ <uvicorn.workers.UvicornWorker object at 0x7fb6e7748220>
│ │ │ └ <function Server.serve at 0x7fb6e7848940>
│ │ └ <uvicorn.server.Server object at 0x7fb6e79a4970>
│ └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
└ <uvloop.Loop running=True closed=False debug=False>
File "/root/.local/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 371, in run_asgi
result = await app(self.scope, self.receive, self.send)
│ │ │ │ │ │ └ <function RequestResponseCycle.send at 0x7fb6e629d9d0>
│ │ │ │ │ └ <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb6e6193c10>
│ │ │ │ └ <function RequestResponseCycle.receive at 0x7fb6e629da60>
│ │ │ └ <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb6e6193c10>
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('127.0.0.1', 7002), 'cl...
│ └ <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb6e6193c10>
└ <uvicorn.middleware.proxy_headers.ProxyHeadersMiddleware object at 0x7fb6e627e220>
File "/root/.local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 59, in __call__
return await self.app(scope, receive, send)
│ │ │ │ └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb6e6193c...
│ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb6e61...
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('127.0.0.1', 7002), 'cl...
│ └ <fastapi.applications.FastAPI object at 0x7fb6e6326e80>
└ <uvicorn.middleware.proxy_headers.ProxyHeadersMiddleware object at 0x7fb6e627e220>
File "/root/.local/lib/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
await super().__call__(scope, receive, send)
│ │ └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb6e6193c...
│ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb6e61...
└ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('127.0.0.1', 7002), 'cl...
File "/root/.local/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
│ │ │ │ └ <bound method RequestResponseCycle.send of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb6e6193c...
│ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb6e61...
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('127.0.0.1', 7002), 'cl...
│ └ <starlette.middleware.errors.ServerErrorMiddleware object at 0x7fb6e62b8490>
└ <fastapi.applications.FastAPI object at 0x7fb6e6326e80>
> File "/root/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
await self.app(scope, receive, _send)
│ │ │ │ └ <function ServerErrorMiddleware.__call__.<locals>._send at 0x7fb6e6196a60>
│ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb6e61...
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('127.0.0.1', 7002), 'cl...
│ └ <starlette.middleware.cors.CORSMiddleware object at 0x7fb6e62b82e0>
└ <starlette.middleware.errors.ServerErrorMiddleware object at 0x7fb6e62b8490>
File "/root/.local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 78, in __call__
await self.app(scope, receive, send)
│ │ │ │ └ <function ServerErrorMiddleware.__call__.<locals>._send at 0x7fb6e6196a60>
│ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb6e61...
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('127.0.0.1', 7002), 'cl...
│ └ <starlette.exceptions.ExceptionMiddleware object at 0x7fb6e62b8400>
└ <starlette.middleware.cors.CORSMiddleware object at 0x7fb6e62b82e0>
File "/root/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
raise exc from None
File "/root/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
await self.app(scope, receive, sender)
│ │ │ │ └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x7fb6e6196af0>
│ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb6e61...
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('127.0.0.1', 7002), 'cl...
│ └ <fastapi.routing.APIRouter object at 0x7fb6e6326e50>
└ <starlette.exceptions.ExceptionMiddleware object at 0x7fb6e62b8400>
File "/root/.local/lib/python3.8/site-packages/starlette/routing.py", line 580, in __call__
await route.handle(scope, receive, send)
│ │ │ │ └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x7fb6e6196af0>
│ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb6e61...
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('127.0.0.1', 7002), 'cl...
│ └ <function Route.handle at 0x7fb6e6c2aee0>
└ <fastapi.routing.APIRoute object at 0x7fb6e62b8850>
File "/root/.local/lib/python3.8/site-packages/starlette/routing.py", line 241, in handle
await self.app(scope, receive, send)
│ │ │ │ └ <function ExceptionMiddleware.__call__.<locals>.sender at 0x7fb6e6196af0>
│ │ │ └ <bound method RequestResponseCycle.receive of <uvicorn.protocols.http.httptools_impl.RequestResponseCycle object at 0x7fb6e61...
│ │ └ {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.1'}, 'http_version': '1.1', 'server': ('127.0.0.1', 7002), 'cl...
│ └ <function request_response.<locals>.app at 0x7fb6e62e3af0>
└ <fastapi.routing.APIRoute object at 0x7fb6e62b8850>
File "/root/.local/lib/python3.8/site-packages/starlette/routing.py", line 52, in app
response = await func(request)
│ └ <starlette.requests.Request object at 0x7fb6e6193e50>
└ <function get_request_handler.<locals>.app at 0x7fb6e62c9700>
File "/root/.local/lib/python3.8/site-packages/fastapi/routing.py", line 204, in app
solved_result = await solve_dependencies(
└ <function solve_dependencies at 0x7fb6e6c25550>
File "/root/.local/lib/python3.8/site-packages/fastapi/dependencies/utils.py", line 548, in solve_dependencies
solved = await call(**sub_values)
│ └ {'repo': <git.repo.base.Repo '/regoclone/.git'>, 'paths': ['.']}
└ <function get_input_paths_or_throw at 0x7fb6e62efd30>
File "�[32m/opal_server/policy/bundles/�[0m�[32m�[1mapi.py�[0m", line �[33m43�[0m, in �[35mget_input_paths_or_throw�[0m
�[35m�[1mwith�[0m �[1mCommitViewer�[0m�[1m(�[0m�[1mrepo�[0m�[35m�[1m.�[0m�[1mhead�[0m�[35m�[1m.�[0m�[1mcommit�[0m�[1m)�[0m �[35m�[1mas�[0m �[1mviewer�[0m�[1m:�[0m
�[36m │ │ └ �[0m�[36m�[1m<property object at 0x7fb6e634b310>�[0m
�[36m │ └ �[0m�[36m�[1m<git.repo.base.Repo '/regoclone/.git'>�[0m
�[36m └ �[0m�[36m�[1m<class 'opal_common.git.commit_viewer.CommitViewer'>�[0m
File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 197, in _get_commit
obj = self._get_object()
│ └ <function SymbolicReference._get_object at 0x7fb6e6377040>
└ <git.HEAD "HEAD">
File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 190, in _get_object
return Object.new_from_sha(self.repo, hex_to_bin(self.dereference_recursive(self.repo, self.path)))
│ │ │ │ │ │ │ │ │ │ └ <member 'path' of 'SymbolicReference' objects>
│ │ │ │ │ │ │ │ │ └ <git.HEAD "HEAD">
│ │ │ │ │ │ │ │ └ <member 'repo' of 'SymbolicReference' objects>
│ │ │ │ │ │ │ └ <git.HEAD "HEAD">
│ │ │ │ │ │ └ <classmethod object at 0x7fb6e63e7c10>
│ │ │ │ │ └ <git.HEAD "HEAD">
│ │ │ │ └ <built-in function a2b_hex>
│ │ │ └ <member 'repo' of 'SymbolicReference' objects>
│ │ └ <git.HEAD "HEAD">
│ └ <classmethod object at 0x7fb6e64054f0>
└ <class 'git.objects.base.Object'>
File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 132, in dereference_recursive
hexsha, ref_path = cls._get_ref_info(repo, ref_path)
│ │ │ │ └ 'refs/heads/master'
│ │ │ └ <git.repo.base.Repo '/regoclone/.git'>
│ │ └ <classmethod object at 0x7fb6e63e7c40>
│ └ <class 'git.refs.head.HEAD'>
└ None
File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 181, in _get_ref_info
return cls._get_ref_info_helper(repo, ref_path)
│ │ │ └ 'refs/heads/master'
│ │ └ <git.repo.base.Repo '/regoclone/.git'>
│ └ <classmethod object at 0x7fb6e63e7c70>
└ <class 'git.refs.head.HEAD'>
File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 164, in _get_ref_info_helper
raise ValueError("Reference at %r does not exist" % ref_path)
└ 'refs/heads/master'
�[31m�[1mValueError�[0m:�[1m Reference at 'refs/heads/master' does not exist�[0m
�[32m2021-07-09T14:55:41.179254+0000�[0m |�[34muvicorn.protocols.http.httptools_impl �[0m|�[1m INFO | 127.0.0.1:41968 - "GET /policy?path=. HTTP/1.1" 500�[0m
Exception in ASGI application
Traceback (most recent call last):
File "/root/.local/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 371, in run_asgi
result = await app(self.scope, self.receive, self.send)
File "/root/.local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 59, in __call__
return await self.app(scope, receive, send)
File "/root/.local/lib/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
await super().__call__(scope, receive, send)
File "/root/.local/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "/root/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
raise exc from None
File "/root/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
await self.app(scope, receive, _send)
File "/root/.local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 78, in __call__
await self.app(scope, receive, send)
File "/root/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
raise exc from None
File "/root/.local/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
await self.app(scope, receive, sender)
File "/root/.local/lib/python3.8/site-packages/starlette/routing.py", line 580, in __call__
await route.handle(scope, receive, send)
File "/root/.local/lib/python3.8/site-packages/starlette/routing.py", line 241, in handle
await self.app(scope, receive, send)
File "/root/.local/lib/python3.8/site-packages/starlette/routing.py", line 52, in app
response = await func(request)
File "/root/.local/lib/python3.8/site-packages/fastapi/routing.py", line 204, in app
solved_result = await solve_dependencies(
File "/root/.local/lib/python3.8/site-packages/fastapi/dependencies/utils.py", line 548, in solve_dependencies
solved = await call(**sub_values)
File "/opal_server/policy/bundles/api.py", line 43, in get_input_paths_or_throw
with CommitViewer(repo.head.commit) as viewer:
File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 197, in _get_commit
obj = self._get_object()
File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 190, in _get_object
return Object.new_from_sha(self.repo, hex_to_bin(self.dereference_recursive(self.repo, self.path)))
File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 132, in dereference_recursive
hexsha, ref_path = cls._get_ref_info(repo, ref_path)
File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 181, in _get_ref_info
return cls._get_ref_info_helper(repo, ref_path)
File "/root/.local/lib/python3.8/site-packages/git/refs/symbolic.py", line 164, in _get_ref_info_helper
raise ValueError("Reference at %r does not exist" % ref_path)
ValueError: Reference at 'refs/heads/master' does not exist
[2021-07-09 14:56:14 +0000] [1] [WARNING] Worker with pid 26 was terminated due to signal 13
�[32m2021-07-09T16:57:32.134994+0000�[0m |�[34mopal_server.data.data_update_publisher �[0m|�[1m INFO | [28] Publishing data update to topics: ['policy_data'], reason: update data, entries: [('https://api.jsonbin.io/b/60dfc0079328b059d7b55dc7', 'PUT', '/')]�[0m
�[32m2021-07-09T16:57:32.138498+0000�[0m |�[34muvicorn.protocols.http.httptools_impl �[0m|�[1m INFO | 127.0.0.1:60420 - "POST /data/config HTTP/1.1" 200�[0m
�[32m2021-07-09T16:57:32.204103+0000�[0m |�[34mfastapi_websocket_pubsub.event_broadc...�[0m|�[1m INFO | Subscribing to ALL TOPICS, and sharing messages with broadcast channel�[0m
�[32m2021-07-09T16:57:32.209478+0000�[0m |�[34mfastapi_websocket_pubsub.event_notifier �[0m|�[1m INFO | New subscription {'id': '6d50ce9d338844edaa5adf585207f482', 'subscriber_id': 'b84f6d94cea54920b0ffc75084a0bb7d', 'topic': '__EventNotifier_ALL_TOPICS__', 'callback': <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_websocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7fb6e6324bb0>>, 'notifier_id': None}�[0m
�[32m2021-07-09T16:57:32.210396+0000�[0m |�[34mfastapi_websocket_pubsub.event_notifier �[0m|�[1m INFO | calling subscription callbacks for sub_id=b84f6d94cea54920b0ffc75084a0bb7d with topic=policy_data�[0m
�[32m2021-07-09T16:57:32.210826+0000�[0m |�[34mfastapi_websocket_pubsub.event_broadc...�[0m|�[1m INFO | Broadcasting incoming event�[0m
�[32m2021-07-09T16:57:32.268775+0000�[0m |�[34mfastapi_websocket_pubsub.event_broadc...�[0m|�[1m INFO | Unsubscribing from ALL TOPICS�[0m
�[32m2021-07-09T16:57:32.269140+0000�[0m |�[34mfastapi_websocket_pubsub.event_notifier �[0m|�[1m INFO | Removing Subscription of topic='__EventNotifier_ALL_TOPICS__' for subscriber=b84f6d94cea54920b0ffc75084a0bb7d�[0m
OPAL client log
[2021-07-09 14:55:27 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2021-07-09 14:55:27 +0000] [1] [INFO] Listening at: http://0.0.0.0:7000 (1)
[2021-07-09 14:55:27 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-07-09 14:55:27 +0000] [7] [INFO] Booting worker with pid: 7
�[32m2021-07-09T14:55:27.865542+0000�[0m |�[34mopal_common.fetcher.fetcher_register �[0m|�[1m INFO | Fetcher Register loaded�[0m
�[32m2021-07-09T14:55:27.888682+0000�[0m |�[34mopal_client.opa.runner �[0m|�[1m INFO | Launching opa runner�[0m
�[32m2021-07-09T14:55:27.889972+0000�[0m |�[34mopal_client.opa.runner �[0m|�[1m INFO | Running OPA inline: opa run --server --addr=:8181 --authentication=off --authorization=off --log-level=info�[0m
�[32m2021-07-09T14:55:27.919718+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Initializing server. �[38;2;191;191;191m{'addrs': [':8181'], 'diagnostic-addrs': [], 'time': '2021-07-09T14:55:27Z'}�[0m�[1m�[0m
�[32m2021-07-09T14:55:28.898638+0000�[0m |�[34mopal_client.opa.runner �[0m|�[1m INFO | Running OPA initial start callbacks�[0m
�[32m2021-07-09T14:55:28.899032+0000�[0m |�[34mopal_client.policy.updater �[0m|�[1m INFO | Launching policy updater�[0m
�[32m2021-07-09T14:55:28.899366+0000�[0m |�[34mopal_client.data.updater �[0m|�[1m INFO | Launching data updater�[0m
�[32m2021-07-09T14:55:28.899532+0000�[0m |�[34mopal_client.policy.updater �[0m|�[1m INFO | Subscribing to topics: ['policy:.']�[0m
�[32m2021-07-09T14:55:28.899910+0000�[0m |�[34mopal_client.data.updater �[0m|�[1m INFO | Subscribing to topics: ['policy_data']�[0m
�[32m2021-07-09T14:55:28.900406+0000�[0m |�[34mfastapi_websocket_pubsub.pub_sub_client �[0m|�[1m INFO | Trying to connect to Pub/Sub server - ws://opal-server-service.hm:7002/ws�[0m
�[32m2021-07-09T14:55:28.900969+0000�[0m |�[34mfastapi_websocket_rpc.websocket_rpc_c...�[0m|�[1m INFO | Trying server - ws://opal-server-service.hm:7002/ws�[0m
�[32m2021-07-09T14:55:28.905403+0000�[0m |�[34mfastapi_websocket_pubsub.pub_sub_client �[0m|�[1m INFO | Trying to connect to Pub/Sub server - ws://opal-server-service.hm:7002/ws�[0m
�[32m2021-07-09T14:55:28.906055+0000�[0m |�[34mfastapi_websocket_rpc.websocket_rpc_c...�[0m|�[1m INFO | Trying server - ws://opal-server-service.hm:7002/ws�[0m
�[32m2021-07-09T14:55:29.034726+0000�[0m |�[34mopal_client.data.updater �[0m|�[1m INFO | Connected to server�[0m
�[32m2021-07-09T14:55:29.035293+0000�[0m |�[34mopal_client.data.updater �[0m|�[1m INFO | Performing data configuration, reason: Initial load�[0m
�[32m2021-07-09T14:55:29.036485+0000�[0m |�[34mopal_client.data.updater �[0m|�[1m INFO | Getting data-sources configuration from 'http://opal-server-service.hm:7002/data/config'�[0m
�[32m2021-07-09T14:55:29.047656+0000�[0m |�[34mopal_client.policy.updater �[0m|�[1m INFO | Connected to server�[0m
�[32m2021-07-09T14:55:29.048375+0000�[0m |�[34mopal_client.policy.updater �[0m|�[1m INFO | Refetching policy code (full bundle)�[0m
�[32m2021-07-09T14:55:29.071238+0000�[0m |�[34mopal_client.data.updater �[0m|�[1m INFO | Triggering data update with id: ecd2ef259e97418a8155a0afa9a5ec8d�[0m
�[32m2021-07-09T14:55:29.072536+0000�[0m |�[34mopal_client.data.updater �[0m|�[1m INFO | Fetching policy data�[0m
�[32m2021-07-09T14:55:29.073086+0000�[0m |�[34mopal_client.data.fetcher �[0m|�[1m INFO | Fetching data from url: https://api.jsonbin.io/b/60dfc0079328b059d7b55dc7�[0m
�[32m2021-07-09T14:55:29.099561+0000�[0m |�[34mfastapi_websocket_pubsub.pub_sub_client �[0m|�[1m INFO | Connected to PubSub server ws://opal-server-service.hm:7002/ws�[0m
�[32m2021-07-09T14:55:29.100549+0000�[0m |�[34mopal_client.policy.fetcher �[0m|�[33m�[1mWARNING | Unexpected response code 500: {'error': 'Uncaught server exception'}�[0m
�[32m2021-07-09T14:55:29.300109+0000�[0m |�[34mopal_client.policy.fetcher �[0m|�[33m�[1mWARNING | Unexpected response code 500: {'error': 'Uncaught server exception'}�[0m
�[32m2021-07-09T14:55:29.650551+0000�[0m |�[34mopal_client.policy.fetcher �[0m|�[33m�[1mWARNING | Unexpected response code 500: {'error': 'Uncaught server exception'}�[0m
�[32m2021-07-09T14:55:31.006926+0000�[0m |�[34mopal_client.data.updater �[0m|�[1m INFO | Saving fetched data to policy-store: source url='https://api.jsonbin.io/b/60dfc0079328b059d7b55dc7', destination path='/'�[0m
�[32m2021-07-09T14:55:31.010731+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Received request. �[38;2;153;153;153mPUT /v1/data�[0m�[1m�[0m
�[32m2021-07-09T14:55:31.013705+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Sent response. �[38;2;153;153;153mPUT /v1/data -> 204�[0m�[1m�[0m
�[32m2021-07-09T14:55:33.289360+0000�[0m |�[34mopal_client.policy.fetcher �[0m|�[33m�[1mWARNING | Unexpected response code 500: {'error': 'Uncaught server exception'}�[0m
�[32m2021-07-09T14:55:41.181722+0000�[0m |�[34mopal_client.policy.fetcher �[0m|�[33m�[1mWARNING | Unexpected response code 500: {'error': 'Uncaught server exception'}�[0m
�[32m2021-07-09T14:55:41.182418+0000�[0m |�[34mopal_client.policy.fetcher �[0m|�[33m�[1mWARNING | Failed all attempts to fetch bundle, got error: ValueError('unexpected response code while fetching bundle: 500')�[0m
�[32m2021-07-09T14:55:41.182792+0000�[0m |�[34mfastapi_websocket_pubsub.pub_sub_client �[0m|�[1m INFO | Connected to PubSub server ws://opal-server-service.hm:7002/ws�[0m
�[32m2021-07-09T15:55:28.918330+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | OPA is out of date. �[38;2;191;191;191m{'current_version': '0.29.4', 'download_opa': 'https://openpolicyagent.org/downloads/v0.30.1/opa_linux_amd64', 'latest_version': '0.30.1', 'release_notes': 'https://github.com/open-policy-agent/opa/releases/tag/v0.30.1', 'time': '2021-07-09T15:55:28Z'}�[0m�[1m�[0m
�[32m2021-07-09T16:53:26.516297+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Received request. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow�[0m�[1m�[0m
�[32m2021-07-09T16:53:26.532144+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Sent response. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow -> 200�[0m�[1m�[0m
�[32m2021-07-09T16:53:30.098411+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Received request. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow�[0m�[1m�[0m
�[32m2021-07-09T16:53:30.100493+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Sent response. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow -> 200�[0m�[1m�[0m
�[32m2021-07-09T16:53:30.762611+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Received request. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow�[0m�[1m�[0m
�[32m2021-07-09T16:53:30.762988+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Sent response. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow -> 200�[0m�[1m�[0m
�[32m2021-07-09T16:55:55.647137+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Received request. �[38;2;153;153;153mGET /v1/data�[0m�[1m�[0m
�[32m2021-07-09T16:55:55.649482+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Sent response. �[38;2;153;153;153mGET /v1/data -> 200�[0m�[1m�[0m
�[32m2021-07-09T16:56:06.166064+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Received request. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow�[0m�[1m�[0m
�[32m2021-07-09T16:56:06.166754+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Sent response. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow -> 200�[0m�[1m�[0m
�[32m2021-07-09T16:57:32.291092+0000�[0m |�[34mopal_client.data.rpc �[0m|�[1m INFO | Received notification of event: policy_data�[0m
�[32m2021-07-09T16:57:32.291589+0000�[0m |�[34mopal_client.data.updater �[0m|�[1m INFO | Updating policy data, reason: update data�[0m
�[32m2021-07-09T16:57:32.292754+0000�[0m |�[34mopal_client.data.updater �[0m|�[1m INFO | Triggering data update with id: 3ef90741831f471f8816f3a147f38637�[0m
�[32m2021-07-09T16:57:32.294238+0000�[0m |�[34mopal_client.data.updater �[0m|�[1m INFO | Fetching policy data�[0m
�[32m2021-07-09T16:57:32.294702+0000�[0m |�[34mopal_client.data.fetcher �[0m|�[1m INFO | Fetching data from url: https://api.jsonbin.io/b/60dfc0079328b059d7b55dc7�[0m
�[32m2021-07-09T16:57:35.218255+0000�[0m |�[34mopal_client.data.updater �[0m|�[1m INFO | Saving fetched data to policy-store: source url='https://api.jsonbin.io/b/60dfc0079328b059d7b55dc7', destination path='/'�[0m
�[32m2021-07-09T16:57:35.222903+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Received request. �[38;2;153;153;153mPUT /v1/data�[0m�[1m�[0m
�[32m2021-07-09T16:57:35.225645+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Sent response. �[38;2;153;153;153mPUT /v1/data -> 204�[0m�[1m�[0m
�[32m2021-07-09T16:57:35.287520+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Received request. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow�[0m�[1m�[0m
�[32m2021-07-09T16:57:35.289553+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Sent response. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow -> 200�[0m�[1m�[0m
�[32m2021-07-09T16:57:36.022564+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Received request. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow�[0m�[1m�[0m
�[32m2021-07-09T16:57:36.022919+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Sent response. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow -> 200�[0m�[1m�[0m
�[32m2021-07-09T16:57:36.557666+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Received request. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow�[0m�[1m�[0m
�[32m2021-07-09T16:57:36.558506+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Sent response. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow -> 200�[0m�[1m�[0m
�[32m2021-07-09T16:57:36.972983+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Received request. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow�[0m�[1m�[0m
�[32m2021-07-09T16:57:36.976383+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Sent response. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow -> 200�[0m�[1m�[0m
�[32m2021-07-09T16:59:44.994748+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Received request. �[38;2;153;153;153mGET /v1/data�[0m�[1m�[0m
�[32m2021-07-09T16:59:44.995646+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Sent response. �[38;2;153;153;153mGET /v1/data -> 200�[0m�[1m�[0m
�[32m2021-07-09T16:59:48.486033+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Received request. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow�[0m�[1m�[0m
�[32m2021-07-09T16:59:48.487769+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Sent response. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow -> 200�[0m�[1m�[0m
�[32m2021-07-09T17:00:00.145229+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Received request. �[38;2;153;153;153mGET /v1/data/users/bob�[0m�[1m�[0m
�[32m2021-07-09T17:00:00.147119+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Sent response. �[38;2;153;153;153mGET /v1/data/users/bob -> 200�[0m�[1m�[0m
�[32m2021-07-09T17:00:04.208660+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Received request. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow�[0m�[1m�[0m
�[32m2021-07-09T17:00:04.210214+0000�[0m |�[34mopal_client.opa.logger �[0m|�[1m INFO | Sent response. �[38;2;153;153;153mPOST /v1/data/app/rbac/allow -> 200�[0m�[1m�[0m
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.