google / upvote_py2 Goto Github PK
View Code? Open in Web Editor NEWA multi-platform binary whitelisting solution
License: Apache License 2.0
A multi-platform binary whitelisting solution
License: Apache License 2.0
Right now it looks like upvote controls the ClientMode setting - I'd prefer to do this with our existing MDM solution via the plist we push to set the basic santa settings - this would allow us to continue to use our various mdm computer groups rather than re-inventing that in upvote
Hi,
Just recently I deployed an upvote (built with bazel-0.25.0, since this is more in line with date of last commit to this repo).
Overall the app functions ok, Santa clients are able to sync. GAE cron jobs also run successfully.
Though looking into logs, there are quite a few 500 errors from /_ah/queue/deferred
handler with backtraces like below
Is this something expected with current repo?
Thanks a lot
Ivan.
backtrace:
'tuple' object has no attribute 'get_async'
Traceback (most recent call last):
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
rv = self.handle_exception(request, response, e)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 316, in post
self.run_from_request()
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 311, in run_from_request
run(self.request.body)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 153, in run
return func(*args, **kwds)
File "/base/data/home/apps/u~proj-id/20210111t191206.432278677749216289/upvote/gae/datastore/utils.py", line 375, in _QueuedPaginatedBatchApply
callback(last_page_results, *extra_args, **extra_kwargs)
File "/base/data/home/apps/u~proj-id/20210111t191206.432278677749216289/upvote/gae/cron/role_syncing.py", line 269, in _SpiderBite
hosts = ndb.get_multi(host_keys)
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3933, in get_multi
for future in get_multi_async(keys, **ctx_options)]
File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3918, in get_multi_async
return [key.get_async(**ctx_options) for key in keys]
AttributeError: 'tuple' object has no attribute 'get_async'
$ ./init_project.sh
Initializing Upvote for GCP project: "white-feat-249718"
+ gcloud config set project white-feat-249718
Updated property [core/project].
+ echo Enabling App Engine...
Enabling App Engine...
++ gcloud app describe '--format=value(id)'
+ [[ '' != \w\h\i\t\e\-\f\e\a\t\-\2\4\9\7\1\8 ]]
+ gcloud app create
You are creating an app for project [white-feat-249718].
WARNING: Creating an App Engine application for a project is irreversible and the region
cannot be changed. More information about regions is at
<https://cloud.google.com/appengine/docs/locations>.
[ Snip ]
Creating App Engine application in project [white-feat-249718] and region [us-central]....done.
Success! The app is now created. Please use `gcloud app deploy` to deploy your first app.
+ echo Enabling APIs used by Upvote...
Enabling APIs used by Upvote...
+ gcloud services enable cloudkms.googleapis.com
Operation "operations/acf.fac977e7-0f57-45e4-8327-ca1329fd2b06" finished successfully.
+ gcloud services enable bigquery-json.googleapis.com
+ echo Creating encryption keys used to store Upvote API secrets...
Creating encryption keys used to store Upvote API secrets...
+ gcloud kms keyrings create ring --location=global
+ gcloud kms keys create virustotal --purpose=encryption --keyring=ring --location=global
+ echo Granting necessary permissions to App Engine...
Granting necessary permissions to App Engine...
++ gcloud iam service-accounts list '--filter=App Engine app default service account' '--format=value(email)'
+ [email protected]
+ gcloud projects add-iam-policy-binding white-feat-249718 --member serviceAccount:[email protected] --role roles/cloudkms.cryptoKeyEncrypterDecrypter
Updated IAM policy for project [white-feat-249718].
bindings:
- members:
- serviceAccount:[email protected]
role: roles/cloudkms.cryptoKeyEncrypterDecrypter
- members:
- serviceAccount:[email protected]
role: roles/editor
- members:
- user:[email protected]
- user:[email protected]
role: roles/owner
etag: BwWQBBec2ao=
version: 1
+ echo Configuring App Engine...
Configuring App Engine...
+ ./manage_crons.py disable_all
+ echo Deploying temporary default version to App Engine...
Deploying temporary default version to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy -- white-feat-249718 app.yaml
+ echo
+ echo Deploying to App Engine...
Deploying to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy -- white-feat-249718 app.yaml
INFO: Call stack for the definition of repository 'remotejdk11_macos' which is a http_archive (rule definition at /private/var/tmp/_bazel_summit.hainey/f4e5186114806b0b8302ec418f57ad1d/external/bazel_tools/tools/build_defs/repo/http.bzl:237:16):
- /DEFAULT.WORKSPACE.SUFFIX:219:1
ERROR: /private/var/tmp/_bazel_summit.hainey/f4e5186114806b0b8302ec418f57ad1d/external/com_google_protobuf/protobuf.bzl:130:19: Traceback (most recent call last): File "/private/var/tmp/_bazel_summit.hainey/f4e5186114806b0b8302ec418f57ad1d/external/com_google_protobuf/protobuf.bzl", line 125
rule(attrs = {"srcs": attr.label_list...()}, <2 more arguments>)
File "/private/var/tmp/_bazel_summit.hainey/f4e5186114806b0b8302ec418f57ad1d/external/com_google_protobuf/protobuf.bzl", line 130, in rule
attr.label(cfg = "host", executable = True, sin..., ...)
'single_file' is no longer supported. use allow_single_file instead. You can use --incompatible_disable_deprecated_attr_params=false to temporarily disable this check.
ERROR: /private/var/tmp/_bazel_summit.hainey/f4e5186114806b0b8302ec418f57ad1d/external/com_google_javascript_closure_compiler/BUILD:7:1: error loading package '@com_google_protobuf//': Ex
tension file 'protobuf.bzl' has errors and referenced by '@com_google_javascript_closure_compiler//:com_google_javascript_closure_compiler'
ERROR: Analysis of target '//upvote/gae:monolith_binary.deploy' failed; build aborted: error loading package '@com_google_protobuf//': Extension file 'protobuf.bzl' has errors
INFO: Elapsed time: 2.280s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
currently loading: @com_google_protobuf//
Fetching @gcloud_auth_httplib2_git; Cloning 136da2cd50aa7deb769062cf1d77259d64743a7f of https://github.com/GoogleCloudPlatform/google-auth-library-python-httplib2.git
Getting this error when installing
environment:
I tried:
error on the last command bazel run upvote/gae:monolith_binary.deploy -- "${PROJ_ID}" app.yaml
+ gcloud config set project redacted
Updated property [core/project].
+ echo Deploying to App Engine...
Deploying to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy -- redacted-app.yaml
Starting local Bazel server and connecting to it...
INFO: Writing tracer profile to '/private/var/tmp/_bazel_/7efb8f1f32f0c308f1333fb4570cfc62/command.profile.gz'
ERROR: /private/var/tmp/_bazel_/7efb8f1f32f0c308f1333fb4570cfc62/external/com_google_protobuf/protobuf.bzl:130:19: Traceback (most recent call last):
File "/private/var/tmp/_bazel_/7efb8f1f32f0c308f1333fb4570cfc62/external/com_google_protobuf/protobuf.bzl", line 125
rule(<3 more arguments>)
File "/private/var/tmp/_bazel_/7efb8f1f32f0c308f1333fb4570cfc62/external/com_google_protobuf/protobuf.bzl", line 130, in rule
attr.label(cfg = "host", <3 more arguments>)
'single_file' is no longer supported. use allow_single_file instead. You can use --incompatible_disable_deprecated_attr_params=false to temporarily disable this check.
INFO: Call stack for the definition of repository 'remote_java_tools_darwin' which is a http_archive (rule definition at /private/var/tmp/_bazel_/7efb8f1f32f0c308f1333fb4570cfc62/external/bazel_tools/tools/build_defs/repo/http.bzl:292:16):
- /private/var/tmp/_bazel_/7efb8f1f32f0c308f1333fb4570cfc62/external/bazel_tools/tools/build_defs/repo/utils.bzl:205:9
- /DEFAULT.WORKSPACE.SUFFIX:280:1
ERROR: /private/var/tmp/_bazel_/7efb8f1f32f0c308f1333fb4570cfc62/external/com_google_javascript_closure_compiler/BUILD:7:1: error loading package '@com_google_protobuf//': Extension file 'protobuf.bzl' has errors and referenced by '@com_google_javascript_closure_compiler//:com_google_javascript_closure_compiler'
ERROR: Analysis of target '//upvote/gae:monolith_binary.deploy' failed; build aborted: error loading package '@com_google_protobuf//': Extension file 'protobuf.bzl' has errors
INFO: Elapsed time: 11.606s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (95 packages loaded, 1507 targets configured)
FAILED: Build did NOT complete successfully (95 packages loaded, 1507 targets configured)
Fetching @local_config_xcode; fetching 7s
Fetching @com_google_javascript_closure_library; Restarting.```
ERROR: /Users/amohr/dev/upvote/upvote/gae/modules/upvote_app/frontend/BUILD:20:1: Checking 116 JS files in //upvote/gae/modules/upvote_app/frontend:closure_lib failed (Exit 1)
upvote/gae/modules/upvote_app/frontend/web_ui/detailpage/detail-controller.js:43: WARNING - Please do not use goog.scope to hide declarations.
It is preferable to either create an @private namespaced declaration, or migrate to goog.module.
let ComponentRegistry;
^
ProTip: "JSC_GOOG_SCOPE_HIDDEN_TYPE" can be added to the `suppress` attribute of:
//upvote/gae/modules/upvote_app/frontend:closure_lib
upvote/gae/modules/upvote_app/frontend/admin_ui/blockablepage/blockablepage-controller.js:17: ERROR - Namespace not provided by any srcs or direct deps of //upvote/gae/modules/upvote_app/frontend:closure_lib.
goog.require('goog.dom.safe');
^
ProTip: "CR_NOT_PROVIDED" or "strictDependencies" can be added to the `suppress` attribute of:
//upvote/gae/modules/upvote_app/frontend:closure_lib
1 error(s), 1 warning(s)
guessing the closure lib version needs to be updated?
we just got this when attempting to do a clean full sync for the first time since one of our machines was not downloading all the rules:
BadRequestError: cursor position is outside the range of the original query
at check_rpc_success (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py:1379)
at __query_result_hook (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/datastore/datastore_query.py:2910)
at get_result (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py:615)
at _on_rpc_completion (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py:513)
at run_to_queue (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py:969)
at run_to_queue (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py:1999)
at _help_tasklet_along (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py:427)
at run_to_queue (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py:962)
at helper (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/context.py:890)
at has_next_async (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py:1793)
at _help_tasklet_along (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py:427)
at _fetch_page_async (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py:1380)
at _help_tasklet_along (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py:427)
at get_result (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py:383)
at fetch_page (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py:1362)
at positional_wrapper (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/utils.py:160)
at post (/base/data/home/apps/m~[PROJ]/santa-api:20190204t133533.415891392575063416/upvote/gae/modules/santa_api/sync.py:923)
at _RecordRequest (/base/data/home/apps/m~[PROJ]/santa-api:20190204t133533.415891392575063416/upvote/gae/utils/handler_utils.py:105)
at dispatch (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:545)
The sync server code currently supports populating those fields on preflight but Upvote offers no UI means of configuring defaults/values for these fields.
I see the path of least resistance being the following:
Environment
configuration struct: SANTA_DIRECTORY_{WHITE,BLACK}LIST_REGEX
For two different reasons:
We added a "state" column to the events admin page to keep track of the current state of each event's blockable state. To do this we needed to do a page-size of requests to get the blockables, and then get any certs which were available. This is a lot of API calls and ends up making our instance count often peak
Similarly on the blockables page we grab any certs which are available for blockables to get a better state result.
I can see this being solved in one of two ways:
On the events query API call, a parameter which controls if you want it to return the associated blockables/certs...and similarly on the blockables query call to return any certs/votes associated with the results
Having batch get support for certs/blockables.
thoughts? It would be good to handle #28 as well with the design.
I'm seeing many PRs that touch the DB schemas and no discussion about if the new releases support DB migration to the new schemas. What is the DB migration strategy and is there one?
we had an accident where I accidentally deployed from the master branch instead of our personal branch, which ended up resetting SANTA_DEFAULT_CLIENT_MODE
. I would have imagined this would have only affected new hosts, however ALL hosts ended up getting reset to lockdown mode triggering a lot of unhappiness :)
I think for hosts that are in the datastore should not be affected by a change to this variable.
per https://github.com/google/upvote/blob/master/upvote/gae/cron.yaml#L20 it's listed as /cron/export/datastore-to-gcs
, however per my reading and tests it seems like it should be /cron/datastore/backup
Here's a set of things we've done, not sure if you guys would like PRs or incorporate any in future releases
fix certificate state propagation on admin blockables page
support compact listing: farmersbusinessnetwork@e8e87dd
fix updatedDt on blockables page, fix details of certificates, and add "type" to admin blockables page: farmersbusinessnetwork@8b3964b (later fixed to use prettifyType: farmersbusinessnetwork@7b456eb)
don't navigate to new blockables page when clicking on item, just update card (as it was intended to do): farmersbusinessnetwork@b3e5bf1
don't navigate when clicking on certificate of blockable in card on admin page: farmersbusinessnetwork@64f2a83
add filtering by event type to Events page: farmersbusinessnetwork@d2e6919
add datadog monitoring support (need to investigate why not working on my private account)
search by primary user on hosts page: farmersbusinessnetwork@4c87e9b
general list enhancements: wildcard search support, sorting, title of search box in hosts doesn't update on radio button click (#20)
add state to list of events
add global whitelist/blacklist to settings page: farmersbusinessnetwork@fa30e7c (be careful as it also whitelists homebrew)
add global/host whitelisting status to events page: farmersbusinessnetwork@c8b91ce
Hosts: display + toggle support for transitive whitelisting, and display white/block listing farmersbusinessnetwork@fe726bc
Events: enable parent_name
: farmersbusinessnetwork@5cf412f
Support for host based path based regex setting, and ability to "go to bundle" from events page: farmersbusinessnetwork@45aa766
Support GUI compiler whitelisting: farmersbusinessnetwork@3e91cba (fixes a refresh bug with "reset" as well) First set binary as compiler, then up-vote to populate correct rule
Bump to use latest bazel appengine which uses gcloud deploy instead of appcfg: farmersbusinessnetwork@05abd2d
Add support for google cloud monitoring: farmersbusinessnetwork@8537d00
Add support for showing and searching hosts by serial number: farmersbusinessnetwork@f1cab34
search by host by ID: farmersbusinessnetwork@2f69dcb + farmersbusinessnetwork@2d30210 + farmersbusinessnetwork@d9122a1
we were in full emergency mode today and hit the emergency button (which doesn't explain what it does on the page), which confusingly brought up a "stop" and "go" radio button with no explanations what they do, so we figured we needed to hit "go", after which another go button popped up, which we hit, and have no effect of disabling lockdown mode.
from in the search() method it assumes all searches by id
will load a card, however on the host page there is no card. Further, you're not allowed to use a searchBase of id
so it's broken in a couple ways
I'm having trouble getting machines to check in. I have tried to follow your setup instructions, and I have constructed a configuration profile for the santa client with
<key>SyncBaseURL</key>
<string>https://<my-app>.appspot.com/api/santa</string>
When I do this I get:
santactl sync
HTTP Response: 302 found
Preflight failed, aborting run
I have tried adding a trailing /
, but that just gives me a 404 error. Looking at this in curl it does seem to be returning 302:
curl -vvv https://<my-app>.appspot.com/api/santa
< location: https://www.google.com/accounts/ServiceLogin?service=ah&passive=true&continue=https://appengine.google.com/_ah/conflogin%3Fcontinue%3Dhttps://<my-app>.appspot.com/api/santa
< x-cloud-trace-context: 6c130ec3499f2df8b4a48318d3272719;o=1
< date: Wed, 09 May 2018 17:52:41 GMT
< content-type: text/html
< server: Google Frontend
< content-length: 0
But not really sure what mistake I might have made as it doesn't seem like any special settings on the app engine side are required.
I've been trying to create a helper script by using upvote locally as an API, and was able to get to retrieving Blockables: https://gist.github.com/thehesiod/0ffde6e56ea56aede919fe6068a73f89
after fixing this issue: googleapis/google-cloud-datastore#211
The way I have it is by adding google-cloud-sdk/platform/google_appengine
and the root upvote folder to the pythonpath, as well as creating a bunch of __init__.py
files in the upvote source folders.
The issue I ran into is the following: AttributeError: 'BeginTransactionRequest' object has no attribute 'transaction_options'
I believe because by installing the googledatastore
module, there are now two sets of datastore protos: site-packages/google/cloud/datastore_v1/proto/datastore_pb2.py
and site-packages/google/cloud/proto/datastore/v1/datastore_pb2.py
The datastore_v1
one declares the transaction_options
field descriptor, while the v1
does not. It's kinda odd that both are v1 and yet different.
I have the following google modules installed with the latest cloud sdk:
google-api-core 1.3.0
google-api-python-client 1.5.5
google-auth 1.5.0
google-auth-httplib2 0.0.3
google-cloud 0.34.0
google-cloud-bigquery 1.5.0
google-cloud-core 0.28.1
google-cloud-datastore 1.7.0
google-resumable-media 0.3.1
googleapis-common-protos 1.5.3
googledatastore 7.0.1
proto-google-cloud-datastore-v1 0.90.4
proto-google-datastore-v1 1.4.0
I tried substituting one for the other but then got errors about mismatched string types. Ideas?
First thank you. Second very easy to get started. Had an instance up and running in under 30mins, and this is from someone who has very little GCP experience and zero experience with bazel. The only snag I hit was after changing the settings.py file. I would add a note stating that admins can update the instance via:
bazel run upvote/gae:monolith_binary.deploy -- "${PROJ_ID}" app.yaml 2> /dev/null || echo
bazel run upvote/gae:monolith_binary.deploy -- "${PROJ_ID}" app.yaml santa_api.yaml bit9_api.yaml
Not actually sure if the first step was required.
~/GitHub/upvote|master โ ./init_project.sh
zsh: permission denied: ./init_project.sh
~/GitHub/upvote|master โ chmod a+x init_project.sh
I'm wondering if this project is still alive, as there are no recent commits and the whole init process does not seem to work anymore.
Hey!
When I run ./inti_project.sh
, I get an error which says Error: ENOENT: no such file or directory, scandir 'node_modules/.bin'
. Any help is appreciated!
Stacktrace -
./init_project.sh
Initializing Upvote for GCP project: "emerald-caster-246406"
+ gcloud config set project emerald-caster-246406
Updated property [core/project].
+ echo Enabling App Engine...
Enabling App Engine...
++ gcloud app describe '--format=value(id)'
+ [[ emerald-caster-246406 != \e\m\e\r\a\l\d\-\c\a\s\t\e\r\-\2\4\6\4\0\6 ]]
+ echo Enabling APIs used by Upvote...
Enabling APIs used by Upvote...
+ gcloud services enable cloudkms.googleapis.com
+ gcloud services enable bigquery-json.googleapis.com
+ echo Creating encryption keys used to store Upvote API secrets...
Creating encryption keys used to store Upvote API secrets...
+ echo Granting necessary permissions to App Engine...
Granting necessary permissions to App Engine...
++ gcloud iam service-accounts list '--filter=App Engine app default service account' '--format=value(email)'
+ SERVICE_ACCOUNT=emerald-caster-246406@appspot.gserviceaccount.com
+ gcloud projects add-iam-policy-binding emerald-caster-246406 --member serviceAccount:[email protected] --role roles/cloudkms.cryptoKeyEncrypterDecrypter
Updated IAM policy for project [emerald-caster-246406].
bindings:
- members:
- serviceAccount:[email protected]
role: roles/cloudkms.cryptoKeyEncrypterDecrypter
- members:
- serviceAccount:[email protected]
role: roles/compute.serviceAgent
- members:
- serviceAccount:service-79565312020@container-engine-robot.iam.gserviceaccount.com
role: roles/container.serviceAgent
- members:
- serviceAccount:[email protected]
- serviceAccount:[email protected]
- serviceAccount:[email protected]
- serviceAccount:[email protected]
role: roles/editor
- members:
- user:[email protected]
role: roles/owner
etag: BwWUZzgfqLs=
version: 1
+ echo Configuring App Engine...
Configuring App Engine...
+ ./manage_crons.py disable_all
+ echo Deploying temporary default version to App Engine...
Deploying temporary default version to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy -- emerald-caster-246406 app.yaml
+ echo
+ echo Deploying to App Engine...
Deploying to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy --incompatible_depset_union=false -- emerald-caster-246406 app.yaml
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN npm No description
npm WARN npm No repository field.
npm WARN npm No license field.
added 4 packages from 2 contributors and audited 4 packages in 5.274s
found 0 vulnerabilities
yarn install v1.12.1
warning package.json: No license field
warning No license field
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 0.66s.
INFO: Call stack for the definition of repository 'build_bazel_rules_nodejs_npm_install_deps' which is a yarn_install (rule definition at /home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl:245:16):
- /home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl:554:3
- /home/umashankar/upvote/WORKSPACE:493:1
ERROR: An error occurred during the fetch of repository 'build_bazel_rules_nodejs_npm_install_deps':
Traceback (most recent call last):
File "/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 243
_create_build_file(repository_ctx, node)
File "/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 88, in _create_build_file
fail(("node failed: \nSTDOUT:\n%s\nST...)))
node failed:
STDOUT:
STDERR:
fs.js:115
throw err;
^
Error: ENOENT: no such file or directory, scandir 'node_modules/.bin'
at Object.readdirSync (fs.js:783:3)
at listFiles (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:168:8)
at generateRootBuildFile (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:96:20)
at main (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:87:3)
at Object.<anonymous> (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:57:3)
at Module._compile (internal/modules/cjs/loader.js:688:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:699:10)
at Module.load (internal/modules/cjs/loader.js:598:32)
at tryModuleLoad (internal/modules/cjs/loader.js:537:12)
at Function.Module._load (internal/modules/cjs/loader.js:529:3)
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN npm No description
npm WARN npm No repository field.
npm WARN npm No license field.
added 4 packages from 2 contributors and audited 4 packages in 12.159s
found 0 vulnerabilities
DEBUG: /home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl:159:5:
***********WARNING***********
npm: npm_install will require a package_lock_json attribute in future versions
*****************************
ERROR: /home/umashankar/upvote/upvote/gae/modules/upvote_app/frontend/BUILD:70:1: no such package '@npm//ng-html2js/bin': no such package '@build_bazel_rules_nodejs_npm_install_deps//': Traceback (most recent call last):
File "/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 243
_create_build_file(repository_ctx, node)
File "/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 88, in _create_build_file
fail(("node failed: \nSTDOUT:\n%s\nST...)))
node failed:
STDOUT:
STDERR:
fs.js:115
throw err;
^
Error: ENOENT: no such file or directory, scandir 'node_modules/.bin'
at Object.readdirSync (fs.js:783:3)
at listFiles (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:168:8)
at generateRootBuildFile (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:96:20)
at main (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:87:3)
at Object.<anonymous> (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:57:3)
at Module._compile (internal/modules/cjs/loader.js:688:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:699:10)
at Module.load (internal/modules/cjs/loader.js:598:32)
at tryModuleLoad (internal/modules/cjs/loader.js:537:12)
at Function.Module._load (internal/modules/cjs/loader.js:529:3)
and referenced by '//upvote/gae/modules/upvote_app/frontend:admin_templates_genrule'
ERROR: Analysis of target '//upvote/gae:monolith_binary.deploy' failed; build aborted: no such package '@npm//ng-html2js/bin': no such package '@build_bazel_rules_nodejs_npm_install_deps//': Traceback (most recent call last):
File "/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 243
_create_build_file(repository_ctx, node)
File "/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 88, in _create_build_file
fail(("node failed: \nSTDOUT:\n%s\nST...)))
node failed:
STDOUT:
STDERR:
fs.js:115
throw err;
^
Error: ENOENT: no such file or directory, scandir 'node_modules/.bin'
at Object.readdirSync (fs.js:783:3)
at listFiles (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:168:8)
at generateRootBuildFile (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:96:20)
at main (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:87:3)
at Object.<anonymous> (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:57:3)
at Module._compile (internal/modules/cjs/loader.js:688:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:699:10)
at Module.load (internal/modules/cjs/loader.js:598:32)
at tryModuleLoad (internal/modules/cjs/loader.js:537:12)
at Function.Module._load (internal/modules/cjs/loader.js:529:3)
INFO: Elapsed time: 32.583s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (119 packages loaded, 25421 targets configured)
FAILED: Build did NOT complete successfully (119 packages loaded, 25421 targets configured)
Fetching @oauth2client_git; Cloning 97320af2733f7bdbe47f067327610e348f953ae1 of https://github.com/google/oauth2client.git 26s
Fetching @npm; Restarting. 26s
seems like the method should also take host, path, and somehow get the globalregex settings
To reproduce:
Set an email address for FAILSAFE_ADMINISTRATORS
in /upvote/gae/shared/common/settings.py
and saved the file.
Run bazel run upvote/gae:monolith_binary.deploy -- ${PROJ_ID}
which shows no errors.
Manually triggered the cron from the Cloud Console's cron page Which failed.
Opened https://<my-app>.appspot.com/admin/settings
which shows the traceback:
File "/base/alloc/tmpfs/dynamic_runtimes/python27/6baee03590d046f1_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/alloc/tmpfs/dynamic_runtimes/python27/6baee03590d046f1_unzipped/python27_lib/versions/1/google/appengine/api/lib_config.py", line 351, in __getattr__
self._update_configs()
File "/base/alloc/tmpfs/dynamic_runtimes/python27/6baee03590d046f1_unzipped/python27_lib/versions/1/google/appengine/api/lib_config.py", line 287, in _update_configs
self._registry.initialize()
File "/base/alloc/tmpfs/dynamic_runtimes/python27/6baee03590d046f1_unzipped/python27_lib/versions/1/google/appengine/api/lib_config.py", line 160, in initialize
import_func(self._modname)
File "/base/data/home/apps/d~its-upvote/auto.408675188517060759/appengine_config.py", line 10, in <module>
import upvote.gae.real_appengine_config
File "/base/data/home/apps/d~its-upvote/auto.408675188517060759/upvote/gae/real_appengine_config.py", line 20, in <module>
from upvote.gae import appengine_config_utils
File "/base/data/home/apps/d~its-upvote/auto.408675188517060759/upvote/gae/appengine_config_utils.py", line 20, in <module>
from upvote.gae.datastore import test_utils
File "/base/data/home/apps/d~its-upvote/auto.408675188517060759/upvote/gae/datastore/test_utils.py", line 26, in <module>
from upvote.gae.datastore.models import base
File "/base/data/home/apps/d~its-upvote/auto.408675188517060759/upvote/gae/datastore/models/base.py", line 29, in <module>
from upvote.gae.datastore.models import bigquery
File "/base/data/home/apps/d~its-upvote/auto.408675188517060759/upvote/gae/datastore/models/bigquery.py", line 22, in <module>
from upvote.gae.shared.common import settings
File "/base/data/home/apps/d~its-upvote/auto.408675188517060759/upvote/gae/shared/common/settings.py", line 72
FAILSAFE_ADMINISTRATORS = [[email protected]]
^
SyntaxError: invalid syntax
FAILSAFE_ADMINISTRATORS
blank and rerunning steps 2, 3, and 4 result in successful access to the admin site.Here's how I did it: farmersbusinessnetwork@4c87e9b
Traceback (most recent call last):
File "/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/base/data/home/apps/m~santaupvote/santa-api:auto.411437352926953149/upvote/gae/modules/santa_api/sync.py", line 110, in dispatch
self.abort(httplib.FORBIDDEN, explanation='XSRF token missing/invalid.')
File "/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 573, in abort
abort(code, *args, **kwargs)
File "/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1804, in abort
raise cls(*args, **kwargs)
HTTPForbidden: XSRF token missing/invalid.
I'm still debugging how deduping works, however what I do know is that if a machine is in monitor mode, executes something that is not allowed, an ALLOW_UNKNOWN event is recorded to the datastore.
If now you move the machine to lockdown, attempt to execute the same executable, you'll get the santa warning, and eventually the event will get synced, and the recorded and last_blocked will get updated, however the event_type will stay at ALLOW_UNKNOWN.
So if you go to the events page it will list the executable as ALLOW_UNKNOWN, when in reality it was blocked.
I followed the installation instructions but I'm getting the following errors and the install just hangs until I quit:
$ bazel run upvote/gae:monolith_binary.deploy -- "${PROJ_ID}" app.yaml
WARNING: /private/var/tmp/_bazel_seadmin/dc15004ff28b5d5c51a9b82d37540f68/external/com_google_protobuf_java/WORKSPACE:1: Workspace name in /private/var/tmp/_bazel_seadmin/dc15004ff28b5d5c51a9b82d37540f68/external/com_google_protobuf_java/WORKSPACE (@com_google_protobuf) does not match the name given in the repository's definition (@com_google_protobuf_java); this will cause a build error in future versions
INFO: Analysed target //upvote/gae:monolith_binary.deploy (1 packages loaded).
INFO: Found 1 target...
Target //upvote/gae:monolith_binary.deploy up-to-date:
bazel-bin/upvote/gae/monolith_binary_deploy.sh
bazel-bin/upvote/gae/monolith_binary.deploy
INFO: Elapsed time: 3.435s, Critical Path: 0.07s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/upvote/gae/monolith_binary.deploy upvote-205913 app.yaml
04:25 PM Application: upvote-205913 (was: auto); version: auto
04:25 PM Host: appengine.google.com
04:25 PM Starting update of app: upvote-205913, version: auto
04:25 PM Getting current resource limits.
0:370: execution error: An error of type -610 has occurred. (-610)
70:78: execution error: An error of type -610 has occurred. (-610)
69:77: execution error: An error of type -610 has occurred. (-610)
^C
Bazel caught interrupt signal; shutting down.
Here are my Basel and gcloud versions:
$ bazel version
Build label: 0.13.1-homebrew
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed May 23 16:57:59 2018 (1527094679)
Build timestamp: 1527094679
Build timestamp as int: 1527094679
$ gcloud version
Google Cloud SDK 203.0.0
bq 2.0.34
core 2018.05.25
gsutil 4.31
Any ideas what's wrong?
Environment (Inside Docker)
OS: Ubuntu 16.04
Bazel: 0.23.2
Python: 2.7.12
Problem
Trying to deploy using init_project.sh
and the following errors occur.
ERROR: /root/.cache/bazel/_bazel_root/2883744d33ed0db99e71fd4612c17bed/external/com_google_closure_stylesheets/BUILD:7:1: no such package '@com_google_javascript_closure_compiler//': java.io.IOException: Error downloading [https://mirror.bazel.build/repo1.maven.org/maven2/com/google/javascript/closure-compiler-unshaded/v20180805/closure-compiler-unshaded-v20180805.jar, http://repo1.maven.org/maven2/com/google/javascript/closure-compiler-unshaded/v20180805/closure-compiler-unshaded-v20180805.jar] to /root/.cache/bazel/_bazel_root/2883744d33ed0db99e71fd4612c17bed/external/com_google_javascript_closure_compiler/closure-compiler-unshaded-v20180805.jar: All mirrors are down: [GET returned 404 Not Found, GET returned 501 HTTPS Required] and referenced by '@com_google_closure_stylesheets//:com_google_closure_stylesheets' ERROR: Analysis of target '//upvote/gae:monolith_binary.deploy' failed; build aborted: no such package '@com_google_javascript_closure_compiler//': java.io.IOException: Error downloading [https://mirror.bazel.build/repo1.maven.org/maven2/com/google/javascript/closure-compiler-unshaded/v20180805/closure-compiler-unshaded-v20180805.jar, http://repo1.maven.org/maven2/com/google/javascript/closure-compiler-unshaded/v20180805/closure-compiler-unshaded-v20180805.jar] to /root/.cache/bazel/_bazel_root/2883744d33ed0db99e71fd4612c17bed/external/com_google_javascript_closure_compiler/closure-compiler-unshaded-v20180805.jar: All mirrors are down: [GET returned 404 Not Found, GET returned 501 HTTPS Required] INFO: Elapsed time: 16.994s INFO: 0 processes. FAILED: Build did NOT complete successfully (18 packages loaded, 849 targets configured) FAILED: Build did NOT complete successfully (18 packages loaded, 849 targets configured) Fetching @chardet_git; Cloning tags/3.0.2 of https://github.com/chardet/chardet.git 4s Fetching @com_google_javascript_closure_compiler; fetching
I'm getting the following:
amohr@amohr-macbook ~/dev/upvote (master) $ gcloud config set project santaupvote
Updated property [core/project].
amohr@amohr-macbook ~/dev/upvote (master) $ bazel run --incompatible_package_name_is_a_function=false upvote/gae:monolith_binary.deploy -- -q app.yaml
INFO: Invocation ID: 6e604bd2-c82b-4660-a20e-c6fd0afc7209
INFO: Analysed target //upvote/gae:monolith_binary.deploy (1 packages loaded, 23021 targets configured).
INFO: Found 1 target...
Target //upvote/gae:monolith_binary.deploy up-to-date:
bazel-bin/upvote/gae/monolith_binary_deploy.sh
bazel-bin/upvote/gae/monolith_binary.deploy
INFO: Elapsed time: 2.709s, Critical Path: 0.11s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
WARNING: In file [/private/var/folders/9z/50thrw1j52g7v2d7yrl87l940000gn/T/war.fWFCAUlo/__main__/app.yaml]: ('module', 'Field module is deprecated; use service instead.')
WARNING: In file [/private/var/folders/9z/50thrw1j52g7v2d7yrl87l940000gn/T/war.fWFCAUlo/__main__/app.yaml]: ('module', 'Field module is deprecated; use service instead.')
ERROR: (gcloud.app.deploy) [/private/var/folders/9z/50thrw1j52g7v2d7yrl87l940000gn/T/war.fWFCAUlo/__main__/app.yaml] and [/private/var/folders/9z/50thrw1j52g7v2d7yrl87l940000gn/T/war.fWFCAUlo/__main__/app.yaml] are both defining the service id [default]. All <service>.yaml files must have unique service ids.
amohr@amohr-macbook ~/dev/upvote (master) $ bazel version
INFO: Invocation ID: 50765701-a2c8-4260-a150-6541638ba794
Build label: 0.21.0
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Dec 19 12:57:09 2018 (1545224229)
Build timestamp: 1545224229
Build timestamp as int: 1545224229
ideas?
The what happens next section references the google specific gmenu tool to force a sync with rules.
this key is sent via santa, but is not stored to the events datastore table...this is useful when things are marked as run as root, so we can see if it was called via sudo or not.
when we set a host to upload logs, we're getting 404 on GET [PROJ].appspot.com/api/santa/logupload/[HOSTID]
We're getting a LOT of 500s from the santa-api with errors like these in the logs:
The request failed because the instance could not start successfully
Threads started by this request continued executing past the hard deadline.
any ideas?
so per the santa docs, the way one is supposed to lock down santad<->upvote communication is via server based client certificate auth. Unfortunately, after looking at ValidateClient
, it seems some sort of header token auth is instead expected by upvote.
I looked a little bit into implementing client certificate auth however it seems webapp2 does not support this, nor does the GAE environment in general. What method of santa<->upvote auth do the authors of upvote expect end-users to implement to be secure?
I'm having issues following the steps to deploy
Deploying to App Engine...
Deploying to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy -- rv3-upvote app.yaml santa_api.yaml
INFO: Invocation ID: 2f292a57-d043-40cf-9751-1c9266c84dcb
Loading:
Loading: 0 packages loaded
ERROR: error loading package '': Encountered error while reading extension file 'appengine/py_appengine.bzl': no such package '@io_bazel_rules_appengine//appengine': The native git_repository rule is deprecated. load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository") for a replacement.
Use --incompatible_remove_native_git_repository=false to temporarily continue using the native rule.
ERROR: error loading package '': Encountered error while reading extension file 'appengine/py_appengine.bzl': no such package '@io_bazel_rules_appengine//appengine': The native git_repository rule is deprecated. load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository") for a replacement.
Use --incompatible_remove_native_git_repository=false to temporarily continue using the native rule.
INFO: Elapsed time: 0.186s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully (0 packages loaded)
Any ideas?
Thanks!
after much effort, in farmersbusinessnetwork@8537d00 I added stackdriver monitoring support (commented out code). Unfortunately I quickly found that you can only do a timeseries event once per minute which means I lose almost all data, and get a ton of errors.
In out AWS environment we use datadog and there you have an agent which runs all the time which does the event sampling.
I'm searched and I can't find any equivalent for using stackdriver monitoring in app-engine. Obviously something is working behind the scenes because when I enabled stackdriver monitoring for the project I get a bunch of built-in metrics...however I see nothing I can use for custom metrics.
I thought of using deferred tasks however the class/method, and arguments get pickled so that won't help.
If there's a solution to this perhaps this project can come with something like what I've done by default.
Thoughts?
Have started seeing more of these:
InternalTransientError: Temporary error in fetching URL: https://www.googleapis.com/bigquery/v2/projects/[PROJ]/datasets/gae_streaming/tables/Host/insertAll, please re-try at _get_fetch_result (/base/alloc/tmpfs/dynamic_runtimes/python27g/7e468a4e2dbc991a/python27/python27_lib/versions/1/google/appengine/api/urlfetch.py:446)
as well:
TimeoutError: (<requests.packages.urllib3.contrib.appengine.AppEngineManager object at 0x2a583ad9b4d0>, DeadlineExceededError('Deadline exceeded while waiting for HTTP response from URL: https://www.googleapis.com/bigquery/v2/projects/[PROJ]/datasets/gae_streaming/tables/Host/insertAll',))
and
ConnectionError: Connection closed unexpectedly by server at URL: https://www.googleapis.com/bigquery/v2/projects/[PROJ]/datasets/gae_streaming/tables/Host/insertAll
sounds like there's some missing retries. Does this mean that our bigquery tables will be missing entries? If these are retried then something should be changed for them not the show up in stackdriver error reporting.
I'm having issues following the steps to deploy
+ echo Deploying to App Engine...
Deploying to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy -- rv3-upvote app.yaml santa_api.yaml
INFO: Invocation ID: bbd6d94c-828d-4fa4-97e6-80cad0f54c3c
Loading:
Loading: 0 packages loaded
ERROR: E:/server/upvote/upvote/gae/BUILD:59:1: Traceback (most recent call last):
File "E:/server/upvote/upvote/gae/BUILD", line 59
upvote_appengine_test(name = "appengine_config_test", si...", <2 more arguments>)
File "E:/server/upvote/upvote/builddefs.bzl", line 8, in upvote_appengine_test
py_appengine_test(name = name, srcs = srcs, deps = dep..., <3 more arguments>)
unexpected keyword 'overwrite_appengine_config' in call to py_appengine_test(name, srcs, deps = [], data = [], libraries = {}, size = None)
ERROR: error loading package 'upvote/gae': Package 'upvote/gae' contains errors
INFO: Elapsed time: 0.402s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully (0 packages loaded)
I have tried overwrite_appengine_config = false,
on upvote\gae\BUILD
file but that didnt work.
And the build is still failing after the last update.
Any ideas?
Thanks!
as per inclusive code guidelines :)
$ python3 -m flake8 . --count --select=E9,F821,F822,F823 --show-source --statistics
./upvote/shared/constants_test.py:131:14: F821 undefined name 'xrange'
for i in xrange(1, len(hierarchy)):
^
./upvote/shared/time_utils_test.py:31:57: F821 undefined name 'xrange'
datetime.datetime(2017, 1, 1, 1, 1, s) for s in xrange(3)]
^
./upvote/gae/shared/common/test_utils.py:35:56: F821 undefined name 'xrange'
return [now + datetime.timedelta(seconds=i) for i in xrange(count)]
^
./upvote/gae/shared/common/taskqueue_utils_test.py:32:14: F821 undefined name 'xrange'
for _ in xrange(expected_size):
^
./upvote/gae/shared/common/taskqueue_utils_test.py:46:61: F821 undefined name 'xrange'
taskqueue_utils.CappedDefer(dir, max_size) for _ in xrange(total_size)]
^
./upvote/gae/shared/common/taskqueue_utils_test.py:65:16: F821 undefined name 'xrange'
for i in xrange(6):
^
./upvote/gae/shared/common/taskqueue_utils_test.py:81:16: F821 undefined name 'xrange'
for i in xrange(6):
^
./upvote/gae/shared/common/taskqueue_utils_test.py:101:16: F821 undefined name 'xrange'
for i in xrange(6):
^
./upvote/gae/shared/common/taskqueue_utils_test.py:130:16: F821 undefined name 'xrange'
for _ in xrange(6):
^
./upvote/gae/shared/common/taskqueue_utils_test.py:157:16: F821 undefined name 'xrange'
for _ in xrange(6):
^
./upvote/gae/shared/common/taskqueue_utils_test.py:212:14: F821 undefined name 'xrange'
for _ in xrange(10):
^
./upvote/gae/shared/common/query_utils_test.py:43:43: F821 undefined name 'xrange'
return [CreateEntity(**kwargs) for _ in xrange(count)]
^
./upvote/gae/shared/common/query_utils_test.py:133:25: F821 undefined name 'xrange'
for entity_count in xrange(50):
^
./upvote/gae/shared/common/query_utils_test.py:155:14: F821 undefined name 'xrange'
for _ in xrange(4):
^
./upvote/gae/shared/common/query_utils_test.py:169:14: F821 undefined name 'xrange'
for _ in xrange(2):
^
./upvote/gae/shared/common/query_utils_test.py:182:14: F821 undefined name 'xrange'
for _ in xrange(4):
^
./upvote/gae/shared/common/query_utils_test.py:194:14: F821 undefined name 'xrange'
for _ in xrange(2):
^
./upvote/gae/shared/common/query_utils_test.py:208:14: F821 undefined name 'xrange'
for _ in xrange(2):
^
./upvote/gae/shared/common/query_utils_test.py:222:14: F821 undefined name 'xrange'
for _ in xrange(3):
^
./upvote/gae/shared/common/query_utils_test.py:238:14: F821 undefined name 'xrange'
for _ in xrange(2):
^
./upvote/gae/shared/common/intermodule_test.py:90:55: F821 undefined name 'xrange'
redirect_urls = ['https://foo%d.com' % i for i in xrange(4)]
^
./upvote/gae/shared/common/query_utils.py:96:14: F821 undefined name 'reduce'
result = reduce( # pylint: disable=bad-builtin
^
./upvote/gae/shared/common/query_utils.py:102:12: F821 undefined name 'reduce'
result = reduce( # pylint: disable=bad-builtin
^
./upvote/gae/datastore/test_utils.py:39:49: F821 undefined name 'xrange'
return ''.join(random.choice(source) for _ in xrange(length))
^
./upvote/gae/datastore/test_utils.py:68:37: F821 undefined name 'xrange'
return [RandomLetters(8) for _ in xrange(size)]
^
./upvote/gae/datastore/test_utils.py:72:32: F821 undefined name 'xrange'
return [RandomInt() for _ in xrange(size)]
^
./upvote/gae/datastore/test_utils.py:84:34: F821 undefined name 'xrange'
return [RandomEmail() for _ in xrange(count)]
^
./upvote/gae/datastore/test_utils.py:127:69: F821 undefined name 'xrange'
random_value = [random.choice(list(prop._choices)) for _ in xrange(3)]
^
./upvote/gae/datastore/test_utils.py:131:47: F821 undefined name 'xrange'
random_value = [value_func() for _ in xrange(3)]
^
./upvote/gae/datastore/test_utils.py:143:63: F821 undefined name 'xrange'
return [RandomDatastoreEntity(model_cls, **kwargs) for _ in xrange(count)]
^
./upvote/gae/datastore/test_utils.py:228:52: F821 undefined name 'xrange'
return [CreateBit9Certificate(**kwargs) for _ in xrange(count)]
^
./upvote/gae/datastore/test_utils.py:247:55: F821 undefined name 'xrange'
'bundle_id': '.'.join(RandomLetters(3) for _ in xrange(3)),
^
./upvote/gae/datastore/test_utils.py:248:52: F821 undefined name 'xrange'
'version': '.'.join(RandomDigits(1) for _ in xrange(3)),
^
./upvote/gae/datastore/test_utils.py:249:58: F821 undefined name 'xrange'
'short_version': '.'.join(RandomDigits(1) for _ in xrange(3)),
^
./upvote/gae/datastore/test_utils.py:278:47: F821 undefined name 'xrange'
return [CreateBit9Binary(**kwargs) for _ in xrange(count)]
^
./upvote/gae/datastore/test_utils.py:283:51: F821 undefined name 'xrange'
return [CreateSantaBlockable(**kwargs) for _ in xrange(count)]
^
./upvote/gae/datastore/test_utils.py:324:16: F821 undefined name 'xrange'
for _ in xrange(count)]
^
./upvote/gae/datastore/test_utils.py:387:43: F821 undefined name 'xrange'
return [CreateEvent(blockable) for _ in xrange(event_count)]
^
./upvote/gae/datastore/test_utils.py:391:48: F821 undefined name 'xrange'
return [CreateSantaEvent(blockable) for _ in xrange(event_count)]
^
./upvote/gae/datastore/test_utils.py:395:47: F821 undefined name 'xrange'
return [CreateBit9Event(blockable) for _ in xrange(event_count)]
^
./upvote/gae/datastore/test_utils.py:431:41: F821 undefined name 'xrange'
return [CreateUser(**kwargs) for _ in xrange(user_count)]
^
./upvote/gae/datastore/test_utils.py:486:46: F821 undefined name 'xrange'
return [CreateSantaHost(**kwargs) for _ in xrange(count)]
^
./upvote/gae/datastore/test_utils.py:491:45: F821 undefined name 'xrange'
return [CreateBit9Host(**kwargs) for _ in xrange(count)]
^
./upvote/gae/datastore/test_utils.py:567:60: F821 undefined name 'xrange'
return [CreateBit9Rule(blockable_key, **kwargs) for _ in xrange(count)]
^
./upvote/gae/datastore/utils_test.py:550:38: F821 undefined name 'xrange'
futures = [ndb.Future() for _ in xrange(3)]
^
./upvote/gae/datastore/utils_test.py:599:38: F821 undefined name 'xrange'
futures = [ndb.Future() for _ in xrange(3)]
^
./upvote/gae/modules/upvote_app/lib/voting_test.py:286:14: F821 undefined name 'xrange'
for _ in xrange(10):
^
./upvote/gae/modules/upvote_app/lib/voting_test.py:573:42: F821 undefined name 'xrange'
test_utils.CreateUser() for _ in xrange(self.local_threshold - 1)]
^
./upvote/gae/modules/upvote_app/lib/voting_test.py:603:42: F821 undefined name 'xrange'
test_utils.CreateUser() for _ in xrange(self.local_threshold)]
^
./upvote/gae/modules/upvote_app/lib/voting_test.py:856:14: F821 undefined name 'xrange'
for _ in xrange(4):
^
./upvote/gae/modules/upvote_app/api/handlers/blockables_test.py:495:14: F821 undefined name 'xrange'
for i in xrange(expected):
^
./upvote/gae/modules/upvote_app/api/handlers/constants_test.py:37:25: F821 undefined name 'unicode'
unicode_elems = map(unicode, common_constants.USER_ROLE.SET_ALL)
^
./upvote/gae/modules/upvote_app/cron/monitoring.py:21:71: F821 undefined name 'long'
rows_to_persist = monitoring.Metric(metrics.BIGQUERY.ROWS_TO_PERSIST, long)
^
./upvote/gae/modules/upvote_app/cron/monitoring.py:22:69: F821 undefined name 'long'
rows_to_stream = monitoring.Metric(metrics.BIGQUERY.ROWS_TO_STREAM, long)
^
./upvote/gae/modules/upvote_app/cron/export_test.py:63:14: F821 undefined name 'xrange'
for i in xrange(entity_count):
^
./upvote/gae/modules/upvote_app/cron/export_test.py:91:14: F821 undefined name 'xrange'
for i in xrange(entity_count):
^
./upvote/gae/modules/upvote_app/cron/export_test.py:242:14: F821 undefined name 'xrange'
for i in xrange(count):
^
./upvote/gae/modules/upvote_app/cron/roles_test.py:291:42: F821 undefined name 'xrange'
test_utils.CreateUser() for _ in xrange(roles.BATCH_SIZE - 1)]
^
./upvote/gae/modules/upvote_app/cron/roles_test.py:315:42: F821 undefined name 'xrange'
test_utils.CreateUser() for _ in xrange(roles.BATCH_SIZE + 1)]
^
./upvote/gae/modules/santa_api/sync_test.py:1297:47: E999 SyntaxError: invalid token
rule_sync_dt=datetime.datetime(2001, 01, 01, 0, 0, 0))
^
./upvote/gae/modules/bit9_api/change_set.py:56:16: F821 undefined name 'xrange'
for _ in xrange(1 + max_retries):
^
./upvote/gae/modules/bit9_api/monitoring.py:21:69: F821 undefined name 'long'
events_to_pull = monitoring.Metric(metrics.BIT9_API.EVENTS_TO_PULL, long)
^
./upvote/gae/modules/bit9_api/monitoring.py:24:41: F821 undefined name 'long'
metrics.BIT9_API.EVENTS_TO_PROCESS, long)
^
./upvote/gae/modules/bit9_api/monitoring.py:26:71: F821 undefined name 'long'
pending_changes = monitoring.Metric(metrics.BIT9_API.PENDING_CHANGES, long)
^
./upvote/gae/modules/bit9_api/sync_test.py:105:24: F821 undefined name 'xrange'
for host_id in xrange(host_count)]
^
./upvote/gae/modules/bit9_api/sync_test.py:109:16: F821 undefined name 'xrange'
for _ in xrange(events_per_host):
^
./upvote/gae/modules/bit9_api/sync_test.py:172:22: F821 undefined name 'xrange'
for batch_num in xrange(batch_count):
^
./upvote/gae/modules/bit9_api/sync_test.py:175:28: F821 undefined name 'xrange'
for event_num in xrange(events_per_batch)])
^
./upvote/gae/modules/bit9_api/sync_test.py:363:18: F821 undefined name 'xrange'
for _ in xrange(4)]
^
./upvote/gae/modules/bit9_api/sync_test.py:376:18: F821 undefined name 'xrange'
for _ in xrange(4)]
^
./upvote/gae/modules/bit9_api/cron_test.py:97:25: F821 undefined name 'obj'
requests.append(obj.to_raw_dict())
^
./upvote/gae/modules/bit9_api/cron_test.py:188:14: F821 undefined name 'xrange'
for i in xrange(1, cron._PULL_MAX_QUEUE_SIZE + 20):
^
./upvote/gae/modules/bit9_api/cron_test.py:204:14: F821 undefined name 'xrange'
for _ in xrange(expected_length):
^
./upvote/gae/modules/bit9_api/cron_test.py:221:14: F821 undefined name 'xrange'
for i in xrange(1, cron._DISPATCH_MAX_QUEUE_SIZE + 20):
^
./upvote/gae/modules/bit9_api/api/model.py:117:18: F821 undefined name 'basestring'
_PYTHON_TYPE = basestring
^
./upvote/gae/modules/bit9_api/api/utils.py:31:34: F821 undefined name 'xrange'
_DELETE_CHARS = ''.join(map(chr, xrange(128, 256)))
^
./upvote/gae/modules/bit9_api/api/utils.py:35:51: F821 undefined name 'unicode'
return to_ascii_str(value) if isinstance(value, unicode) else value
^
./upvote/gae/modules/bit9_api/api/utils.py:56:24: F821 undefined name 'unicode'
if not isinstance(s, unicode) and not isinstance(s, str):
^
./common/datastore_locks.py:201:46: F821 undefined name 'xrange'
intervals = [_INITIAL_DELAY + i for i in xrange(max_acquire_attempts - 1)]
^
./common/ng_template.py:47:12: E999 SyntaxError: invalid syntax
print '\n'.join(result)
^
./common/cloud_kms/kms_ndb_test.py:32:50: F821 undefined name 'unicode'
side_effect=lambda data, a2, a3, **kwargs: unicode(data))
^
2 E999 SyntaxError: invalid syntax
79 F821 undefined name 'unicode'
81
I'm starting to roll out santa/upvote to a number of machines and trying to figure out the best way to manage the initial creation of the whitelist (presumably easier going forward as we won't expect much new stuff to show up)
It would be nice if there was a "publishers" view on the upvote dashboard to make it easier to go through and mass whitelist certificates at the global level, perhaps sorted by how many apps fall under each certificate
Calling preflight gives a 500 error.
Backtrace:
ImportError: No module named pkg_resources
at (/base/data/home/apps/eupvote-xxx/santa-api:auto.411705267076089495/external/gcloud_bigquery_archive/google/cloud/bigquery/init.py:31)upvote-xxx/santa-api:auto.411705267076089495/upvote/gae/bigquery/tables.py:25)
at (/base/data/home/apps/e
at (/base/data/home/apps/eupvote-xxx/santa-api:auto.411705267076089495/upvote/gae/datastore/models/santa.py:21)upvote-xxx/santa-api:auto.411705267076089495/upvote/gae/datastore/models/rule.py:17)
at (/base/data/home/apps/e
at (/base/data/home/apps/eupvote-xxx/santa-api:auto.411705267076089495/upvote/gae/real_appengine_config.py:20)upvote-xxx/santa-api:auto.411705267076089495/appengine_config.py:10)
at (/base/data/home/apps/e
at import_module (/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_dist/lib/python2.7/importlib/init.py:37)
at initialize (/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_lib/versions/1/google/appengine/api/lib_config.py:165)
at _update_configs (/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_lib/versions/1/google/appengine/api/lib_config.py:294)
at getattr (/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_lib/versions/1/google/appengine/api/lib_config.py:358)
at Handle (/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py:240)
Howdy!
I'm seeing an increase in DeadlineExceededError errors in Stackdriver.
TimeoutError:` (<requests.packages.urllib3.contrib.appengine.AppEngineManager object at 0x3c101e4444>, DeadlineExceededError('Deadline exceeded while waiting for HTTP response from URL: https://www.googleapis.com/bigquery/v2/projects/sync-server/datasets/gae_streaming/tables/Execution/insertAll',))
Any guidance on debugging approaches or tweaks in code to make to limit this error?
I think someone forgot to check in this file?
After merging I'm getting:
+ bazel run upvote/gae:monolith_binary.deploy -- --project santaupvote --version auto app.yaml santa_api.yaml
INFO: Build options have changed, discarding analysis cache.
ERROR: /Users/amohr/dev/upvote/upvote/gae/modules/upvote_app/frontend/BUILD:77:1: no such package '@npm//ng-html2js/bin': no such package '@nodejs//': Traceback (most recent call last):
File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 418
_prepare_node(repository_ctx)
File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 325, in _prepare_node
repository_ctx.file("bin/npm_node_repositories", conte...])), ...)
File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 329, in repository_ctx.file
"".join(["\necho Running npm \"$@\" in {...])
File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 329, in "".join
"\necho Running npm \"$@\" in {root}\n(cd \"{root}\"; \"$SCRIPT_DIR/{node}\" \"$SCRIPT_DIR/{script}\" --scripts-prepend-node-path=false \"$@\")\n".format(root = repository_ctx.path(packa..., <2 more arguments>)
File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 333, in "\necho Running npm \"$@\" in {root}\n
repository_ctx.path(package_json)
Unable to load package for //:package.json: not found. and referenced by '//upvote/gae/modules/upvote_app/frontend:ui_templates_genrule'
ERROR: Analysis of target '//upvote/gae:monolith_binary.deploy' failed; build aborted: no such package '@npm//ng-html2js/bin': no such package '@nodejs//': Traceback (most recent call last):
File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 418
_prepare_node(repository_ctx)
File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 325, in _prepare_node
repository_ctx.file("bin/npm_node_repositories", conte...])), ...)
File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 329, in repository_ctx.file
"".join(["\necho Running npm \"$@\" in {...])
File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 329, in "".join
"\necho Running npm \"$@\" in {root}\n(cd \"{root}\"; \"$SCRIPT_DIR/{node}\" \"$SCRIPT_DIR/{script}\" --scripts-prepend-node-path=false \"$@\")\n".format(root = repository_ctx.path(packa..., <2 more arguments>)
File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 333, in "\necho Running npm \"$@\" in {root}\n
repository_ctx.path(package_json)
I'm attempting to understand how I can actually create/manage a compiler rule with Upvote. The use-case being, we would enable transitive whitelisting for certain folks who are using certain tools.
I see the codebase fully supports enabling transitive whitelisting mode on clients (and it works). However I see limited references to compiler rules, the main one being WHITELIST_COMPILER
as part of a list of Santa rule policies.
Even if there is no UI support, is this something I can manually add to the Datastore Rules table, so it would get synced to clients?
`Deploying to App Engine...
Any help is appreciated. Thanks.
Didn't run into access issues....not sure what this is
I'm having issues following the steps to deploy
Deploying to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy -- upvote-poc-1 app.yaml santa_api.yaml bit9_api.yaml --verbose_failures
INFO: Analysed target //upvote/gae:monolith_binary.deploy (0 packages loaded).
INFO: Found 1 target...
ERROR: missing input file '@local_jdk//:jre/lib/rt.jar'
ERROR: /private/var/tmp/_bazel_eric/7b851cf8ed89b6be50e03e0a8350cb67/external/io_bazel_rules_closure/java/io/bazel/rules/closure/BUILD:17:1: @io_bazel_rules_closure//java/io/bazel/rules/closure:ClosureWorker: missing input file '@local_jdk//:jre/lib/rt.jar'
Target //upvote/gae:monolith_binary.deploy failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /private/var/tmp/_bazel_eric/7b851cf8ed89b6be50e03e0a8350cb67/external/io_bazel_rules_closure/java/io/bazel/rules/closure/BUILD:17:1 1 input file(s) do not exist
INFO: Elapsed time: 2.483s, Critical Path: 0.01s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
Any ideas?
Thanks!
Please check the logs -
+ echo Configuring App Engine...
Configuring App Engine...
+ echo Deploying temporary default version to App Engine...
Deploying temporary default version to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy -- upvote-project app.yaml
added 4 packages from 2 contributors and audited 4 packages in 3.854s
found 0 vulnerabilities
yarn install v1.12.1
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 0.48s.
added 4 packages from 2 contributors and audited 4 packages in 1.513s
found 0 vulnerabilities
added 4 packages from 2 contributors and audited 4 packages in 1.441s
found 0 vulnerabilities
external/com_google_protobuf/src/google/protobuf/generated_message_reflection.cc(2424): warning C4506: no definition for inline function 'google::protobuf::Message *google::protobuf::internal::GenericTypeHandler<google::protobuf::Message>::NewFromPrototype(const GenericType *,google::protobuf::Arena *)'
with
[
GenericType=google::protobuf::Message
]
external/com_google_protobuf/src/google/protobuf/generated_message_reflection.cc(2424): warning C4506: no definition for inline function 'google::protobuf::Arena *google::protobuf::internal::GenericTypeHandler<google::protobuf::Message>::GetArena(GenericType *)'
with
[
GenericType=google::protobuf::Message
]
external/com_google_protobuf/src/google/protobuf/generated_message_reflection.cc(2424): warning C4506: no definition for inline function 'void *google::protobuf::internal::GenericTypeHandler<google::protobuf::Message>::GetMaybeArenaPointer(GenericType *)'
with
[
GenericType=google::protobuf::Message
]
external/com_google_protobuf/src/google/protobuf/map_field.cc(465): warning C4506: no definition for inline function 'google::protobuf::Message *google::protobuf::internal::GenericTypeHandler<google::protobuf::Message>::NewFromPrototype(const GenericType *,google::protobuf::Arena *)'
with
[
GenericType=google::protobuf::Message
]
external/com_google_protobuf/src/google/protobuf/map_field.cc(465): warning C4506: no definition for inline function 'google::protobuf::Arena *google::protobuf::internal::GenericTypeHandler<google::protobuf::Message>::GetArena(GenericType *)'
with
[
GenericType=google::protobuf::Message
]
external/com_google_protobuf/src/google/protobuf/map_field.cc(465): warning C4506: no definition for inline function 'void *google::protobuf::internal::GenericTypeHandler<google::protobuf::Message>::GetMaybeArenaPointer(GenericType *)'
with
[
GenericType=google::protobuf::Message
]
+ echo
+ echo Deploying to App Engine...
Deploying to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy -- upvote-project app.yaml santa_api.yaml
INFO: Invocation ID: bb4c697d-d1cf-48c9-9b5f-9b1cbf5c0006
Loading:
Loading: 0 packages loaded
DEBUG: Rule 'io_bazel_rules_appengine' modified arguments {"shallow_since": "1544654581 -0500"}
DEBUG: Rule 'build_bazel_rules_nodejs' modified arguments {"commit": "6d9a5c9333e5bca58125e9c96b99f3c274bf51e2", "shallow_since": "1543957109 -0800"} and dropped ["tag"]
Analyzing: target //upvote/gae:monolith_binary.deploy (0 packages loaded, 0 targets configured)
DEBUG: Rule 'gcloud_auth_git' modified arguments {"commit": "709953d34c0c93744606a38e84abce557e73dfcb", "shallow_since": "1492459993 -0700"} and dropped ["tag"]
DEBUG: Rule 'pyasn1_git' modified arguments {"shallow_since": "1444500249 +0000"}
DEBUG: Rule 'pyasn1_modules_git' modified arguments {"commit": "75a1d105027e86aa7d2720e6f0fa6349933ad55d", "shallow_since": "1501107647 +0200"} and dropped ["tag"]
DEBUG: Rule 'rsa_git' modified arguments {"shallow_since": "1497086972 +0200"}
DEBUG: Rule 'cachetools_git' modified arguments {"commit": "e27332bc82f4e327aedaec17c9b656ae719322ed", "shallow_since": "1475496371 +0200"} and dropped ["tag"]
DEBUG: Rule 'gcloud_auth_httplib2_git' modified arguments {"shallow_since": "1497375871 -0700"}
DEBUG: Rule 'gcloud_resumable_media_git' modified arguments {"commit": "d296d036453539f03b956f33112baa740d5a723f", "shallow_since": "1500660227 -0700"} and dropped ["tag"]
DEBUG: Rule 'requests_git' modified arguments {"commit": "5091c15bd71ec934423a71ce1b53668cd15db68f", "shallow_since": "1496094855 -0400"} and dropped ["tag"]
DEBUG: Rule 'chardet_git' modified arguments {"commit": "c47f6d79845ec7fe9f3bfd2fc66d47812f571d12", "shallow_since": "1492024800 -0400"} and dropped ["tag"]
DEBUG: Rule 'idna_git' modified arguments {"commit": "0088bfce9c3270e15d8356f0709110c1a9589d35", "shallow_since": "1488857158 -0800"} and dropped ["tag"]
DEBUG: Rule 'urllib3_git' modified arguments {"commit": "8e049799777cdf592ae54ae762d140151766123c", "shallow_since": "1500541470 +0100"} and dropped ["tag"]
DEBUG: Rule 'certifi_git' modified arguments {"commit": "4f35e3529c78ced74040cf5d80bf8ec4aac9a190", "shallow_since": "1492457773 -0400"} and dropped ["tag"]
DEBUG: Rule 'googleapiclient_git' modified arguments {"commit": "f19307d2079e3c3854cac887f059655a66c49896", "shallow_since": "1478045551 -0700"} and dropped ["tag"]
DEBUG: Rule 'oauth2client_git' modified arguments {"shallow_since": "1496765066 -0700"}
DEBUG: Rule 'absl_git' modified arguments {"shallow_since": "1506641175 -0700"}
DEBUG: Rule 'material_steppers' modified arguments {"shallow_since": "1491412933 -0300"}
DEBUG: C:/users/win/_bazel_win/pj2hik4j/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl:159:5:
***********WARNING***********
npm: npm_install will require a package_lock_json attribute in future versions
*****************************
INFO: Analysed target //upvote/gae:monolith_binary.deploy (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Building: no action
WARNING: failed to create one or more convenience symlinks for prefix 'bazel-':
cannot create symbolic link bazel-bin -> C:/users/win/_bazel_win/pj2hik4j/execroot/__main__/bazel-out/x64_windows-fastbuild/bin: Cannot create junction (name=C:\users\win\desktop\upvote\upvote\bazel-bin, target=C:\users\win\_bazel_win\pj2hik4j\execroot\__main__\bazel-out\x64_windows-fastbuild\bin): ERROR: src/main/native/windows/file-jni.cc(88): nativeCreateJunction(C:\users\win\desktop\upvote\upvote\bazel-bin, C:\users\win\_bazel_win\pj2hik4j\execroot\__main__\bazel-out\x64_windows-fastbuild\bin): ERROR: src/main/native/windows/file.cc(346): DeviceIoControl(\\?\C:\users\win\desktop\upvote\upvote\bazel-bin): Access is denied.
cannot create symbolic link bazel-testlogs -> C:/users/win/_bazel_win/pj2hik4j/execroot/__main__/bazel-out/x64_windows-fastbuild/testlogs: Cannot create junction (name=C:\users\win\desktop\upvote\upvote\bazel-testlogs, target=C:\users\win\_bazel_win\pj2hik4j\execroot\__main__\bazel-out\x64_windows-fastbuild\testlogs): the junction's path got modified unexpectedly
cannot create symbolic link bazel-genfiles -> C:/users/win/_bazel_win/pj2hik4j/execroot/__main__/bazel-out/x64_windows-fastbuild/genfiles: Cannot create junction (name=C:\users\win\desktop\upvote\upvote\bazel-genfiles, target=C:\users\win\_bazel_win\pj2hik4j\execroot\__main__\bazel-out\x64_windows-fastbuild\genfiles): the junction's path got modified unexpectedly
cannot create symbolic link bazel-out -> C:/users/win/_bazel_win/pj2hik4j/execroot/__main__/bazel-out: Cannot create junction (name=C:\users\win\desktop\upvote\upvote\bazel-out, target=C:\users\win\_bazel_win\pj2hik4j\execroot\__main__\bazel-out): the junction's path got modified unexpectedly
cannot create symbolic link bazel-upvote -> C:/users/win/_bazel_win/pj2hik4j/execroot/__main__: Cannot create junction (name=C:\users\win\desktop\upvote\upvote\bazel-upvote, target=C:\users\win\_bazel_win\pj2hik4j\execroot\__main__): the junction's path got modified unexpectedly
[0 / 1] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //upvote/gae:monolith_binary.deploy up-to-date:
C:/users/win/_bazel_win/pj2hik4j/execroot/__main__/bazel-out/x64_windows-fastbuild/bin/upvote/gae/monolith_binary_deploy.sh
C:/users/win/_bazel_win/pj2hik4j/execroot/__main__/bazel-out/x64_windows-fastbuild/bin/upvote/gae/monolith_binary.deploy
C:/users/win/_bazel_win/pj2hik4j/execroot/__main__/bazel-out/x64_windows-fastbuild/bin/upvote/gae/monolith_binary.deploy.exe
INFO: Elapsed time: 35.375s, Critical Path: 0.01s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Running command line: C:/users/win/_bazel_win/pj2hik4j/execroot/__main__/bazel-out/x64_windows-fastbuild/bin/upvote/gae/monolith_binary.deploy.exe upvote-project app.yaml santa_api.yaml
INFO: Build completed successfully, 1 total action
C:\users\win\_bazel_win\pj2hik4j\execroot\__main__\bazel-out\x64_windows-fastbuild\bin\upvote\gae\monolith_binary.deploy: line 34: cd: /tmp/war.WIYuYnvI/__main__: No such file or directory
C:\users\win\_bazel_win\pj2hik4j\execroot\__main__\bazel-out\x64_windows-fastbuild\bin\upvote\gae\monolith_binary.deploy: line 37: /c/users/win/_bazel~1/pj2hik4j/execroot/__main__/bazel-~1/x64_wi~1/bin/upvote/gae/monoli~2.run/../com_google_cloud_sdk/bin/gcloud: No such file or directory
I have updated all the dependencies but still not able to deploy it to app engine.
Any ideas whats wrong here?
Thanks!
so currently this page displays item.occurredDt
from date returned by /api/web/events/query?asAdmin=true&more=true&perPage=10
, however this data does not contain this key, instead it seems to make more sense to use lastBlockedDt
? Or maybe recordedDt?
The other thing is it would be nice to be able to sort this column. Also I added Search by Event Type, I think that's useful as well.
per settings.py:
# If provided, a regex string that matches execution paths (read: not files)
,
this ends up going to santa via santa_const.PREFLIGHT.WHITELIST_REGEX
, and the docs in santa say:
A regex to whitelist if the binary or certificate scopes did not allow execution. Regexes are specified in ICU format.
and the examples I see say this doesn't have to be a directory.
The main "blockables" view would be easier to use if we could either sort based on the status or have a separate "Awaiting Votes" view to see what apps/certs still need to be investigated
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.