google / upvote_py2 Goto Github PK

A multi-platform binary whitelisting solution

License: Apache License 2.0

Python 69.25% JavaScript 20.92% CSS 0.56% HTML 4.76% Shell 0.16% Starlark 4.36%

upvote_py2's Introduction

Note: upvote_py2 is being archived

After thoughtful discussion, Upvote maintainers have decided to stop updating the Upvote codebase in Github, now named upvote_py2. The decision came down to our moving away from the App Engine Python 2 runtime. We have developed substantial new Python 3 code that would take more effort to port back to this environment than we can spend while focused on developing new features internally for Google.

To those still actively using Upvote: we are hopeful that someone else will take on the work of porting and maintaining a Python 3 version of Upvote to share with the community. If that happens we would be interested in contributing code and ideas from our internal version of Upvote. Development of Santa is not affected by this.

Social allowlisting works at Google. We hope it will continue to work for other organizations that need to scale allowlist management. We look forward to sharing ideas and improvements despite not having the resources at this time to support an external codebase.

Thanks,

Ben, on behalf of the Upvote team

Upvote

Upvote is a multi-platform binary whitelisting solution. It provides both a sync server and management interface for binary enforcement clients. Upvote currently supports Santa on macOS and Bit9 (now known as Carbon Black Protection) on Windows.

Features

First-party sync server for Santa
- Written in coordination with Santa's development team
User-oriented Policy Creation
- Apply policies to users instead of hosts
- No migration necessary when users get new hosts
BigQuery streaming
- Fast, easy, and scalable relational access to Santa and Bit9 execution data
Bundled Voting for .app bundles on macOS
- Easily create policy for an entire bundle at once
VirusTotal Integration
- View VirusTotal results directly in the detail page

Screenshot

Setup

See the docs page for full instructions.

Docs

Background
Setup
Santa
- Santa Syncing
- Santa Bundle Voting
Bit9
- Bit9 Syncing

Contributing

We are current working hard to get Upvote ready for external contributions. However, at this time, we do not have the necessary approvals to do so.

In the meantime, please feel free to file GitHub issues or post in our Google Group, upvote-discuss, with any comments, bugs, or feature requests.

Contributors

Core Contributors: Chief, Matthew

Special thanks to Danny, Haru, Maxim

And to the Santa team: Russell, Tom, Ed, Phillip

Disclaimer

This is not an official Google product.

upvote_py2's People

Contributors

Stargazers

Watchers

upvote_py2's Issues

Using stackdriver monitoring

after much effort, in farmersbusinessnetwork@8537d00 I added stackdriver monitoring support (commented out code). Unfortunately I quickly found that you can only do a timeseries event once per minute which means I lose almost all data, and get a ton of errors.

In out AWS environment we use datadog and there you have an agent which runs all the time which does the event sampling.

I'm searched and I can't find any equivalent for using stackdriver monitoring in app-engine. Obviously something is working behind the scenes because when I enabled stackdriver monitoring for the project I get a bunch of built-in metrics...however I see nothing I can use for custom metrics.

I thought of using deferred tasks however the class/method, and arguments get pickled so that won't help.

If there's a solution to this perhaps this project can come with something like what I've done by default.

Thoughts?

missing package.json

I think someone forgot to check in this file?

After merging I'm getting:

+ bazel run upvote/gae:monolith_binary.deploy -- --project santaupvote --version auto app.yaml santa_api.yaml
INFO: Build options have changed, discarding analysis cache.
ERROR: /Users/amohr/dev/upvote/upvote/gae/modules/upvote_app/frontend/BUILD:77:1: no such package '@npm//ng-html2js/bin': no such package '@nodejs//': Traceback (most recent call last):
	File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 418
		_prepare_node(repository_ctx)
	File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 325, in _prepare_node
		repository_ctx.file("bin/npm_node_repositories", conte...])), ...)
	File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 329, in repository_ctx.file
		"".join(["\necho Running npm \"$@\" in {...])
	File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 329, in "".join
		"\necho Running npm \"$@\" in {root}\n(cd \"{root}\"; \"$SCRIPT_DIR/{node}\" \"$SCRIPT_DIR/{script}\" --scripts-prepend-node-path=false \"$@\")\n".format(root = repository_ctx.path(packa..., <2 more arguments>)
	File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 333, in "\necho Running npm \"$@\" in {root}\n
		repository_ctx.path(package_json)
Unable to load package for //:package.json: not found. and referenced by '//upvote/gae/modules/upvote_app/frontend:ui_templates_genrule'
ERROR: Analysis of target '//upvote/gae:monolith_binary.deploy' failed; build aborted: no such package '@npm//ng-html2js/bin': no such package '@nodejs//': Traceback (most recent call last):
	File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 418
		_prepare_node(repository_ctx)
	File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 325, in _prepare_node
		repository_ctx.file("bin/npm_node_repositories", conte...])), ...)
	File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 329, in repository_ctx.file
		"".join(["\necho Running npm \"$@\" in {...])
	File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 329, in "".join
		"\necho Running npm \"$@\" in {root}\n(cd \"{root}\"; \"$SCRIPT_DIR/{node}\" \"$SCRIPT_DIR/{script}\" --scripts-prepend-node-path=false \"$@\")\n".format(root = repository_ctx.path(packa..., <2 more arguments>)
	File "/private/var/tmp/_bazel_amohr/13181c36bc3e64d8ab6186cf32fd8fae/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl", line 333, in "\necho Running npm \"$@\" in {root}\n
		repository_ctx.path(package_json)

DB migration for new commits

I'm seeing many PRs that touch the DB schemas and no discussion about if the new releases support DB migration to the new schemas. What is the DB migration strategy and is there one?

500 server error from /_ah/queue/deferred?

Hi,

Just recently I deployed an upvote (built with bazel-0.25.0, since this is more in line with date of last commit to this repo).
Overall the app functions ok, Santa clients are able to sync. GAE cron jobs also run successfully.

Though looking into logs, there are quite a few 500 errors from /_ah/queue/deferred handler with backtraces like below

Is this something expected with current repo?

Thanks a lot
Ivan.

backtrace:

'tuple' object has no attribute 'get_async'
Traceback (most recent call last):
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
    rv = self.handle_exception(request, response, e)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
    return handler.dispatch()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
    return method(*args, **kwargs)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 316, in post
    self.run_from_request()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 311, in run_from_request
    run(self.request.body)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 153, in run
    return func(*args, **kwds)
  File "/base/data/home/apps/u~proj-id/20210111t191206.432278677749216289/upvote/gae/datastore/utils.py", line 375, in _QueuedPaginatedBatchApply
    callback(last_page_results, *extra_args, **extra_kwargs)
  File "/base/data/home/apps/u~proj-id/20210111t191206.432278677749216289/upvote/gae/cron/role_syncing.py", line 269, in _SpiderBite
    hosts = ndb.get_multi(host_keys)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3933, in get_multi
    for future in get_multi_async(keys, **ctx_options)]
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/39c5fa12ec80ea1d/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3918, in get_multi_async
    return [key.get_async(**ctx_options) for key in keys]
AttributeError: 'tuple' object has no attribute 'get_async'

Easier certificate whitelisting

I'm starting to roll out santa/upvote to a number of machines and trying to figure out the best way to manage the initial creation of the whitelist (presumably easier going forward as we won't expect much new stuff to show up)

It would be nice if there was a "publishers" view on the upvote dashboard to make it easier to go through and mass whitelist certificates at the global level, perhaps sorted by how many apps fall under each certificate

Event.parent_name dropped from incoming santa events

this key is sent via santa, but is not stored to the events datastore table...this is useful when things are marked as run as root, so we can see if it was called via sudo or not.

Allow configuration of the `SantaHost.directory_{white,black}list_regex` fields

The sync server code currently supports populating those fields on preflight but Upvote offers no UI means of configuring defaults/values for these fields.

I see the path of least resistance being the following:

Add two fields to the Environment configuration struct: SANTA_DIRECTORY_{WHITE,BLACK}LIST_REGEX
For manual override, add an interface to the admin UI's host view.

Error Running init_project.sh

$ ./init_project.sh 

Initializing Upvote for GCP project: "white-feat-249718"
+ gcloud config set project white-feat-249718
Updated property [core/project].
+ echo Enabling App Engine...
Enabling App Engine...
++ gcloud app describe '--format=value(id)'
+ [[ '' != \w\h\i\t\e\-\f\e\a\t\-\2\4\9\7\1\8 ]]
+ gcloud app create
You are creating an app for project [white-feat-249718].
WARNING: Creating an App Engine application for a project is irreversible and the region
cannot be changed. More information about regions is at
<https://cloud.google.com/appengine/docs/locations>.

[ Snip ]

Creating App Engine application in project [white-feat-249718] and region [us-central]....done.                                                                                           
Success! The app is now created. Please use `gcloud app deploy` to deploy your first app.
+ echo Enabling APIs used by Upvote...
Enabling APIs used by Upvote...
+ gcloud services enable cloudkms.googleapis.com
Operation "operations/acf.fac977e7-0f57-45e4-8327-ca1329fd2b06" finished successfully.
+ gcloud services enable bigquery-json.googleapis.com
+ echo Creating encryption keys used to store Upvote API secrets...
Creating encryption keys used to store Upvote API secrets...
+ gcloud kms keyrings create ring --location=global
+ gcloud kms keys create virustotal --purpose=encryption --keyring=ring --location=global
+ echo Granting necessary permissions to App Engine...
Granting necessary permissions to App Engine...
++ gcloud iam service-accounts list '--filter=App Engine app default service account' '--format=value(email)'
+ [email protected]
+ gcloud projects add-iam-policy-binding white-feat-249718 --member serviceAccount:[email protected] --role roles/cloudkms.cryptoKeyEncrypterDecrypter
Updated IAM policy for project [white-feat-249718].
bindings:
- members:
  - serviceAccount:[email protected]
  role: roles/cloudkms.cryptoKeyEncrypterDecrypter
- members:
  - serviceAccount:[email protected]
  role: roles/editor
- members:
  - user:[email protected]
  - user:[email protected]
  role: roles/owner
etag: BwWQBBec2ao=
version: 1
+ echo Configuring App Engine...
Configuring App Engine...
+ ./manage_crons.py disable_all
+ echo Deploying temporary default version to App Engine...
Deploying temporary default version to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy -- white-feat-249718 app.yaml
+ echo

+ echo Deploying to App Engine...
Deploying to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy -- white-feat-249718 app.yaml
INFO: Call stack for the definition of repository 'remotejdk11_macos' which is a http_archive (rule definition at /private/var/tmp/_bazel_summit.hainey/f4e5186114806b0b8302ec418f57ad1d/external/bazel_tools/tools/build_defs/repo/http.bzl:237:16):
 - /DEFAULT.WORKSPACE.SUFFIX:219:1
ERROR: /private/var/tmp/_bazel_summit.hainey/f4e5186114806b0b8302ec418f57ad1d/external/com_google_protobuf/protobuf.bzl:130:19: Traceback (most recent call last):        File "/private/var/tmp/_bazel_summit.hainey/f4e5186114806b0b8302ec418f57ad1d/external/com_google_protobuf/protobuf.bzl", line 125
                rule(attrs = {"srcs": attr.label_list...()}, <2 more arguments>)
        File "/private/var/tmp/_bazel_summit.hainey/f4e5186114806b0b8302ec418f57ad1d/external/com_google_protobuf/protobuf.bzl", line 130, in rule
                attr.label(cfg = "host", executable = True, sin..., ...)
'single_file' is no longer supported. use allow_single_file instead. You can use --incompatible_disable_deprecated_attr_params=false to temporarily disable this check.
ERROR: /private/var/tmp/_bazel_summit.hainey/f4e5186114806b0b8302ec418f57ad1d/external/com_google_javascript_closure_compiler/BUILD:7:1: error loading package '@com_google_protobuf//': Ex
tension file 'protobuf.bzl' has errors and referenced by '@com_google_javascript_closure_compiler//:com_google_javascript_closure_compiler'
ERROR: Analysis of target '//upvote/gae:monolith_binary.deploy' failed; build aborted: error loading package '@com_google_protobuf//': Extension file 'protobuf.bzl' has errors
INFO: Elapsed time: 2.280s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
    currently loading: @com_google_protobuf//
    Fetching @gcloud_auth_httplib2_git; Cloning 136da2cd50aa7deb769062cf1d77259d64743a7f of https://github.com/GoogleCloudPlatform/google-auth-library-python-httplib2.git

Setup docs enhancement

First thank you. Second very easy to get started. Had an instance up and running in under 30mins, and this is from someone who has very little GCP experience and zero experience with bazel. The only snag I hit was after changing the settings.py file. I would add a note stating that admins can update the instance via:

bazel run upvote/gae:monolith_binary.deploy -- "${PROJ_ID}" app.yaml 2> /dev/null || echo
bazel run upvote/gae:monolith_binary.deploy -- "${PROJ_ID}" app.yaml santa_api.yaml bit9_api.yaml

Not actually sure if the first step was required.

emergency mode seems to not be effective and has no instructions

we were in full emergency mode today and hit the emergency button (which doesn't explain what it does on the page), which confusingly brought up a "stop" and "go" radio button with no explanations what they do, so we figured we needed to hit "go", after which another go button popped up, which we hit, and have no effect of disabling lockdown mode.

clean full sync error

we just got this when attempting to do a clean full sync for the first time since one of our machines was not downloading all the rules:

BadRequestError: cursor position is outside the range of the original query
at check_rpc_success (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py:1379)
at __query_result_hook (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/datastore/datastore_query.py:2910)
at get_result (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py:615)
at _on_rpc_completion (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py:513)
at run_to_queue (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py:969)
at run_to_queue (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py:1999)
at _help_tasklet_along (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py:427)
at run_to_queue (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py:962)
at helper (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/context.py:890)
at has_next_async (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py:1793)
at _help_tasklet_along (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py:427)
at _fetch_page_async (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py:1380)
at _help_tasklet_along (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py:427)
at get_result (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py:383)
at fetch_page (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py:1362)
at positional_wrapper (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/1/google/appengine/ext/ndb/utils.py:160)
at post (/base/data/home/apps/m~[PROJ]/santa-api:20190204t133533.415891392575063416/upvote/gae/modules/santa_api/sync.py:923)
at _RecordRequest (/base/data/home/apps/m~[PROJ]/santa-api:20190204t133533.415891392575063416/upvote/gae/utils/handler_utils.py:105)
at dispatch (/base/alloc/tmpfs/dynamic_runtimes/python27g/f612b68945952bd1/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:545)

missing indicator of regex white/blacklisting in events listing

seems like the method should also take host, path, and somehow get the globalregex settings

Prevent upvote controlling the ClientMode

Right now it looks like upvote controls the ClientMode setting - I'd prefer to do this with our existing MDM solution via the plist we push to set the basic santa settings - this would allow us to continue to use our various mdm computer groups rather than re-inventing that in upvote

ImportError on preflight call

Calling preflight gives a 500 error.

Backtrace:

ImportError: No module named pkg_resources
at (/base/data/home/apps/eupvote-xxx/santa-api:auto.411705267076089495/external/gcloud_bigquery_archive/google/cloud/bigquery/init.py:31)
at (/base/data/home/apps/eupvote-xxx/santa-api:auto.411705267076089495/upvote/gae/bigquery/tables.py:25)
at (/base/data/home/apps/eupvote-xxx/santa-api:auto.411705267076089495/upvote/gae/datastore/models/santa.py:21)
at (/base/data/home/apps/eupvote-xxx/santa-api:auto.411705267076089495/upvote/gae/datastore/models/rule.py:17)
at (/base/data/home/apps/eupvote-xxx/santa-api:auto.411705267076089495/upvote/gae/real_appengine_config.py:20)
at (/base/data/home/apps/eupvote-xxx/santa-api:auto.411705267076089495/appengine_config.py:10)
at import_module (/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_dist/lib/python2.7/importlib/init.py:37)
at initialize (/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_lib/versions/1/google/appengine/api/lib_config.py:165)
at _update_configs (/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_lib/versions/1/google/appengine/api/lib_config.py:294)
at getattr (/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_lib/versions/1/google/appengine/api/lib_config.py:358)
at Handle (/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py:240)

change 'whitelist' to 'allowlist'

as per inclusive code guidelines :)

whitelist/blacklist docs/var names seem to be wrong

per settings.py:
# If provided, a regex string that matches execution paths (read: not files),

this ends up going to santa via santa_const.PREFLIGHT.WHITELIST_REGEX, and the docs in santa say:
A regex to whitelist if the binary or certificate scopes did not allow execution. Regexes are specified in ICU format.

and the examples I see say this doesn't have to be a directory.

transient errors

Have started seeing more of these:
InternalTransientError: Temporary error in fetching URL: https://www.googleapis.com/bigquery/v2/projects/[PROJ]/datasets/gae_streaming/tables/Host/insertAll, please re-try at _get_fetch_result (/base/alloc/tmpfs/dynamic_runtimes/python27g/7e468a4e2dbc991a/python27/python27_lib/versions/1/google/appengine/api/urlfetch.py:446)

as well:
TimeoutError: (<requests.packages.urllib3.contrib.appengine.AppEngineManager object at 0x2a583ad9b4d0>, DeadlineExceededError('Deadline exceeded while waiting for HTTP response from URL: https://www.googleapis.com/bigquery/v2/projects/[PROJ]/datasets/gae_streaming/tables/Host/insertAll',))

and
ConnectionError: Connection closed unexpectedly by server at URL: https://www.googleapis.com/bigquery/v2/projects/[PROJ]/datasets/gae_streaming/tables/Host/insertAll

sounds like there's some missing retries. Does this mean that our bigquery tables will be missing entries? If these are retried then something should be changed for them not the show up in stackdriver error reporting.

Error in npm

Hey!
When I run ./inti_project.sh, I get an error which says Error: ENOENT: no such file or directory, scandir 'node_modules/.bin'. Any help is appreciated!

Stacktrace -

./init_project.sh 
Initializing Upvote for GCP project: "emerald-caster-246406"
+ gcloud config set project emerald-caster-246406
Updated property [core/project].
+ echo Enabling App Engine...
Enabling App Engine...
++ gcloud app describe '--format=value(id)'
+ [[ emerald-caster-246406 != \e\m\e\r\a\l\d\-\c\a\s\t\e\r\-\2\4\6\4\0\6 ]]
+ echo Enabling APIs used by Upvote...
Enabling APIs used by Upvote...
+ gcloud services enable cloudkms.googleapis.com
+ gcloud services enable bigquery-json.googleapis.com
+ echo Creating encryption keys used to store Upvote API secrets...
Creating encryption keys used to store Upvote API secrets...
+ echo Granting necessary permissions to App Engine...
Granting necessary permissions to App Engine...
++ gcloud iam service-accounts list '--filter=App Engine app default service account' '--format=value(email)'
+ SERVICE_ACCOUNT=emerald-caster-246406@appspot.gserviceaccount.com
+ gcloud projects add-iam-policy-binding emerald-caster-246406 --member serviceAccount:[email protected] --role roles/cloudkms.cryptoKeyEncrypterDecrypter
Updated IAM policy for project [emerald-caster-246406].
bindings:
- members:
  - serviceAccount:[email protected]
  role: roles/cloudkms.cryptoKeyEncrypterDecrypter
- members:
  - serviceAccount:[email protected]
  role: roles/compute.serviceAgent
- members:
  - serviceAccount:service-79565312020@container-engine-robot.iam.gserviceaccount.com
  role: roles/container.serviceAgent
- members:
  - serviceAccount:[email protected]
  - serviceAccount:[email protected]
  - serviceAccount:[email protected]
  - serviceAccount:[email protected]
  role: roles/editor
- members:
  - user:[email protected]
  role: roles/owner
etag: BwWUZzgfqLs=
version: 1
+ echo Configuring App Engine...
Configuring App Engine...
+ ./manage_crons.py disable_all
+ echo Deploying temporary default version to App Engine...
Deploying temporary default version to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy -- emerald-caster-246406 app.yaml
+ echo

+ echo Deploying to App Engine...
Deploying to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy --incompatible_depset_union=false -- emerald-caster-246406 app.yaml
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN npm No description
npm WARN npm No repository field.
npm WARN npm No license field.

added 4 packages from 2 contributors and audited 4 packages in 5.274s
found 0 vulnerabilities

yarn install v1.12.1
warning package.json: No license field
warning No license field
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 0.66s.
INFO: Call stack for the definition of repository 'build_bazel_rules_nodejs_npm_install_deps' which is a yarn_install (rule definition at /home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl:245:16):
 - /home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs/internal/node/node_repositories.bzl:554:3
 - /home/umashankar/upvote/WORKSPACE:493:1
ERROR: An error occurred during the fetch of repository 'build_bazel_rules_nodejs_npm_install_deps':
   Traceback (most recent call last):
	File "/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 243
		_create_build_file(repository_ctx, node)
	File "/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 88, in _create_build_file
		fail(("node failed: \nSTDOUT:\n%s\nST...)))
node failed: 
STDOUT:

STDERR:
fs.js:115
    throw err;
    ^

Error: ENOENT: no such file or directory, scandir 'node_modules/.bin'
    at Object.readdirSync (fs.js:783:3)
    at listFiles (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:168:8)
    at generateRootBuildFile (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:96:20)
    at main (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:87:3)
    at Object.<anonymous> (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:57:3)
    at Module._compile (internal/modules/cjs/loader.js:688:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:699:10)
    at Module.load (internal/modules/cjs/loader.js:598:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:537:12)
    at Function.Module._load (internal/modules/cjs/loader.js:529:3)
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN npm No description
npm WARN npm No repository field.
npm WARN npm No license field.

added 4 packages from 2 contributors and audited 4 packages in 12.159s
found 0 vulnerabilities

DEBUG: /home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl:159:5: 
***********WARNING***********
npm: npm_install will require a package_lock_json attribute in future versions
*****************************
ERROR: /home/umashankar/upvote/upvote/gae/modules/upvote_app/frontend/BUILD:70:1: no such package '@npm//ng-html2js/bin': no such package '@build_bazel_rules_nodejs_npm_install_deps//': Traceback (most recent call last):
	File "/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 243
		_create_build_file(repository_ctx, node)
	File "/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 88, in _create_build_file
		fail(("node failed: \nSTDOUT:\n%s\nST...)))
node failed: 
STDOUT:

STDERR:
fs.js:115
    throw err;
    ^

Error: ENOENT: no such file or directory, scandir 'node_modules/.bin'
    at Object.readdirSync (fs.js:783:3)
    at listFiles (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:168:8)
    at generateRootBuildFile (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:96:20)
    at main (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:87:3)
    at Object.<anonymous> (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:57:3)
    at Module._compile (internal/modules/cjs/loader.js:688:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:699:10)
    at Module.load (internal/modules/cjs/loader.js:598:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:537:12)
    at Function.Module._load (internal/modules/cjs/loader.js:529:3)
 and referenced by '//upvote/gae/modules/upvote_app/frontend:admin_templates_genrule'
ERROR: Analysis of target '//upvote/gae:monolith_binary.deploy' failed; build aborted: no such package '@npm//ng-html2js/bin': no such package '@build_bazel_rules_nodejs_npm_install_deps//': Traceback (most recent call last):
	File "/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 243
		_create_build_file(repository_ctx, node)
	File "/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl", line 88, in _create_build_file
		fail(("node failed: \nSTDOUT:\n%s\nST...)))
node failed: 
STDOUT:

STDERR:
fs.js:115
    throw err;
    ^

Error: ENOENT: no such file or directory, scandir 'node_modules/.bin'
    at Object.readdirSync (fs.js:783:3)
    at listFiles (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:168:8)
    at generateRootBuildFile (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:96:20)
    at main (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:87:3)
    at Object.<anonymous> (/home/umashankar/.cache/bazel/_bazel_umashankar/291234c7517d3001bef7300e84ef7cbb/external/build_bazel_rules_nodejs_npm_install_deps/generate_build_file.js:57:3)
    at Module._compile (internal/modules/cjs/loader.js:688:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:699:10)
    at Module.load (internal/modules/cjs/loader.js:598:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:537:12)
    at Function.Module._load (internal/modules/cjs/loader.js:529:3)
INFO: Elapsed time: 32.583s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (119 packages loaded, 25421 targets configured)
FAILED: Build did NOT complete successfully (119 packages loaded, 25421 targets configured)
    Fetching @oauth2client_git; Cloning 97320af2733f7bdbe47f067327610e348f953ae1 of https://github.com/google/oauth2client.git 26s
    Fetching @npm; Restarting. 26s

"Awaiting Votes" view

The main "blockables" view would be easier to use if we could either sort based on the status or have a separate "Awaiting Votes" view to see what apps/certs still need to be investigated

Enhancements / Bug fixes

Here's a set of things we've done, not sure if you guys would like PRs or incorporate any in future releases
fix certificate state propagation on admin blockables page
- farmersbusinessnetwork@3490ba3
- speedup: farmersbusinessnetwork@d5f0731
support compact listing: farmersbusinessnetwork@e8e87dd
fix updatedDt on blockables page, fix details of certificates, and add "type" to admin blockables page: farmersbusinessnetwork@8b3964b (later fixed to use prettifyType: farmersbusinessnetwork@7b456eb)
don't navigate to new blockables page when clicking on item, just update card (as it was intended to do): farmersbusinessnetwork@b3e5bf1
don't navigate when clicking on certificate of blockable in card on admin page: farmersbusinessnetwork@64f2a83
add filtering by event type to Events page: farmersbusinessnetwork@d2e6919
add datadog monitoring support (need to investigate why not working on my private account)
- farmersbusinessnetwork@95dd7e6
- farmersbusinessnetwork@97aa281
- farmersbusinessnetwork@a776c78
- farmersbusinessnetwork@271486a
- farmersbusinessnetwork@e580708
search by primary user on hosts page: farmersbusinessnetwork@4c87e9b
general list enhancements: wildcard search support, sorting, title of search box in hosts doesn't update on radio button click (#20)
add state to list of events
- farmersbusinessnetwork@9a8ea96
- farmersbusinessnetwork@20d486f
- speedup: farmersbusinessnetwork@8921958
add global whitelist/blacklist to settings page: farmersbusinessnetwork@fa30e7c (be careful as it also whitelists homebrew)
add global/host whitelisting status to events page: farmersbusinessnetwork@c8b91ce
Hosts: display + toggle support for transitive whitelisting, and display white/block listing farmersbusinessnetwork@fe726bc
Events: enable parent_name: farmersbusinessnetwork@5cf412f
Support for host based path based regex setting, and ability to "go to bundle" from events page: farmersbusinessnetwork@45aa766
Support GUI compiler whitelisting: farmersbusinessnetwork@3e91cba (fixes a refresh bug with "reset" as well) First set binary as compiler, then up-vote to populate correct rule
Bump to use latest bazel appengine which uses gcloud deploy instead of appcfg: farmersbusinessnetwork@05abd2d
Add support for google cloud monitoring: farmersbusinessnetwork@8537d00
Add support for showing and searching hosts by serial number: farmersbusinessnetwork@f1cab34
search by host by ID: farmersbusinessnetwork@2f69dcb + farmersbusinessnetwork@2d30210 + farmersbusinessnetwork@d9122a1

Error when running init_project.sh

I'm having issues following the steps to deploy

Deploying to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy -- upvote-poc-1 app.yaml santa_api.yaml bit9_api.yaml --verbose_failures
INFO: Analysed target //upvote/gae:monolith_binary.deploy (0 packages loaded).
INFO: Found 1 target...
ERROR: missing input file '@local_jdk//:jre/lib/rt.jar'
ERROR: /private/var/tmp/_bazel_eric/7b851cf8ed89b6be50e03e0a8350cb67/external/io_bazel_rules_closure/java/io/bazel/rules/closure/BUILD:17:1: @io_bazel_rules_closure//java/io/bazel/rules/closure:ClosureWorker: missing input file '@local_jdk//:jre/lib/rt.jar'
Target //upvote/gae:monolith_binary.deploy failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /private/var/tmp/_bazel_eric/7b851cf8ed89b6be50e03e0a8350cb67/external/io_bazel_rules_closure/java/io/bazel/rules/closure/BUILD:17:1 1 input file(s) do not exist
INFO: Elapsed time: 2.483s, Critical Path: 0.01s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

Any ideas?

Thanks!

unexpected keyword 'overwrite_appengine_config' error while deploying the app to app engine

I'm having issues following the steps to deploy

+ echo Deploying to App Engine...
Deploying to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy -- rv3-upvote app.yaml santa_api.yaml
INFO: Invocation ID: bbd6d94c-828d-4fa4-97e6-80cad0f54c3c
Loading:
Loading: 0 packages loaded
ERROR: E:/server/upvote/upvote/gae/BUILD:59:1: Traceback (most recent call last):
        File "E:/server/upvote/upvote/gae/BUILD", line 59
                upvote_appengine_test(name = "appengine_config_test", si...", <2 more arguments>)
        File "E:/server/upvote/upvote/builddefs.bzl", line 8, in upvote_appengine_test
                py_appengine_test(name = name, srcs = srcs, deps = dep..., <3 more arguments>)
unexpected keyword 'overwrite_appengine_config' in call to py_appengine_test(name, srcs, deps = [], data = [], libraries = {}, size = None)
ERROR: error loading package 'upvote/gae': Package 'upvote/gae' contains errors
INFO: Elapsed time: 0.402s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully (0 packages loaded)

I have tried overwrite_appengine_config = false, on upvote\gae\BUILD file but that didnt work.

And the build is still failing after the last update.

Any ideas?

Thanks!

build issue with latest PR

ERROR: /Users/amohr/dev/upvote/upvote/gae/modules/upvote_app/frontend/BUILD:20:1: Checking 116 JS files in //upvote/gae/modules/upvote_app/frontend:closure_lib failed (Exit 1)
upvote/gae/modules/upvote_app/frontend/web_ui/detailpage/detail-controller.js:43: WARNING - Please do not use goog.scope to hide declarations.
It is preferable to either create an @private namespaced declaration, or migrate to goog.module.
let ComponentRegistry;
    ^
  ProTip: "JSC_GOOG_SCOPE_HIDDEN_TYPE" can be added to the `suppress` attribute of:
  //upvote/gae/modules/upvote_app/frontend:closure_lib

upvote/gae/modules/upvote_app/frontend/admin_ui/blockablepage/blockablepage-controller.js:17: ERROR - Namespace not provided by any srcs or direct deps of //upvote/gae/modules/upvote_app/frontend:closure_lib.
goog.require('goog.dom.safe');
             ^
  ProTip: "CR_NOT_PROVIDED" or "strictDependencies" can be added to the `suppress` attribute of:
  //upvote/gae/modules/upvote_app/frontend:closure_lib

1 error(s), 1 warning(s)

guessing the closure lib version needs to be updated?

Transitive / Compiler Rule Support

I'm attempting to understand how I can actually create/manage a compiler rule with Upvote. The use-case being, we would enable transitive whitelisting for certain folks who are using certain tools.

I see the codebase fully supports enabling transitive whitelisting mode on clients (and it works). However I see limited references to compiler rules, the main one being WHITELIST_COMPILER as part of a list of Santa rule policies.

Even if there is no UI support, is this something I can manually add to the Datastore Rules table, so it would get synced to clients?

cron datastore backup URL incorrect

per https://github.com/google/upvote/blob/master/upvote/gae/cron.yaml#L20 it's listed as /cron/export/datastore-to-gcs, however per my reading and tests it seems like it should be /cron/datastore/backup

logupload endpoint missing

when we set a host to upload logs, we're getting 404 on GET [PROJ].appspot.com/api/santa/logupload/[HOSTID]

Event deduping leads to incorrect event results

I'm still debugging how deduping works, however what I do know is that if a machine is in monitor mode, executes something that is not allowed, an ALLOW_UNKNOWN event is recorded to the datastore.

If now you move the machine to lockdown, attempt to execute the same executable, you'll get the santa warning, and eventually the event will get synced, and the recorded and last_blocked will get updated, however the event_type will stay at ALLOW_UNKNOWN.

So if you go to the events page it will list the executable as ALLOW_UNKNOWN, when in reality it was blocked.

Error running bezel with the init script

`Deploying to App Engine...

bazel run upvote/gae:monolith_binary.deploy -- upvote-233719 app.yaml santa_api.yaml
DEBUG: Rule 'io_bazel_rules_appengine' modified arguments {"shallow_since": "1544654581 -0500"}
ERROR: error loading package '': Encountered error while reading extension file 'closure/defs.bzl': no such package '@io_bazel_rules_closure//closure': java.io.IOException: Error downloading [https://github.com/bazelbuild/rules_closure/archive/d1110778a2e94bcdac5d5d00044dcb6cd07f1d51.tar.gz] to /private/var/tmp/_bazel_tonylin/bca64804a1a5aa72f10fce1fb59a3167/external/io_bazel_rules_closure/d1110778a2e94bcdac5d5d00044dcb6cd07f1d51.tar.gz: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
ERROR: error loading package '': Encountered error while reading extension file 'closure/defs.bzl': no such package '@io_bazel_rules_closure//closure': java.io.IOException: Error downloading [https://github.com/bazelbuild/rules_closure/archive/d1110778a2e94bcdac5d5d00044dcb6cd07f1d51.tar.gz] to /private/var/tmp/_bazel_tonylin/bca64804a1a5aa72f10fce1fb59a3167/external/io_bazel_rules_closure/d1110778a2e94bcdac5d5d00044dcb6cd07f1d51.tar.gz: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
INFO: Elapsed time: 13.410s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)
Fetching @io_bazel_rules_closure; fetching 13s`

Any help is appreciated. Thanks.

Didn't run into access issues....not sure what this is

Santa Client Certificate Authentication

so per the santa docs, the way one is supposed to lock down santad<->upvote communication is via server based client certificate auth. Unfortunately, after looking at ValidateClient, it seems some sort of header token auth is instead expected by upvote.

I looked a little bit into implementing client certificate auth however it seems webapp2 does not support this, nor does the GAE environment in general. What method of santa<->upvote auth do the authors of upvote expect end-users to implement to be secure?

all hosts changed in datastore by updating SANTA_DEFAULT_CLIENT_MODE in upvote/gae/shared/common/settings.py

we had an accident where I accidentally deployed from the master branch instead of our personal branch, which ended up resetting SANTA_DEFAULT_CLIENT_MODE. I would have imagined this would have only affected new hosts, however ALL hosts ended up getting reset to lockdown mode triggering a lot of unhappiness :)

I think for hosts that are in the datastore should not be affected by a change to this variable.

DeadlineExceededError

Howdy!

I'm seeing an increase in DeadlineExceededError errors in Stackdriver.

TimeoutError:` (<requests.packages.urllib3.contrib.appengine.AppEngineManager object at 0x3c101e4444>, DeadlineExceededError('Deadline exceeded while waiting for HTTP response from URL: https://www.googleapis.com/bigquery/v2/projects/sync-server/datasets/gae_streaming/tables/Execution/insertAll',))

Any guidance on debugging approaches or tweaks in code to make to limit this error?

datetime bug in events.html?

so currently this page displays item.occurredDt from date returned by /api/web/events/query?asAdmin=true&more=true&perPage=10, however this data does not contain this key, instead it seems to make more sense to use lastBlockedDt? Or maybe recordedDt? The other thing is it would be nice to be able to sort this column. Also I added Search by Event Type, I think that's useful as well.

preflight failed

I'm having trouble getting machines to check in. I have tried to follow your setup instructions, and I have constructed a configuration profile for the santa client with

<key>SyncBaseURL</key>
<string>https://<my-app>.appspot.com/api/santa</string>

When I do this I get:

santactl sync
HTTP Response: 302 found
Preflight failed, aborting run

I have tried adding a trailing /, but that just gives me a 404 error. Looking at this in curl it does seem to be returning 302:

curl -vvv https://<my-app>.appspot.com/api/santa

< location: https://www.google.com/accounts/ServiceLogin?service=ah&passive=true&continue=https://appengine.google.com/_ah/conflogin%3Fcontinue%3Dhttps://<my-app>.appspot.com/api/santa
< x-cloud-trace-context: 6c130ec3499f2df8b4a48318d3272719;o=1
< date: Wed, 09 May 2018 17:52:41 GMT
< content-type: text/html
< server: Google Frontend
< content-length: 0

But not really sure what mistake I might have made as it doesn't seem like any special settings on the app engine side are required.

Setting Failsafe_Administrators Produces a Traceback Error

To reproduce:

Set an email address for FAILSAFE_ADMINISTRATORS in /upvote/gae/shared/common/settings.py and saved the file.
Run bazel run upvote/gae:monolith_binary.deploy -- ${PROJ_ID} which shows no errors.
Manually triggered the cron from the Cloud Console's cron page Which failed.
Opened https://<my-app>.appspot.com/admin/settings which shows the traceback:

  File "/base/alloc/tmpfs/dynamic_runtimes/python27/6baee03590d046f1_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/base/alloc/tmpfs/dynamic_runtimes/python27/6baee03590d046f1_unzipped/python27_lib/versions/1/google/appengine/api/lib_config.py", line 351, in __getattr__
    self._update_configs()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27/6baee03590d046f1_unzipped/python27_lib/versions/1/google/appengine/api/lib_config.py", line 287, in _update_configs
    self._registry.initialize()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27/6baee03590d046f1_unzipped/python27_lib/versions/1/google/appengine/api/lib_config.py", line 160, in initialize
    import_func(self._modname)
  File "/base/data/home/apps/d~its-upvote/auto.408675188517060759/appengine_config.py", line 10, in <module>
    import upvote.gae.real_appengine_config
  File "/base/data/home/apps/d~its-upvote/auto.408675188517060759/upvote/gae/real_appengine_config.py", line 20, in <module>
    from upvote.gae import appengine_config_utils
  File "/base/data/home/apps/d~its-upvote/auto.408675188517060759/upvote/gae/appengine_config_utils.py", line 20, in <module>
    from upvote.gae.datastore import test_utils
  File "/base/data/home/apps/d~its-upvote/auto.408675188517060759/upvote/gae/datastore/test_utils.py", line 26, in <module>
    from upvote.gae.datastore.models import base
  File "/base/data/home/apps/d~its-upvote/auto.408675188517060759/upvote/gae/datastore/models/base.py", line 29, in <module>
    from upvote.gae.datastore.models import bigquery
  File "/base/data/home/apps/d~its-upvote/auto.408675188517060759/upvote/gae/datastore/models/bigquery.py", line 22, in <module>
    from upvote.gae.shared.common import settings
  File "/base/data/home/apps/d~its-upvote/auto.408675188517060759/upvote/gae/shared/common/settings.py", line 72
    FAILSAFE_ADMINISTRATORS = [[email protected]]
                                             ^
SyntaxError: invalid syntax

Leaving FAILSAFE_ADMINISTRATORS blank and rerunning steps 2, 3, and 4 result in successful access to the admin site.

What happens next section references google specific gmenu tool

The what happens next section references the google specific gmenu tool to force a sync with rules.

bazel issue when deploying to app engine

I'm having issues following the steps to deploy

Deploying to App Engine...

Deploying to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy -- rv3-upvote app.yaml santa_api.yaml
INFO: Invocation ID: 2f292a57-d043-40cf-9751-1c9266c84dcb
Loading:
Loading: 0 packages loaded
ERROR: error loading package '': Encountered error while reading extension file 'appengine/py_appengine.bzl': no such package '@io_bazel_rules_appengine//appengine': The native git_repository rule is deprecated. load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository") for a replacement.
Use --incompatible_remove_native_git_repository=false to temporarily continue using the native rule.
ERROR: error loading package '': Encountered error while reading extension file 'appengine/py_appengine.bzl': no such package '@io_bazel_rules_appengine//appengine': The native git_repository rule is deprecated. load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository") for a replacement.
Use --incompatible_remove_native_git_repository=false to temporarily continue using the native rule.
INFO: Elapsed time: 0.186s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully (0 packages loaded)

Any ideas?

Thanks!

Error of type -610

I followed the installation instructions but I'm getting the following errors and the install just hangs until I quit:

$ bazel run upvote/gae:monolith_binary.deploy -- "${PROJ_ID}" app.yaml                     
WARNING: /private/var/tmp/_bazel_seadmin/dc15004ff28b5d5c51a9b82d37540f68/external/com_google_protobuf_java/WORKSPACE:1: Workspace name in /private/var/tmp/_bazel_seadmin/dc15004ff28b5d5c51a9b82d37540f68/external/com_google_protobuf_java/WORKSPACE (@com_google_protobuf) does not match the name given in the repository's definition (@com_google_protobuf_java); this will cause a build error in future versions
INFO: Analysed target //upvote/gae:monolith_binary.deploy (1 packages loaded).
INFO: Found 1 target...
Target //upvote/gae:monolith_binary.deploy up-to-date:
  bazel-bin/upvote/gae/monolith_binary_deploy.sh
  bazel-bin/upvote/gae/monolith_binary.deploy
INFO: Elapsed time: 3.435s, Critical Path: 0.07s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action

INFO: Running command line: bazel-bin/upvote/gae/monolith_binary.deploy upvote-205913 app.yaml
04:25 PM Application: upvote-205913 (was: auto); version: auto
04:25 PM Host: appengine.google.com
04:25 PM Starting update of app: upvote-205913, version: auto
04:25 PM Getting current resource limits.
0:370: execution error: An error of type -610 has occurred. (-610)
70:78: execution error: An error of type -610 has occurred. (-610)
69:77: execution error: An error of type -610 has occurred. (-610)
^C
Bazel caught interrupt signal; shutting down.

Here are my Basel and gcloud versions:

$ bazel version  
Build label: 0.13.1-homebrew
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed May 23 16:57:59 2018 (1527094679)
Build timestamp: 1527094679
Build timestamp as int: 1527094679

$ gcloud version
Google Cloud SDK 203.0.0
bq 2.0.34
core 2018.05.25
gsutil 4.31

Any ideas what's wrong?

Add support for filtering hosts by primary user

Here's how I did it: farmersbusinessnetwork@4c87e9b

getting XSRF errors with latest code

Traceback (most recent call last):
  File "/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
    return handler.dispatch()
  File "/base/data/home/apps/m~santaupvote/santa-api:auto.411437352926953149/upvote/gae/modules/santa_api/sync.py", line 110, in dispatch
    self.abort(httplib.FORBIDDEN, explanation='XSRF token missing/invalid.')
  File "/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 573, in abort
    abort(code, *args, **kwargs)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27/8882c914eb6132e9_unzipped/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1804, in abort
    raise cls(*args, **kwargs)
HTTPForbidden: XSRF token missing/invalid.

init_project.sh is not executable

~/GitHub/upvote|master ⇒ ./init_project.sh
zsh: permission denied: ./init_project.sh
~/GitHub/upvote|master ⇒ chmod a+x init_project.sh

init script deploy error

Getting this error when installing

environment:

bazel version: 1.1.0
OSX 10.14.6
java version "1.8.0_131"

I tried:

bazel clean --expunge
ensured xcode was up to date

error on the last command bazel run upvote/gae:monolith_binary.deploy -- "${PROJ_ID}" app.yaml

+ gcloud config set project redacted
Updated property [core/project].
+ echo Deploying to App Engine...
Deploying to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy -- redacted-app.yaml
Starting local Bazel server and connecting to it...
INFO: Writing tracer profile to '/private/var/tmp/_bazel_/7efb8f1f32f0c308f1333fb4570cfc62/command.profile.gz'
ERROR: /private/var/tmp/_bazel_/7efb8f1f32f0c308f1333fb4570cfc62/external/com_google_protobuf/protobuf.bzl:130:19: Traceback (most recent call last):
    File "/private/var/tmp/_bazel_/7efb8f1f32f0c308f1333fb4570cfc62/external/com_google_protobuf/protobuf.bzl", line 125
        rule(<3 more arguments>)
    File "/private/var/tmp/_bazel_/7efb8f1f32f0c308f1333fb4570cfc62/external/com_google_protobuf/protobuf.bzl", line 130, in rule
        attr.label(cfg = "host", <3 more arguments>)
'single_file' is no longer supported. use allow_single_file instead. You can use --incompatible_disable_deprecated_attr_params=false to temporarily disable this check.
INFO: Call stack for the definition of repository 'remote_java_tools_darwin' which is a http_archive (rule definition at /private/var/tmp/_bazel_/7efb8f1f32f0c308f1333fb4570cfc62/external/bazel_tools/tools/build_defs/repo/http.bzl:292:16):
 - /private/var/tmp/_bazel_/7efb8f1f32f0c308f1333fb4570cfc62/external/bazel_tools/tools/build_defs/repo/utils.bzl:205:9
 - /DEFAULT.WORKSPACE.SUFFIX:280:1
ERROR: /private/var/tmp/_bazel_/7efb8f1f32f0c308f1333fb4570cfc62/external/com_google_javascript_closure_compiler/BUILD:7:1: error loading package '@com_google_protobuf//': Extension file 'protobuf.bzl' has errors and referenced by '@com_google_javascript_closure_compiler//:com_google_javascript_closure_compiler'
ERROR: Analysis of target '//upvote/gae:monolith_binary.deploy' failed; build aborted: error loading package '@com_google_protobuf//': Extension file 'protobuf.bzl' has errors
INFO: Elapsed time: 11.606s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (95 packages loaded, 1507 targets configured)
FAILED: Build did NOT complete successfully (95 packages loaded, 1507 targets configured)
    Fetching @local_config_xcode; fetching 7s
    Fetching @com_google_javascript_closure_library; Restarting.```

Blockables/Certs/Votes API batching

For two different reasons:
We added a "state" column to the events admin page to keep track of the current state of each event's blockable state. To do this we needed to do a page-size of requests to get the blockables, and then get any certs which were available. This is a lot of API calls and ends up making our instance count often peak

Similarly on the blockables page we grab any certs which are available for blockables to get a better state result.

I can see this being solved in one of two ways:

On the events query API call, a parameter which controls if you want it to return the associated blockables/certs...and similarly on the blockables query call to return any certs/votes associated with the results
Having batch get support for certs/blockables.

thoughts? It would be good to handle #28 as well with the design.

santa-api /_ah/warmup 500 errors

We're getting a LOT of 500s from the santa-api with errors like these in the logs:

The request failed because the instance could not start successfully
Threads started by this request continued executing past the hard deadline.

any ideas?

UI: searching by host ID broken

from in the search() method it assumes all searches by id will load a card, however on the host page there is no card. Further, you're not allowed to use a searchBase of id so it's broken in a couple ways

documentation on how to use upvote locally as library with service account creds

I've been trying to create a helper script by using upvote locally as an API, and was able to get to retrieving Blockables: https://gist.github.com/thehesiod/0ffde6e56ea56aede919fe6068a73f89
after fixing this issue: googleapis/google-cloud-datastore#211

The way I have it is by adding google-cloud-sdk/platform/google_appengine and the root upvote folder to the pythonpath, as well as creating a bunch of __init__.py files in the upvote source folders.

The issue I ran into is the following: AttributeError: 'BeginTransactionRequest' object has no attribute 'transaction_options'

I believe because by installing the googledatastore module, there are now two sets of datastore protos: site-packages/google/cloud/datastore_v1/proto/datastore_pb2.py and site-packages/google/cloud/proto/datastore/v1/datastore_pb2.py

The datastore_v1 one declares the transaction_options field descriptor, while the v1 does not. It's kinda odd that both are v1 and yet different.

I have the following google modules installed with the latest cloud sdk:

google-api-core                 1.3.0    
google-api-python-client        1.5.5    
google-auth                     1.5.0    
google-auth-httplib2            0.0.3    
google-cloud                    0.34.0   
google-cloud-bigquery           1.5.0    
google-cloud-core               0.28.1   
google-cloud-datastore          1.7.0    
google-resumable-media          0.3.1    
googleapis-common-protos        1.5.3    
googledatastore                 7.0.1    
proto-google-cloud-datastore-v1 0.90.4   
proto-google-datastore-v1       1.4.0

I tried substituting one for the other but then got errors about mismatched string types. Ideas?

Is this project still alive?

I'm wondering if this project is still alive, as there are no recent commits and the whole init process does not seem to work anymore.

Unable to deploy single service with latest release

I'm getting the following:

amohr@amohr-macbook ~/dev/upvote (master) $ gcloud config set project santaupvote
Updated property [core/project].
amohr@amohr-macbook ~/dev/upvote (master) $  bazel run --incompatible_package_name_is_a_function=false upvote/gae:monolith_binary.deploy -- -q app.yaml
INFO: Invocation ID: 6e604bd2-c82b-4660-a20e-c6fd0afc7209
INFO: Analysed target //upvote/gae:monolith_binary.deploy (1 packages loaded, 23021 targets configured).
INFO: Found 1 target...
Target //upvote/gae:monolith_binary.deploy up-to-date:
  bazel-bin/upvote/gae/monolith_binary_deploy.sh
  bazel-bin/upvote/gae/monolith_binary.deploy
INFO: Elapsed time: 2.709s, Critical Path: 0.11s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
WARNING: In file [/private/var/folders/9z/50thrw1j52g7v2d7yrl87l940000gn/T/war.fWFCAUlo/__main__/app.yaml]: ('module', 'Field module is deprecated; use service instead.')
WARNING: In file [/private/var/folders/9z/50thrw1j52g7v2d7yrl87l940000gn/T/war.fWFCAUlo/__main__/app.yaml]: ('module', 'Field module is deprecated; use service instead.')
ERROR: (gcloud.app.deploy) [/private/var/folders/9z/50thrw1j52g7v2d7yrl87l940000gn/T/war.fWFCAUlo/__main__/app.yaml] and [/private/var/folders/9z/50thrw1j52g7v2d7yrl87l940000gn/T/war.fWFCAUlo/__main__/app.yaml] are both defining the service id [default]. All <service>.yaml files must have unique service ids.

amohr@amohr-macbook ~/dev/upvote (master) $ bazel version
INFO: Invocation ID: 50765701-a2c8-4260-a150-6541638ba794
Build label: 0.21.0
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Dec 19 12:57:09 2018 (1545224229)
Build timestamp: 1545224229
Build timestamp as int: 1545224229

ideas?

Effort required to upgrade from legacy Python

$ python3 -m flake8 . --count --select=E9,F821,F822,F823 --show-source --statistics

./upvote/shared/constants_test.py:131:14: F821 undefined name 'xrange'
    for i in xrange(1, len(hierarchy)):
             ^
./upvote/shared/time_utils_test.py:31:57: F821 undefined name 'xrange'
        datetime.datetime(2017, 1, 1, 1, 1, s) for s in xrange(3)]
                                                        ^
./upvote/gae/shared/common/test_utils.py:35:56: F821 undefined name 'xrange'
  return [now + datetime.timedelta(seconds=i) for i in xrange(count)]
                                                       ^
./upvote/gae/shared/common/taskqueue_utils_test.py:32:14: F821 undefined name 'xrange'
    for _ in xrange(expected_size):
             ^
./upvote/gae/shared/common/taskqueue_utils_test.py:46:61: F821 undefined name 'xrange'
        taskqueue_utils.CappedDefer(dir, max_size) for _ in xrange(total_size)]
                                                            ^
./upvote/gae/shared/common/taskqueue_utils_test.py:65:16: F821 undefined name 'xrange'
      for i in xrange(6):
               ^
./upvote/gae/shared/common/taskqueue_utils_test.py:81:16: F821 undefined name 'xrange'
      for i in xrange(6):
               ^
./upvote/gae/shared/common/taskqueue_utils_test.py:101:16: F821 undefined name 'xrange'
      for i in xrange(6):
               ^
./upvote/gae/shared/common/taskqueue_utils_test.py:130:16: F821 undefined name 'xrange'
      for _ in xrange(6):
               ^
./upvote/gae/shared/common/taskqueue_utils_test.py:157:16: F821 undefined name 'xrange'
      for _ in xrange(6):
               ^
./upvote/gae/shared/common/taskqueue_utils_test.py:212:14: F821 undefined name 'xrange'
    for _ in xrange(10):
             ^
./upvote/gae/shared/common/query_utils_test.py:43:43: F821 undefined name 'xrange'
  return [CreateEntity(**kwargs) for _ in xrange(count)]
                                          ^
./upvote/gae/shared/common/query_utils_test.py:133:25: F821 undefined name 'xrange'
    for entity_count in xrange(50):
                        ^
./upvote/gae/shared/common/query_utils_test.py:155:14: F821 undefined name 'xrange'
    for _ in xrange(4):
             ^
./upvote/gae/shared/common/query_utils_test.py:169:14: F821 undefined name 'xrange'
    for _ in xrange(2):
             ^
./upvote/gae/shared/common/query_utils_test.py:182:14: F821 undefined name 'xrange'
    for _ in xrange(4):
             ^
./upvote/gae/shared/common/query_utils_test.py:194:14: F821 undefined name 'xrange'
    for _ in xrange(2):
             ^
./upvote/gae/shared/common/query_utils_test.py:208:14: F821 undefined name 'xrange'
    for _ in xrange(2):
             ^
./upvote/gae/shared/common/query_utils_test.py:222:14: F821 undefined name 'xrange'
    for _ in xrange(3):
             ^
./upvote/gae/shared/common/query_utils_test.py:238:14: F821 undefined name 'xrange'
    for _ in xrange(2):
             ^
./upvote/gae/shared/common/intermodule_test.py:90:55: F821 undefined name 'xrange'
    redirect_urls = ['https://foo%d.com' % i for i in xrange(4)]
                                                      ^
./upvote/gae/shared/common/query_utils.py:96:14: F821 undefined name 'reduce'
    result = reduce(  # pylint: disable=bad-builtin
             ^
./upvote/gae/shared/common/query_utils.py:102:12: F821 undefined name 'reduce'
  result = reduce(  # pylint: disable=bad-builtin
           ^
./upvote/gae/datastore/test_utils.py:39:49: F821 undefined name 'xrange'
  return ''.join(random.choice(source) for _ in xrange(length))
                                                ^
./upvote/gae/datastore/test_utils.py:68:37: F821 undefined name 'xrange'
  return [RandomLetters(8) for _ in xrange(size)]
                                    ^
./upvote/gae/datastore/test_utils.py:72:32: F821 undefined name 'xrange'
  return [RandomInt() for _ in xrange(size)]
                               ^
./upvote/gae/datastore/test_utils.py:84:34: F821 undefined name 'xrange'
  return [RandomEmail() for _ in xrange(count)]
                                 ^
./upvote/gae/datastore/test_utils.py:127:69: F821 undefined name 'xrange'
        random_value = [random.choice(list(prop._choices)) for _ in xrange(3)]
                                                                    ^
./upvote/gae/datastore/test_utils.py:131:47: F821 undefined name 'xrange'
        random_value = [value_func() for _ in xrange(3)]
                                              ^
./upvote/gae/datastore/test_utils.py:143:63: F821 undefined name 'xrange'
  return [RandomDatastoreEntity(model_cls, **kwargs) for _ in xrange(count)]
                                                              ^
./upvote/gae/datastore/test_utils.py:228:52: F821 undefined name 'xrange'
  return [CreateBit9Certificate(**kwargs) for _ in xrange(count)]
                                                   ^
./upvote/gae/datastore/test_utils.py:247:55: F821 undefined name 'xrange'
      'bundle_id': '.'.join(RandomLetters(3) for _ in xrange(3)),
                                                      ^
./upvote/gae/datastore/test_utils.py:248:52: F821 undefined name 'xrange'
      'version': '.'.join(RandomDigits(1) for _ in xrange(3)),
                                                   ^
./upvote/gae/datastore/test_utils.py:249:58: F821 undefined name 'xrange'
      'short_version': '.'.join(RandomDigits(1) for _ in xrange(3)),
                                                         ^
./upvote/gae/datastore/test_utils.py:278:47: F821 undefined name 'xrange'
  return [CreateBit9Binary(**kwargs) for _ in xrange(count)]
                                              ^
./upvote/gae/datastore/test_utils.py:283:51: F821 undefined name 'xrange'
  return [CreateSantaBlockable(**kwargs) for _ in xrange(count)]
                                                  ^
./upvote/gae/datastore/test_utils.py:324:16: F821 undefined name 'xrange'
      for _ in xrange(count)]
               ^
./upvote/gae/datastore/test_utils.py:387:43: F821 undefined name 'xrange'
  return [CreateEvent(blockable) for _ in xrange(event_count)]
                                          ^
./upvote/gae/datastore/test_utils.py:391:48: F821 undefined name 'xrange'
  return [CreateSantaEvent(blockable) for _ in xrange(event_count)]
                                               ^
./upvote/gae/datastore/test_utils.py:395:47: F821 undefined name 'xrange'
  return [CreateBit9Event(blockable) for _ in xrange(event_count)]
                                              ^
./upvote/gae/datastore/test_utils.py:431:41: F821 undefined name 'xrange'
  return [CreateUser(**kwargs) for _ in xrange(user_count)]
                                        ^
./upvote/gae/datastore/test_utils.py:486:46: F821 undefined name 'xrange'
  return [CreateSantaHost(**kwargs) for _ in xrange(count)]
                                             ^
./upvote/gae/datastore/test_utils.py:491:45: F821 undefined name 'xrange'
  return [CreateBit9Host(**kwargs) for _ in xrange(count)]
                                            ^
./upvote/gae/datastore/test_utils.py:567:60: F821 undefined name 'xrange'
  return [CreateBit9Rule(blockable_key, **kwargs) for _ in xrange(count)]
                                                           ^
./upvote/gae/datastore/utils_test.py:550:38: F821 undefined name 'xrange'
    futures = [ndb.Future() for _ in xrange(3)]
                                     ^
./upvote/gae/datastore/utils_test.py:599:38: F821 undefined name 'xrange'
    futures = [ndb.Future() for _ in xrange(3)]
                                     ^
./upvote/gae/modules/upvote_app/lib/voting_test.py:286:14: F821 undefined name 'xrange'
    for _ in xrange(10):
             ^
./upvote/gae/modules/upvote_app/lib/voting_test.py:573:42: F821 undefined name 'xrange'
        test_utils.CreateUser() for _ in xrange(self.local_threshold - 1)]
                                         ^
./upvote/gae/modules/upvote_app/lib/voting_test.py:603:42: F821 undefined name 'xrange'
        test_utils.CreateUser() for _ in xrange(self.local_threshold)]
                                         ^
./upvote/gae/modules/upvote_app/lib/voting_test.py:856:14: F821 undefined name 'xrange'
    for _ in xrange(4):
             ^
./upvote/gae/modules/upvote_app/api/handlers/blockables_test.py:495:14: F821 undefined name 'xrange'
    for i in xrange(expected):
             ^
./upvote/gae/modules/upvote_app/api/handlers/constants_test.py:37:25: F821 undefined name 'unicode'
    unicode_elems = map(unicode, common_constants.USER_ROLE.SET_ALL)
                        ^
./upvote/gae/modules/upvote_app/cron/monitoring.py:21:71: F821 undefined name 'long'
rows_to_persist = monitoring.Metric(metrics.BIGQUERY.ROWS_TO_PERSIST, long)
                                                                      ^
./upvote/gae/modules/upvote_app/cron/monitoring.py:22:69: F821 undefined name 'long'
rows_to_stream = monitoring.Metric(metrics.BIGQUERY.ROWS_TO_STREAM, long)
                                                                    ^
./upvote/gae/modules/upvote_app/cron/export_test.py:63:14: F821 undefined name 'xrange'
    for i in xrange(entity_count):
             ^
./upvote/gae/modules/upvote_app/cron/export_test.py:91:14: F821 undefined name 'xrange'
    for i in xrange(entity_count):
             ^
./upvote/gae/modules/upvote_app/cron/export_test.py:242:14: F821 undefined name 'xrange'
    for i in xrange(count):
             ^
./upvote/gae/modules/upvote_app/cron/roles_test.py:291:42: F821 undefined name 'xrange'
        test_utils.CreateUser() for _ in xrange(roles.BATCH_SIZE - 1)]
                                         ^
./upvote/gae/modules/upvote_app/cron/roles_test.py:315:42: F821 undefined name 'xrange'
        test_utils.CreateUser() for _ in xrange(roles.BATCH_SIZE + 1)]
                                         ^
./upvote/gae/modules/santa_api/sync_test.py:1297:47: E999 SyntaxError: invalid token
        rule_sync_dt=datetime.datetime(2001, 01, 01, 0, 0, 0))
                                              ^
./upvote/gae/modules/bit9_api/change_set.py:56:16: F821 undefined name 'xrange'
      for _ in xrange(1 + max_retries):
               ^
./upvote/gae/modules/bit9_api/monitoring.py:21:69: F821 undefined name 'long'
events_to_pull = monitoring.Metric(metrics.BIT9_API.EVENTS_TO_PULL, long)
                                                                    ^
./upvote/gae/modules/bit9_api/monitoring.py:24:41: F821 undefined name 'long'
    metrics.BIT9_API.EVENTS_TO_PROCESS, long)
                                        ^
./upvote/gae/modules/bit9_api/monitoring.py:26:71: F821 undefined name 'long'
pending_changes = monitoring.Metric(metrics.BIT9_API.PENDING_CHANGES, long)
                                                                      ^
./upvote/gae/modules/bit9_api/sync_test.py:105:24: F821 undefined name 'xrange'
        for host_id in xrange(host_count)]
                       ^
./upvote/gae/modules/bit9_api/sync_test.py:109:16: F821 undefined name 'xrange'
      for _ in xrange(events_per_host):
               ^
./upvote/gae/modules/bit9_api/sync_test.py:172:22: F821 undefined name 'xrange'
    for batch_num in xrange(batch_count):
                     ^
./upvote/gae/modules/bit9_api/sync_test.py:175:28: F821 undefined name 'xrange'
          for event_num in xrange(events_per_batch)])
                           ^
./upvote/gae/modules/bit9_api/sync_test.py:363:18: F821 undefined name 'xrange'
        for _ in xrange(4)]
                 ^
./upvote/gae/modules/bit9_api/sync_test.py:376:18: F821 undefined name 'xrange'
        for _ in xrange(4)]
                 ^
./upvote/gae/modules/bit9_api/cron_test.py:97:25: F821 undefined name 'obj'
        requests.append(obj.to_raw_dict())
                        ^
./upvote/gae/modules/bit9_api/cron_test.py:188:14: F821 undefined name 'xrange'
    for i in xrange(1, cron._PULL_MAX_QUEUE_SIZE + 20):
             ^
./upvote/gae/modules/bit9_api/cron_test.py:204:14: F821 undefined name 'xrange'
    for _ in xrange(expected_length):
             ^
./upvote/gae/modules/bit9_api/cron_test.py:221:14: F821 undefined name 'xrange'
    for i in xrange(1, cron._DISPATCH_MAX_QUEUE_SIZE + 20):
             ^
./upvote/gae/modules/bit9_api/api/model.py:117:18: F821 undefined name 'basestring'
  _PYTHON_TYPE = basestring
                 ^
./upvote/gae/modules/bit9_api/api/utils.py:31:34: F821 undefined name 'xrange'
_DELETE_CHARS = ''.join(map(chr, xrange(128, 256)))
                                 ^
./upvote/gae/modules/bit9_api/api/utils.py:35:51: F821 undefined name 'unicode'
  return to_ascii_str(value) if isinstance(value, unicode) else value
                                                  ^
./upvote/gae/modules/bit9_api/api/utils.py:56:24: F821 undefined name 'unicode'
  if not isinstance(s, unicode) and not isinstance(s, str):
                       ^
./common/datastore_locks.py:201:46: F821 undefined name 'xrange'
    intervals = [_INITIAL_DELAY + i for i in xrange(max_acquire_attempts - 1)]
                                             ^
./common/ng_template.py:47:12: E999 SyntaxError: invalid syntax
  print '\n'.join(result)
           ^
./common/cloud_kms/kms_ndb_test.py:32:50: F821 undefined name 'unicode'
      side_effect=lambda data, a2, a3, **kwargs: unicode(data))
                                                 ^
2     E999 SyntaxError: invalid syntax
79    F821 undefined name 'unicode'
81

no such package '@com_google_javascript_closure_compiler//'

Environment (Inside Docker)
OS: Ubuntu 16.04
Bazel: 0.23.2
Python: 2.7.12

Problem
Trying to deploy using init_project.sh and the following errors occur.

ERROR: /root/.cache/bazel/_bazel_root/2883744d33ed0db99e71fd4612c17bed/external/com_google_closure_stylesheets/BUILD:7:1: no such package '@com_google_javascript_closure_compiler//': java.io.IOException: Error downloading [https://mirror.bazel.build/repo1.maven.org/maven2/com/google/javascript/closure-compiler-unshaded/v20180805/closure-compiler-unshaded-v20180805.jar, http://repo1.maven.org/maven2/com/google/javascript/closure-compiler-unshaded/v20180805/closure-compiler-unshaded-v20180805.jar] to /root/.cache/bazel/_bazel_root/2883744d33ed0db99e71fd4612c17bed/external/com_google_javascript_closure_compiler/closure-compiler-unshaded-v20180805.jar: All mirrors are down: [GET returned 404 Not Found, GET returned 501 HTTPS Required] and referenced by '@com_google_closure_stylesheets//:com_google_closure_stylesheets' ERROR: Analysis of target '//upvote/gae:monolith_binary.deploy' failed; build aborted: no such package '@com_google_javascript_closure_compiler//': java.io.IOException: Error downloading [https://mirror.bazel.build/repo1.maven.org/maven2/com/google/javascript/closure-compiler-unshaded/v20180805/closure-compiler-unshaded-v20180805.jar, http://repo1.maven.org/maven2/com/google/javascript/closure-compiler-unshaded/v20180805/closure-compiler-unshaded-v20180805.jar] to /root/.cache/bazel/_bazel_root/2883744d33ed0db99e71fd4612c17bed/external/com_google_javascript_closure_compiler/closure-compiler-unshaded-v20180805.jar: All mirrors are down: [GET returned 404 Not Found, GET returned 501 HTTPS Required] INFO: Elapsed time: 16.994s INFO: 0 processes. FAILED: Build did NOT complete successfully (18 packages loaded, 849 targets configured) FAILED: Build did NOT complete successfully (18 packages loaded, 849 targets configured) Fetching @chardet_git; Cloning tags/3.0.2 of https://github.com/chardet/chardet.git 4s Fetching @com_google_javascript_closure_compiler; fetching

bazel issue when trying to deploy to app engine

Please check the logs -

+ echo Configuring App Engine...
Configuring App Engine...
+ echo Deploying temporary default version to App Engine...
Deploying temporary default version to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy -- upvote-project app.yaml
added 4 packages from 2 contributors and audited 4 packages in 3.854s
found 0 vulnerabilities

yarn install v1.12.1
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 0.48s.
added 4 packages from 2 contributors and audited 4 packages in 1.513s
found 0 vulnerabilities

added 4 packages from 2 contributors and audited 4 packages in 1.441s
found 0 vulnerabilities

external/com_google_protobuf/src/google/protobuf/generated_message_reflection.cc(2424): warning C4506: no definition for inline function 'google::protobuf::Message *google::protobuf::internal::GenericTypeHandler<google::protobuf::Message>::NewFromPrototype(const GenericType *,google::protobuf::Arena *)'
        with
        [
            GenericType=google::protobuf::Message
        ]
external/com_google_protobuf/src/google/protobuf/generated_message_reflection.cc(2424): warning C4506: no definition for inline function 'google::protobuf::Arena *google::protobuf::internal::GenericTypeHandler<google::protobuf::Message>::GetArena(GenericType *)'
        with
        [
            GenericType=google::protobuf::Message
        ]
external/com_google_protobuf/src/google/protobuf/generated_message_reflection.cc(2424): warning C4506: no definition for inline function 'void *google::protobuf::internal::GenericTypeHandler<google::protobuf::Message>::GetMaybeArenaPointer(GenericType *)'
        with
        [
            GenericType=google::protobuf::Message
        ]
external/com_google_protobuf/src/google/protobuf/map_field.cc(465): warning C4506: no definition for inline function 'google::protobuf::Message *google::protobuf::internal::GenericTypeHandler<google::protobuf::Message>::NewFromPrototype(const GenericType *,google::protobuf::Arena *)'
        with
        [
            GenericType=google::protobuf::Message
        ]
external/com_google_protobuf/src/google/protobuf/map_field.cc(465): warning C4506: no definition for inline function 'google::protobuf::Arena *google::protobuf::internal::GenericTypeHandler<google::protobuf::Message>::GetArena(GenericType *)'
        with
        [
            GenericType=google::protobuf::Message
        ]
external/com_google_protobuf/src/google/protobuf/map_field.cc(465): warning C4506: no definition for inline function 'void *google::protobuf::internal::GenericTypeHandler<google::protobuf::Message>::GetMaybeArenaPointer(GenericType *)'
        with
        [
            GenericType=google::protobuf::Message
        ]
+ echo

+ echo Deploying to App Engine...
Deploying to App Engine...
+ bazel run upvote/gae:monolith_binary.deploy -- upvote-project app.yaml santa_api.yaml
INFO: Invocation ID: bb4c697d-d1cf-48c9-9b5f-9b1cbf5c0006
Loading:
Loading: 0 packages loaded
DEBUG: Rule 'io_bazel_rules_appengine' modified arguments {"shallow_since": "1544654581 -0500"}
DEBUG: Rule 'build_bazel_rules_nodejs' modified arguments {"commit": "6d9a5c9333e5bca58125e9c96b99f3c274bf51e2", "shallow_since": "1543957109 -0800"} and dropped ["tag"]
Analyzing: target //upvote/gae:monolith_binary.deploy (0 packages loaded, 0 targets configured)
DEBUG: Rule 'gcloud_auth_git' modified arguments {"commit": "709953d34c0c93744606a38e84abce557e73dfcb", "shallow_since": "1492459993 -0700"} and dropped ["tag"]
DEBUG: Rule 'pyasn1_git' modified arguments {"shallow_since": "1444500249 +0000"}
DEBUG: Rule 'pyasn1_modules_git' modified arguments {"commit": "75a1d105027e86aa7d2720e6f0fa6349933ad55d", "shallow_since": "1501107647 +0200"} and dropped ["tag"]
DEBUG: Rule 'rsa_git' modified arguments {"shallow_since": "1497086972 +0200"}
DEBUG: Rule 'cachetools_git' modified arguments {"commit": "e27332bc82f4e327aedaec17c9b656ae719322ed", "shallow_since": "1475496371 +0200"} and dropped ["tag"]
DEBUG: Rule 'gcloud_auth_httplib2_git' modified arguments {"shallow_since": "1497375871 -0700"}
DEBUG: Rule 'gcloud_resumable_media_git' modified arguments {"commit": "d296d036453539f03b956f33112baa740d5a723f", "shallow_since": "1500660227 -0700"} and dropped ["tag"]
DEBUG: Rule 'requests_git' modified arguments {"commit": "5091c15bd71ec934423a71ce1b53668cd15db68f", "shallow_since": "1496094855 -0400"} and dropped ["tag"]
DEBUG: Rule 'chardet_git' modified arguments {"commit": "c47f6d79845ec7fe9f3bfd2fc66d47812f571d12", "shallow_since": "1492024800 -0400"} and dropped ["tag"]
DEBUG: Rule 'idna_git' modified arguments {"commit": "0088bfce9c3270e15d8356f0709110c1a9589d35", "shallow_since": "1488857158 -0800"} and dropped ["tag"]
DEBUG: Rule 'urllib3_git' modified arguments {"commit": "8e049799777cdf592ae54ae762d140151766123c", "shallow_since": "1500541470 +0100"} and dropped ["tag"]
DEBUG: Rule 'certifi_git' modified arguments {"commit": "4f35e3529c78ced74040cf5d80bf8ec4aac9a190", "shallow_since": "1492457773 -0400"} and dropped ["tag"]
DEBUG: Rule 'googleapiclient_git' modified arguments {"commit": "f19307d2079e3c3854cac887f059655a66c49896", "shallow_since": "1478045551 -0700"} and dropped ["tag"]
DEBUG: Rule 'oauth2client_git' modified arguments {"shallow_since": "1496765066 -0700"}
DEBUG: Rule 'absl_git' modified arguments {"shallow_since": "1506641175 -0700"}
DEBUG: Rule 'material_steppers' modified arguments {"shallow_since": "1491412933 -0300"}
DEBUG: C:/users/win/_bazel_win/pj2hik4j/external/build_bazel_rules_nodejs/internal/npm_install/npm_install.bzl:159:5:
***********WARNING***********
npm: npm_install will require a package_lock_json attribute in future versions
*****************************
INFO: Analysed target //upvote/gae:monolith_binary.deploy (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Building: no action
WARNING: failed to create one or more convenience symlinks for prefix 'bazel-':
  cannot create symbolic link bazel-bin -> C:/users/win/_bazel_win/pj2hik4j/execroot/__main__/bazel-out/x64_windows-fastbuild/bin:  Cannot create junction (name=C:\users\win\desktop\upvote\upvote\bazel-bin, target=C:\users\win\_bazel_win\pj2hik4j\execroot\__main__\bazel-out\x64_windows-fastbuild\bin): ERROR: src/main/native/windows/file-jni.cc(88): nativeCreateJunction(C:\users\win\desktop\upvote\upvote\bazel-bin, C:\users\win\_bazel_win\pj2hik4j\execroot\__main__\bazel-out\x64_windows-fastbuild\bin): ERROR: src/main/native/windows/file.cc(346): DeviceIoControl(\\?\C:\users\win\desktop\upvote\upvote\bazel-bin): Access is denied.

  cannot create symbolic link bazel-testlogs -> C:/users/win/_bazel_win/pj2hik4j/execroot/__main__/bazel-out/x64_windows-fastbuild/testlogs:  Cannot create junction (name=C:\users\win\desktop\upvote\upvote\bazel-testlogs, target=C:\users\win\_bazel_win\pj2hik4j\execroot\__main__\bazel-out\x64_windows-fastbuild\testlogs): the junction's path got modified unexpectedly
  cannot create symbolic link bazel-genfiles -> C:/users/win/_bazel_win/pj2hik4j/execroot/__main__/bazel-out/x64_windows-fastbuild/genfiles:  Cannot create junction (name=C:\users\win\desktop\upvote\upvote\bazel-genfiles, target=C:\users\win\_bazel_win\pj2hik4j\execroot\__main__\bazel-out\x64_windows-fastbuild\genfiles): the junction's path got modified unexpectedly
  cannot create symbolic link bazel-out -> C:/users/win/_bazel_win/pj2hik4j/execroot/__main__/bazel-out:  Cannot create junction (name=C:\users\win\desktop\upvote\upvote\bazel-out, target=C:\users\win\_bazel_win\pj2hik4j\execroot\__main__\bazel-out): the junction's path got modified unexpectedly
  cannot create symbolic link bazel-upvote -> C:/users/win/_bazel_win/pj2hik4j/execroot/__main__:  Cannot create junction (name=C:\users\win\desktop\upvote\upvote\bazel-upvote, target=C:\users\win\_bazel_win\pj2hik4j\execroot\__main__): the junction's path got modified unexpectedly
[0 / 1] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //upvote/gae:monolith_binary.deploy up-to-date:
  C:/users/win/_bazel_win/pj2hik4j/execroot/__main__/bazel-out/x64_windows-fastbuild/bin/upvote/gae/monolith_binary_deploy.sh
  C:/users/win/_bazel_win/pj2hik4j/execroot/__main__/bazel-out/x64_windows-fastbuild/bin/upvote/gae/monolith_binary.deploy
  C:/users/win/_bazel_win/pj2hik4j/execroot/__main__/bazel-out/x64_windows-fastbuild/bin/upvote/gae/monolith_binary.deploy.exe
INFO: Elapsed time: 35.375s, Critical Path: 0.01s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Running command line: C:/users/win/_bazel_win/pj2hik4j/execroot/__main__/bazel-out/x64_windows-fastbuild/bin/upvote/gae/monolith_binary.deploy.exe upvote-project app.yaml santa_api.yaml
INFO: Build completed successfully, 1 total action
C:\users\win\_bazel_win\pj2hik4j\execroot\__main__\bazel-out\x64_windows-fastbuild\bin\upvote\gae\monolith_binary.deploy: line 34: cd: /tmp/war.WIYuYnvI/__main__: No such file or directory
C:\users\win\_bazel_win\pj2hik4j\execroot\__main__\bazel-out\x64_windows-fastbuild\bin\upvote\gae\monolith_binary.deploy: line 37: /c/users/win/_bazel~1/pj2hik4j/execroot/__main__/bazel-~1/x64_wi~1/bin/upvote/gae/monoli~2.run/../com_google_cloud_sdk/bin/gcloud: No such file or directory

I have updated all the dependencies but still not able to deploy it to app engine.
Any ideas whats wrong here?

Thanks!

google / upvote_py2 Goto Github PK

upvote_py2's Introduction

Upvote

Features

Screenshot

Setup

Docs

Contributing

Contributors

Disclaimer

upvote_py2's People

Contributors

Stargazers

Watchers

Forkers

upvote_py2's Issues

Recommend Projects

Recommend Topics

Recommend Org