google / gae-secure-scaffold-python Goto Github PK

License: Apache License 2.0

JavaScript 8.32% Python 70.42% HTML 5.54% Smarty 15.71%

gae-secure-scaffold-python's Introduction

Secure GAE Scaffold for Python 2

Introduction

Please note: this is not an official Google product.

This scaffold is for users of App Engine's Python 2.7 runtime. For websites deployed to the Python 3 runtime, please see Secure Scaffold for Python 3.

This contains a boilerplate AppEngine application meant to provide a secure base on which to build additional functionality. Structure:

/ - top level directory for common files, e.g. app.yaml
/js - directory for uncompiled Javascript resources.
/src - directory for all source code
/static - directory for static content
/templates - directory for Django/Jinja2 templates your app renders.
/templates/soy - directory for Closure Templates your application uses.

Javascript resources for your application can be written using Closure, and compiled by Google's Closure Compiler (detailed below in the dependencies section).

The scaffold provides the following basic security guarantees by default through a set of base classes found in src/base/handlers.py. These handlers:

Set assorted security headers (Strict-Transport-Security, X-Frame-Options, X-XSS-Protection, X-Content-Type-Options, Content-Security-Policy) with strong default values to help avoid attacks like Cross-Site Scripting (XSS) and Cross-Site Script Inclusion. See _SetCommonResponseHeaders() and SetAjaxResponseHeaders().
Prevent the XSS-prone construction of HTML via string concatenation by forcing the use of a template system (Django/Jinja2 supported). The template systems have non-contextual autoescaping enabled by default. See the render(), render_json() methods in BaseHandler and BaseAjaxHandler. For contextual autoescaping, you should use Closure Templates in strict mode (https://developers.google.com/closure/templates/docs/security).
Test for the presence of headers that guarantee requests to Cron or Task endpoints are made by the AppEngine serving environment or an application administrator. See the dispatch() method in BaseCronHandler and BaseTaskHandler.
Verify XSRF tokens by default on authenticated requests using any verb other that GET, HEAD, or OPTIONS. See the _RequestContainsValidXsrfToken() method for more information.

In addition to the protections above, the scaffold monkey patches assorted APIs that use insecure or dangerous defaults (see src/base/api_fixer.py).

Obviously no framework is perfect, and the flexibility of Python offers many ways for a motivated developer to circumvent the protections offered. Under the assumption that developers are not malicious, using the scaffold should centralize many security mechanisms, provide safe defaults, and structure the code in a way that facilitates security review.

Sample implementations can be found in src/handlers.py. These demonstrate basic functionality, and should be removed / replaced by code specific to your application.

Prerequisites

These instructions have been tested with the following software:

node.js >= 0.8.0
- 0.8.0 is the minimum required to build with Grunt.
git
curl

Dependency Setup

From the root of the repository:

git submodule init
git submodule update
cd closure-compiler - refer to closure-compiler/README.md on how to build the compiler. Feel free to use this GWT-skipping variant: mvn -pl externs/pom.xml,pom-main.xml,pom-main-shaded.xml
cd ../closure-templates && mvn && cd ..
npm install
mkdir $HOME/bin; cd $HOME/bin
npm install grunt-cli
- Alternatively, sudo npm install -g grunt-cli will install system-wide and you may skip the next step.
export PATH=$HOME/bin/node_modules/grunt-cli/bin:$PATH
- It is advisable to add this to login profile scripts (.bashrc, etc.).
Visit https://cloud.google.com/appengine/docs/python/download, and choose the alternative option to "download the original App Engine SDK for Python." Choose the "Linux" platform (even if you use OS X). Unzip the file, such that $HOME/bin/google_appengine/ is populated with the contents of the .zip.

To install dependencies for unit testing:

sudo easy_install pip
sudo pip install unittest2

Scaffold Development

Testing

To run unit tests:

python run_tests.py ~/bin/google_appengine src

Local Development

To run the development appserver locally:

grunt clean
grunt
grunt appengine:run:app

Note that the development appserver will be running on a snapshot of code at the time you run it. If you make changes, you can run the various Grunt tasks in order to propagate them to the local appserver. For instance, grunt copy will refresh the source code (local and third party), static files, and templates. grunt closureSoys and/or grunt closureBuilder will rebuild the templates or your provided Javascript and the updated versions will be written in the output directory.

Deployment

To deploy to AppEngine:

grunt clean
grunt --appid=<appid>
grunt appengine:update:app --appid=<appid>

Specifying --appid= will override any value set in config.json. You may modify the config.json file to avoid having to pass this parameter on every invocation.

Notes

Files in js/ are compiled by the Closure Compiler (if available) and placed in out/static/app.js. Included in this compilation pass is the the output of the closureSoys:js task (intermediate artifacts: out/generated/js/*.js).

Closure Templates that you provide are also compiled using the Python backend, and are available using the constants.CLOSURE template strategy (the default). The generated source code is stored in out/generated/*.py. To use them, pass the callable template as the first argument to render(), and a dictionary containing the template values as the second argument, e.g.:

from generated import helloworld

[...]

self.render(helloworld.helloWorld, { 'name': 'first last' })

The /static and /template directories are replicated in out/, and the files in src/ are rebased into out/ (so src/base/foo.py becomes out/base/foo.py).

gae-secure-scaffold-python's People

Contributors

Stargazers

Watchers

gae-secure-scaffold-python's Issues

Not able to run closurebuilder.py in same directory

Hi there and thanks for the awesome scaffolding. Any idea how to get around this issue? I followed the README but when I run grunt I get this error. I read a bit online here and it seems like you can't have closurebuilder in the same directory as your app. Has anyone ran into this problem and have a straightforward solution? Thank you.

Grunt error below:

Traceback (most recent call last):
File "closure-library/closure/bin/build/closurebuilder.py", line 274, in
main()
File "closure-library/closure/bin/build/closurebuilder.py", line 214, in main
tree = depstree.DepsTree(sources)
File "/Users/adaro/LFL/cbc-frontend/cbc-pledge-frontend/closure-library/closure/bin/build/depstree.py", line 48, in init
provide, [self._provides_map[provide], source])
depstree.MultipleProvideError: Namespace "goog.events.BrowserFeature" provided more than once in sources:
PathSource closure-library/closure/goog/events/browserfeature.js
PathSource js/app.js
FAILED to run command for target: js

xsrf _Compare function is vulnerable to timing attacks

_Compare should be replaced with hmac.compare_digest

https://docs.python.org/2/library/hmac.html

It's preferred to use hmac.compare_digest over hand-rolling
a constant-time comparison function, because it is difficult or impossible to
implement correctly in pure-python.

Fun example: https://bugs.python.org/issue15061#msg162758

Note that it takes different time to create a result of ord() depending
whether it's <=100 or > 100 due to caching of small numbers.

So definitely prefer hmac.compare_digest, if timing attacks are a worry.

Cannot whitelist methods for pickling.

I'm trying to use the ndb deferred library to set up a background task, but this doesn't seem to be supported as I get the error to the effect of Pickling is forbidden.

I've tried various methods, but cannot add the methods to the pickle whitelist to get the setup described in the docs.

What would be even better is to allow pickling of any instance method to make this work without having to do something dirty like create custom deferred tasks in base, but that seems to be tangential to this issue, as the error doesn't appear to have to do with whitelisting.

Why is there a 'Content-Disposition' header added to ajax responses?

See: https://github.com/google/gae-secure-scaffold-python/blob/master/src/base/handlers.py#L325

Windows 10 build question

I build project on Win10，

prolem is below:
Running "closureBuilder:js" (closureBuilder) task
Executing: python closure-library/closure/bin/build/closurebuilder.py -n app --root=closure-library --root=js --root=out/generated/js -o compiled --output_file=out/static/app.js --compiler_jar=closure-compiler/target/closure-compiler-v20160517.jar --compiler_flags="--compilation_level=ADVANCED_OPTIMIZATIONS"

Error: Command failed: python closure-library/closure/bin/build/closurebuilder.py -n app --root=closure-library --root=js --root=out/generated/js -o compiled --output_file=out/static/app.js --compiler_jar=closure-compiler/target/closure-compiler-v20160517.jar --compiler_flags="--compilation_level=ADVANCED_OPTIMIZATIONS"
closure-library/closure/bin/build/closurebuilder.py: Scanning paths...
closure-library/closure/bin/build/closurebuilder.py: 1523 sources scanned.
closure-library/closure/bin/build/closurebuilder.py: Building dependency tree..
Traceback (most recent call last):
File "closure-library/closure/bin/build/closurebuilder.py", line 293, in
main()
File "closure-library/closure/bin/build/closurebuilder.py", line 229, in main
tree = depstree.DepsTree(sources)
File "C:\Users\sheng.han\google\gae-secure-scaffold-python\closure-library\closure\bin\build\depstree.py", line 56, in init
raise NamespaceNotFoundError(require, source)
depstree.NamespaceNotFoundError: Namespace "goog.soy.data.SanitizedUri" never provided. Required in PathSource out\generated\js\soyutils_usegoog.js
FAILED to run command for target: js

pip requirements.txt for all dependencies

Are you able to include a list of all the pip dependencies in one requirements.txt file? For example:

Django==1.11.25
gcloud==0.18.3
google-api-python-client==1.7.11
pylint==1.9.5
unittest2==1.1.0
webapp2==2.5.2
WebOb==1.8.5

These can then all be installed easily in one go using virtualenv and the command:

pip install virtualenv
virtualenv env && source env/bin/activate
pip install -r requirements.txt

reportOnly is removed from csp_policy which leads to unwanted behaviour

self.app.config is sometimes reused between requests.

When reportOnly is removed from the csp_policy here the csp will behave as non-readonly on some consecutive requests.

The reportOnly item should be removed because it's not a valid csp entity. But it should only be removed for the current request. A solution would be to take a copy of the csp_policy.

Only tested locally using dev_appserver

Secure scaffold for Python 3

This is a great project!

It would be great to have something like this for Python3, having a boilerplate AppEngine application meant to provide a secure base on which to build additional functionality.

Vendored third party libraries aren't accounted for in test runner

I was having trouble with the test runner not finding third party libraries I had vendored in. I wanted to make sure the project I was working on could be downloaded by the client and the tests run without much work. I thought I was doing something wrong with the vendoring, but I pointed in the right direction by Jon Parret. He has an example here.

https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/appengine/localtesting/runner.py

Following his lead third party libraries added to appengine_config.py can run within the test runner without needing to change the PythonPath or anything odd like that.

Migrate from frame-src (deprecated) to child-src (replacement)

Reported by @lweichselbaum, https://developer.mozilla.org/en-US/docs/Web/Security/CSP/CSP_policy_directives#frame-src.

Documentation doesn't match links and SDK installation paths

On the fifth step it can be read:

Visit https://developers.google.com/appengine/downloads, copy URL of "Linux/Other Platforms" zip file for current AppEngine SDK. Do this regardless of whether you are on Linux or OS X.

The link captioned as "Linux/Other Platforms" is not there anymore the main CTA inside that page actually points to the Google Cloud SDK which can be confusing.

In addition, installing the Google App Engine SDK from the old .dmg places its main folder in /usr/local and running

bash util.sh -d

fails since the script is looking for the development server in /Users/<USER_NAME>/bin/google_appengine/dev_appserver.py.

Changes to index.yaml occur in out/ not src/

When running in development, the Python source code is run from out, which means that changes made by the development appserver to index.yaml are lost when rebuilding.

Are there best practices for detecting these changes and copying them back to src, or for running in development from src not out?