level12 / keg Goto Github PK
View Code? Open in Web Editor NEWKeg: more than Flask
License: Other
Keg: more than Flask
License: Other
A byte string doesn't not get picked up by the keyring implementation and probably should.
https://github.com/level12/keg/blob/master/keg/keyring.py#L53
Until #32 is settled, Click's upper bound needs to be < 5.0
to prevent a landslide of warnings.
class PayeeFormBaseView(FISBaseView):
blueprint = admin
def post(self, payee):
...
def get(self):
...
class PayeeFormContract(PayeeFormBaseView):
rule('/payee-forms/<int:payee_id>/contract', post=True)
....
The above doesn't work as expected. It ends up not creating the route for PayeeFormContract
.
On 03/23/2015 05:30 PM, Marc Schlaich wrote:
Hey Randy,
in the upcoming pytest-cov 2.0 you can easily create different data files for each tox environment and automatically combine them at the end of the tox run. See here for an example project:
https://github.com/schlamar/pytest-cov/tree/2.0/example-tox-projectUnfortunately, there is no release date yet as I'm pretty busy right now. However, the 2.0 branch should be quite stable, so you could give it a try.
Marc
See: http://pubs.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap08.html
Specifically:
Environment variable names used by the utilities in the Shell and Utilities volume of IEEE Std 1003.1-2001 consist solely of uppercase letters, digits, and the '_' (underscore)
This means all dotted import names (like those used in the tests) must be replaced with an underscore.
Remove code that is now in SecretStorage-Setup.
https://github.com/level12/keg/blob/master/keg/web.py#L154 will interpret an empty string as False
and call self.render()
.
For example... assume template_which_is_empty.html
exists but is empty. This logic will fail.
class View(keg.web.BaseView)
@keg.web.route('/route')
def route_func(self):
return flask.render_template('template_which_is_empty.html')
The application will load once and then load again. Is this expected behavior?
Traceback (most recent call last):
File "C:\Work\Python\app-dist\fis-residuals\fisresid\tests\test_views_admin.py", line 612, in test_merchant_add
resp = self.ta.get(url)
File "C:\work\python\stage_fis\lib\site-packages\webtest\app.py", line 322, in get
expect_errors=expect_errors)
File "C:\work\python\stage_fis\lib\site-packages\flask_webtest.py", line 215, in do_request
response = super(TestApp, self).do_request(*args, **kwargs)
File "C:\work\python\stage_fis\lib\site-packages\webtest\app.py", line 605, in do_request
res = req.get_response(app, catch_exc_info=True)
File "C:\work\python\stage_fis\lib\site-packages\webob\request.py", line 1313, in send
application, catch_exc_info=True)
File "C:\work\python\stage_fis\lib\site-packages\webob\request.py", line 1281, in call_application
app_iter = application(self.environ, start_response)
File "C:\work\python\stage_fis\lib\site-packages\webtest\lint.py", line 198, in lint_app
iterator = application(environ, start_response_wrapper)
File "C:\work\python\stage_fis\lib\site-packages\werkzeug\local.py", line 366, in <lambda>
__call__ = lambda x, *a, **kw: x._get_current_object()(*a, **kw)
File "C:\work\python\stage_fis\lib\site-packages\flask\app.py", line 1836, in __call__
return self.wsgi_app(environ, start_response)
File "C:\work\python\stage_fis\lib\site-packages\flask\app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "C:\work\python\stage_fis\lib\site-packages\flask\app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "C:\work\python\stage_fis\lib\site-packages\flask\app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "C:\work\python\stage_fis\lib\site-packages\flask\app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "C:\work\python\stage_fis\lib\site-packages\flask\app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "C:\work\python\stage_fis\lib\site-packages\flask\app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "C:\work\python\stage_fis\lib\site-packages\flask\app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "C:\work\python\stage_fis\lib\site-packages\flask\views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "c:\work\python\blaze\keg\keg\web.py", line 157, in dispatch_request
response = self.render()
File "c:\work\python\blaze\keg\keg\web.py", line 212, in render
return flask.render_template(self.calc_template_name(), **self.template_args)
File "C:\work\python\stage_fis\lib\site-packages\flask\templating.py", line 128, in render_template
context, ctx.app)
File "C:\work\python\stage_fis\lib\site-packages\flask\templating.py", line 110, in _render
rv = template.render(context)
File "C:\work\python\stage_fis\lib\site-packages\jinja2\environment.py", line 969, in render
return self.environment.handle_exception(exc_info, True)
File "C:\work\python\stage_fis\lib\site-packages\jinja2\environment.py", line 742, in handle_exception
reraise(exc_type, exc_value, tb)
File "C:\Work\Python\app-dist\fis-residuals\fisresid\templates\admin\payee_form_merchants.html", line 2, in top-level template code
{% assets_include %}
File "c:\work\python\blaze\keg\keg\templating.py", line 55, in _include_support
ctx.assets.load_related(template_name)
File "c:\work\python\blaze\keg\keg\assets.py", line 40, in load_related
.format(template_name))
AssetException: Could not find related assets for template: admin/payee_form_merchants.html
================ 65 tests deselected by '-ktest_merchant_add' =================
1 failed, 65 deselected, 1 warnings in 2.26 seconds
The SQL in db.dialects for clearing and prepping a postgresql database doesn't work when the username has a dash, e.g. "racebetter-beta".
There are some test fails/errors in Windows for the keg tests. At least one is path related. I need to determine if others are related to the config file I copied from the template, or if something else is happening.
I init keg.db.db
in keg.db
which means it's really hard to customize KegSQLAlchemy/SQLAlchemy
classes by subclassing and overriding. This proved especially true when trying to setup custom json serialization methods for a customer project.
I believe keg.db.db
would be better served being a local proxy like flask.request
and redirecting that to the real instance on the app. Then the app would need to have a setup method for that class.
Travis doesn't support it. However, there was some discussion on distutils list a while back where someone was asking for help in writing a go script to spin up a windows VM on Azure for testing.
Click does not recommend this, as it causes subtle bugs. Click 5.0 and above issue a landslide of warnings when it detects this.
SETTING = '${ something }$' # the key should be "something" not " something "
/home/rsyring/.virtualenvs/racebetter/lib/python3.5/site-packages/keg/db/init.py:4: ExtDeprecationWarning: Importing flask.ext.sqlalchemy is deprecated, use flask_sqlalchemy instead.
from flask.ext.sqlalchemy import SQLAlchemy
There are many generic ways to inject code before a response is generated, but I can't find any way to insert a response "middleware" (i.e. a way to modify responses for a certain class hierarchy).
This is what I did to get an acceptable solution of managing debug from the CLI:
import os
import pathlib
import click
from flask.cli import FlaskGroup, pass_script_info
import app
@click.group(cls=FlaskGroup, create_app=app.cli_create)
@click.option('--debug/--no-debug', default=False)
@pass_script_info
def main(info, debug):
# This should probably look at FLASK_DEBUG before overriding it when the
# debug flag is not passed to the CLI
os.environ['FLASK_DEBUG'] = '1' if debug else '0'
You can find the issues with setting up debugging by reading this code: https://github.com/pallets/flask/blob/master/flask/cli.py#L550-L556 and ultimate https://github.com/pallets/flask/blob/master/flask/helpers.py#L59-L63.
Basically, the only way to kick the debugger on is to set the FLASK_DEBUG
environment variable. This main will run prior to the run
command, so it would be possible to write a simple function to parse/load the configuration here and then set FLASK_DEBUG
based on that. This seemed like much less hassle and "worked" well enough for me.
Invocation would be app --debug run
to turn on the debugger.
explicit is better than implicit
Randy Syring [4:52 PM]
Wierd...
climate develop db init --clear-first
creates the stage schema...climate develop db init
doesn't....
[4:52]
@tonto: yes, its is a bit unintuative
[4:54]
The reason is that the code to create the schemas only gets called for the "prep_empty" event: https://github.com/level12/keg/blob/master/keg/db/dialect_ops.py#L65
GitHub
level12/keg
keg - Keg: more than Flask
Nick Zaccardi [4:55 PM]
I see that.
Randy Syring [4:55 PM]
Which only gets called on db_clear()
: https://github.com/level12/keg/blob/master/keg/db/__init__.py#L109
GitHub
level12/keg
keg - Keg: more than Flask
Nick Zaccardi [4:55 PM]
It should probably get called both, times, yea?
Randy Syring [4:56 PM]
The logic could be cleaned up, yes. But, it's not that simple.
[4:58]
My assumption has always been that the DB starts off ready to create objects as needed, i.e. the database and schemas are already in place. So, it's kind of due to the way that we clear the PostgreSQL schemas of their objects (by dropping the schemas) that we even need to worry about this.
[4:58]
That is why it's called prep_empty()
and tied to the clear step.
Nick Zaccardi [4:58 PM]
What about when we first create the database?
[4:59]
I suppose my point is, what is the correct way to create a brand new environment?
Randy Syring [4:59 PM]
in the past, i've always assumed that initial creation of the DB (by the developer) would leave the DB in such a state that all objects could be created by the application.
[4:59]
Alembic?
[5:00]
default DB state + Alembic run should make it ready?
Nick Zaccardi [5:00 PM]
When I create the database it doesn't create all the schemas, it only creates one.
Randy Syring [5:00 PM]
yes, my assumption breaks down in those environments.
Nick Zaccardi [5:00 PM]
So what I am hearing you say is, the provision step should create it.
[5:01]
That is fine, I was falsely under the impression that db init
did that.
Randy Syring [5:01 PM]
That seems reasonable to me...but I could also see adjusting Keg to be smart about creating schemas that don't exist that it knows it needs.
Nick Zaccardi [5:01 PM]
Right.
Randy Syring [5:01 PM]
In fact, that would probably be better...I'm only explaining why it works the way it works now, not saying that is the best way.
https://www.reddit.com/r/Python/comments/3nctlm/what_python_tools_should_i_be_using_on_every/cvn0ybf
parser.add_argument('-v', '--verbose', action='count', default=0)
parser.add_argument('-q', '--quiet', action='count', default=0)
logging_level = logging.WARN + 10*args.quiet - 10*args.verbose
# script -vv -> DEBUG
# script -v -> INFO
# script -> WARNING
# script -q -> ERROR
# script -qq -> CRITICAL
# script -qqq -> no logging at all
We used to have to use SecretStorage-Setup a package I wrote to link in some system libraries to virtualenvs.
However, with the latest python packages available, it seems like there are just two packages to be installed on an Ubuntu system (I tested 14.04) to get the keyring to work using the dbus secretstorage subsystem.
pip install
edSo, for any Keg application that wants to use Keyring substitution on Linux, just pip install dbus-python
.
No one is monitoring that channel, we should remove the badge.
There are always database changes and alemibic integrates well with SQLAlchemy. Alas, integrate Alembic right into the system.
Ported from Bitbucket
When the next Flask comes out, this function is defined on Flask and won't be needed on KegApp.
The name of an applications configuration file is not always obvious. Consider creating docs to help explain what that should be.
It would be really helpful if <app> develop run
would auto reload when a change is made in a file.
Maybe this already exists?
The config variables DEVELOPER_USERNAME, DEVELOPER_EMAIL, and DEVELOPER_PASSWORD are defaults that will be needed in the config for super-user setup. At some point, we should connect KEG_EMAIL_OVERRIDE_TO to DEVELOPER_EMAIL.
Pull over the issues first.
File "c:\work\python\blaze\keg\keg\app.py", line 89, in init
self.init_keyring()
File "c:\work\python\blaze\keg\keg\app.py", line 119, in init_keyring
self.keyring_manager.substitute(self.config)
File "c:\work\python\blaze\keg\keg\keyring.py", line 70, in substitute
keyring_value = getpass.getpass('Enter value for "{0}": '.format(keyring_key))
File "C:\PYTHON\PYTHON27_32\Lib\getpass.py", line 95, in win_getpass
msvcrt.putch(c)
TypeError: must be char, not unicode
# defined here so it can be pickled
class CustomType(object):
def __init__(self):
self._count = 0
def count(self):
self._count += 1
return self._count
class SessionView(BaseView):
blueprint = testing_blueprint
url = '/test-session'
@route(get=True)
def custom(self):
if 'custom_type' not in flask.session:
flask.session['custom_type'] = CustomType()
return '{}'.format(flask.session['custom_type'].count())
class TestSession(object):
def test_custom_type(self):
ta1, _ = login()
ta2, _ = login()
assert ta1.get('/test-session/custom').body == '1' # Fails with a value of b'2'
With a vanilla Flask app this is not the case:
import flask
from flask_webtest import TestApp
app = flask.Flask(__name__)
app.config['SECRET_KEY'] = 'test'
# defined here so it can be pickled
class CounterType(object):
def __init__(self, start):
self._count = start
def count(self):
self._count += 1
return self._count
def __repr__(self):
return self._count
@app.route('/count')
def count():
flask.session['hits'] = CounterType(flask.session.get('hits', 0)).count()
return '{}'.format(flask.session['hits'])
class TestSession(object):
def test(self):
counter = lambda ta: ta.get('/count').body
ta_1 = TestApp(app)
assert ta_1.get('/count').body == b'1'
ta_2 = TestApp(app)
assert counter(ta_2) == b'1'
assert counter(ta_2) == b'2'
if __name__ == '__main__':
app.run(debug=True)
Right now, we are assuming a Keg app can run without Blinker. However, since we are kicking off our testing prep with a signal, it's safe to assume Blinker will need to be installed.
https://github.com/level12/keg/blob/master/keg/web.py#L155 can be moved out of the if
so that it auto_populate
will always run.
init_app()
is used used for manager objects that are Globally instantiated but work in the context of multiple flask apps. Since the Logging instance doesn't really do that, each Logging instance is dedicated to a single app instance, all it's init could happen in __init__()
.clear_keg_handlers()
to make sure we don't create duplicate handlers. In essence, this is creating a last-app-loaded-wins situation, which may not be desirable.SQLite requires a flag to be set to enforce FK constraints. It would be nice if Keg could provide a setting to enable this behavior automatically when connected to a SQLite database.
http://stackoverflow.com/a/15542046
@event.listens_for(sa.engine.Engine, 'connect')
def _set_sqlite_pragma(connection, conn_record):
from sqlite3 import Connection as SQLite3Connection
if isinstance(connection, SQLite3Connection):
cursor = connection.cursor()
cursor.execute("PRAGMA foreign_keys=ON;")
cursor.close()
The default profile is currently controlled from both config files and the _DEFAULT_PROFILE environment variable.
But, when we run tests, in keg.testing, ContextManager
hard-codes the profile name as 'TestProfile.' Furthermore, CLIBase
and Click's CliRunner
create the app without any explicit testing profile. So, in reality, when doing CLI Tests locally, the DevProfile is likely being used.
Need to make this more consistent.
The whole point of having an init()
is to have a method that is safe to override without having to worry about passing parameters up the stack or call super()
.
The whole point of having the init() happen separate from the app setup was to facilitate being able to use the app as a decorator. When using a factory setup method, you can't get at the app instance to use it as a decorator, but Flask requires the app instance to use as a decorator. So when the app gets bigger and you want to move some of your stuff around, the app setup just gets ugly IMO.
Options:
init()
to something else like start()
init
name isn't so helpful and we should have a "on_ready() contract" where the name would be more indicative of what we really are trying to accomplish.I did the following for RaceBetter:
import logging
from logging.config import dictConfig
from logging.handlers import RotatingFileHandler
import os.path as osp
import keg.logging
from pythonjsonlogger import jsonlogger
class Logging(keg.logging.Logging):
def dict_config(self):
log_config = self.config.get('LOGGING_CONFIG')
if log_config:
dictConfig(log_config)
def init_app(self):
super().init_app()
self.dict_config()
self.init_json_log()
def init_json_log(self):
log_fname = self.log_fname.replace('.', '_json.')
log_fpath = osp.join(self.log_dpath, log_fname)
log_max = self.config['KEG_LOG_MAX_BYTES']
log_backups = self.config['KEG_LOG_MAX_BACKUPS']
file_handler = RotatingFileHandler(log_fpath, maxBytes=log_max, backupCount=log_backups,
encoding='utf-8')
file_handler.setLevel(logging.INFO)
include_fields = '%(asctime) %(pathname) %(funcName) %(lineno) %(message) %(levelname)' \
' %(name)s %(process) %(processName) %(message)'
formatter = jsonlogger.JsonFormatter(include_fields)
file_handler.setFormatter(formatter)
for logger in self.loggers:
logger.addHandler(file_handler)
# Make the handler easier to get to by other code for custom logger/handler setups like
# we did for the heartbeat CLI command.
self.json_file_handler = file_handler
Seems like it could have been easier. I also had to set KEG_LOG_MAX_BYTES
to a really high level to turn off file rotation by Python so the system logrotate could handle it.
self.engine.echo = True
should be a config setting for Keg for debugging and testing.
It would be nice, if debug=False
that the output would go to the log file and if testing=True
that it would go out to the screen.
"""
Goals:
1. get pg_restore to run with ZERO errors/warnings unless those errors/warnings actually
matter.
2. The restored DB should match the backed up DB exactly except for ownership.
Problems identified:
* delete DB & recreate runs into issues because of active connections to the DB which can only
be killed by a postgresql superuser (which the devs running ansible shouldn't be).
* clear_db() which drops the schema and recreates it is an option BUT only if the user owns the
schema being dropped. That is a solvable problem, but becomes more complicated when you
have a production schema owned by "shentel" which is what you would want but then you need the
beta user to restore. Could probably solve through the use of group roles if desired but that
might limit the ability to separate someone who has access to beta from keeping out of
production.
* using -c in the restore command would clear existing objects (which is great) but might leave
extras in the current DB that were not being restored. That may or not be an issue for you
but my goal was to have the db.
* Using pg_restore without specifing the schema (-n public) results in errors relating to
ownership of plpgsql extension. See:
* http://dba.stackexchange.com/questions/84798/how-to-make-pg-dump-skip-extension
* http://www.postgresql.org/message-id/[email protected] (bug rpt)
Known issues:
* This method of restore results in no ownership being persisted. All objects are owned by the
current db user, the one running the restore. This works well for us, but could be a problem
in an environment which requires more fine grain ownership control.
* It's possible there are some other DB objects (like types) which, if existing, aren't getting
delete yet, and would cause problems on restore. In that case, write a new sub-function for
clear_all() to handle those types and delete them.
* Our method of clearing assumes the restoring user has permission to delete everything.
"""
import os
import subprocess
from blazeweb.globals import settings
from shentel.model.orm import db
url = db.engine.url
execute = db.engine.execute
def get_table_list_from_db():
"""
return a list of table names from the current
databases public schema
"""
sql = "select table_name from information_schema.tables "\
"where table_schema='public'"
return [name for (name, ) in execute(sql)]
def get_seq_list_from_db():
"""return a list of the sequence names from the current
databases public schema
"""
sql = "select sequence_name from information_schema.sequences " \
"where sequence_schema='public'"
return [name for (name, ) in execute(sql)]
def get_type_list_from_db():
"""return a list of the sequence names from the current
databases public schema
"""
sql = """
SELECT t.typname as type
FROM pg_type t
LEFT JOIN pg_catalog.pg_namespace n
ON n.oid = t.typnamespace
WHERE
( t.typrelid = 0 OR
(
SELECT c.relkind = 'c'
FROM pg_catalog.pg_class c
WHERE c.oid = t.typrelid
)
)
AND NOT EXISTS (
SELECT 1
FROM pg_catalog.pg_type el
WHERE el.oid = t.typelem
AND el.typarray = t.oid
)
AND n.nspname = 'public'
"""
return [name for (name, ) in execute(sql)]
def drop_all():
for table in get_table_list_from_db():
try:
execute("DROP TABLE %s CASCADE" % table)
except Exception as e:
print e
for seq in get_seq_list_from_db():
try:
execute("DROP SEQUENCE %s CASCADE" % table)
except Exception as e:
print e
for dbtype in get_type_list_from_db():
try:
execute("DROP TYPE %s CASCADE" % dbtype)
except Exception as e:
print e
def get_env():
env = os.environ.copy()
env['PGPASSWORD'] = url.password
return env
def action_010_cleardb():
drop_all()
def action_030_restore_db():
subprocess.check_call(['pg_restore', '-h', url.host, '-U', url.username, '--no-owner', '-d',
url.database, '-n', 'public', settings.restore_fpath], env=get_env())
When using KegApp.command, which references a KegGroup instance, the command being decorated should be wrapped in some kind of error handler by default.
Would like to have a decorator API that matches what Flask allows.
@classmethod
def group(cls, *args, **kwargs):
return cls.cli_group.group(*args, **kwargs)
@SomeApp.group(help='Commands for working with...')
def agroup():
pass
@agroup.command()
def acommand():
pass
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.