Git Product home page Git Product logo

flask-profiler's Introduction

Flask-profiler

version: 1.8 Build Status

Flask-profiler measures endpoints defined in your flask application; and provides you fine-grained report through a web interface.

It gives answers to these questions:

  • Where are the bottlenecks in my application?
  • Which endpoints are the slowest in my application?
  • Which are the most frequently called endpoints?
  • What causes my slow endpoints? In which context, with what args and kwargs are they slow?
  • How much time did a specific request take?

In short, if you are curious about what your endpoints are doing and what requests they are receiving, give a try to flask-profiler.

With flask-profiler's web interface, you can monitor all your endpoints' performance and investigate endpoints and received requests by drilling down through filters.

Screenshots

Dashboard view displays a summary.

Alt text

You can create filters to investigate certain type requests.

Alt text

Alt text

You can see all the details of a request. Alt text

Quick Start

It is easy to understand flask-profiler going through an example. Let's dive in.

Install flask-profiler by pip.

pip install flask_profiler

Edit your code where you are creating Flask app.

# your app.py
from flask import Flask
import flask_profiler

app = Flask(__name__)
app.config["DEBUG"] = True

# You need to declare necessary configuration to initialize
# flask-profiler as follows:
app.config["flask_profiler"] = {
    "enabled": app.config["DEBUG"],
    "storage": {
        "engine": "sqlite"
    },
    "basicAuth":{
        "enabled": True,
        "username": "admin",
        "password": "admin"
    },
    "ignore": [
	    "^/static/.*"
	]
}


@app.route('/product/<id>', methods=['GET'])
def getProduct(id):
    return "product id is " + str(id)


@app.route('/product/<id>', methods=['PUT'])
def updateProduct(id):
    return "product {} is being updated".format(id)


@app.route('/products', methods=['GET'])
def listProducts():
    return "suppose I send you product list..."

@app.route('/static/something/', methods=['GET'])
def staticSomething():
    return "this should not be tracked..."

# In order to active flask-profiler, you have to pass flask
# app as an argument to flask-profiler.
# All the endpoints declared so far will be tracked by flask-profiler.
flask_profiler.init_app(app)


# endpoint declarations after flask_profiler.init_app() will be
# hidden to flask_profiler.
@app.route('/doSomething', methods=['GET'])
def doSomething():
    return "flask-profiler will not measure this."


# But in case you want an endpoint to be measured by flask-profiler,
# you can specify this explicitly by using profile() decorator
@app.route('/doSomethingImportant', methods=['GET'])
@flask_profiler.profile()
def doSomethingImportant():
    return "flask-profiler will measure this request."

if __name__ == '__main__':
    app.run(host="127.0.0.1", port=5000)

Now run your app.py

python app.py

And make some requests like:

curl http://127.0.0.1:5000/products
curl http://127.0.0.1:5000/product/123
curl -X PUT -d arg1=val1 http://127.0.0.1:5000/product/123

If everything is okay, Flask-profiler will measure these requests. You can see the result heading to http://127.0.0.1:5000/flask-profiler/ or get results as JSON http://127.0.0.1:5000/flask-profiler/api/measurements?sort=elapsed,desc

If you like to initialize your extensions in other files or use factory apps pattern, you can also create a instance of the Profiler class, this will register all your endpoints once you app run by first time. E.g:

from flask import Flask
from flask_profiler import Profiler

profiler = Profiler()

app = Flask(__name__)

app.config["DEBUG"] = True

# You need to declare necessary configuration to initialize
# flask-profiler as follows:
app.config["flask_profiler"] = {
    "enabled": app.config["DEBUG"],
    "storage": {
        "engine": "sqlite"
    },
    "basicAuth":{
        "enabled": True,
        "username": "admin",
        "password": "admin"
    },
    "ignore": [
        "^/static/.*"
    ]
}

profiler = Profiler()  # You can have this in another module
profiler.init_app(app)
# Or just Profiler(app)

@app.route('/product/<id>', methods=['GET'])
def getProduct(id):
    return "product id is " + str(id)

Using with different database system

You can use flaskprofiler with SqlLite, MongoDB, Postgresql, Mysql or MongoDB database systems. However, it is easy to support other database systems. If you would like to have others, please go to contribution documentation. (It is really easy.)

SQLite

In order to use SQLite, just specify it as the value of storage.engine directive as follows.

app.config["flask_profiler"] = {
    "storage": {
        "engine": "sqlite",
    }
}

Below the other options are listed.

Filter key Description Default
storage.FILE SQLite database file name flask_profiler.sql
storage.TABLE table name in which profiling data will reside measurements

MongoDB

In order to use MongoDB, just specify it as the value of storage.engine directive as follows.

app.config["flask_profiler"] = {
    "storage": {
        "engine": "mongodb",
    }
}

SQLAchemy

In order to use SQLAchemy, just specify it as the value of storage.engine directive as follows. Also first create an empty database with the name "flask_profiler".

app.config["flask_profiler"] = {
    "storage": {
        "engine": "sqlalchemy",
        "db_url": "postgresql://user:pass@localhost:5432/flask_profiler"  # optional, if no db_url specified then sqlite will be used.
    }
}

Custom database engine

Specify engine as string module and class path.

app.config["flask_profiler"] = {
    "storage": {
        "engine": "custom.project.flask_profiler.mysql.MysqlStorage",
        "MYSQL": "mysql://user:password@localhost/flask_profiler"
    }
}

The other options are listed below.

Filter key Description Default
storage.MONGO_URL mongodb connection string mongodb://localhost
storage.DATABASE database name flask_profiler
storage.COLLECTION collection name measurements

Sampling

Control the number of samples taken by flask-profiler

You would want control over how many times should the flask profiler take samples while running in production mode. You can supply a function and control the sampling according to your business logic.

Example 1: Sample 1 in 100 times with random numbers

app.config["flask_profiler"] = {
    "sampling_function": lambda: True if random.sample(list(range(1, 101)), 1) == [42] else False
}

Example 2: Sample for specific users

app.config["flask_profiler"] = {
    "sampling_function": lambda: True if user is 'divyendu' else False
}

If sampling function is not present, all requests will be sampled.

Changing flask-profiler endpoint root

By default, we can access flask-profiler at /flask-profiler

app.config["flask_profiler"] = {
        "endpointRoot": "secret-flask-profiler"
}

Ignored endpoints

Flask-profiler will try to track every endpoint defined so far when init_app() is invoked. If you want to exclude some of the endpoints, you can define matching regex for them as follows:

app.config["flask_profiler"] = {
        "ignore": [
	        "^/static/.*",
	        "/api/users/\w+/password"
        ]
}

Contributing

Contributions are welcome!

Review the Contributing Guidelines for details on how to:

  • Submit issues
  • Add solutions to existing challenges
  • Add new challenges

Authors

License

MIT

flask-profiler's People

Contributors

anandtripathi5 avatar brucedone avatar dependabot[bot] avatar fatihsucu avatar grechut avatar huwentao1 avatar hyunchel avatar iurisilvio avatar josiahtillman avatar kkwezard avatar lee-pai-long avatar marksteve avatar mohsenasm avatar muatik avatar oldani avatar otaufer avatar paradox41 avatar prabhatpathania avatar saturn avatar sonnybaker avatar syy avatar timgates42 avatar vojtechbartos avatar zhipcui avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

flask-profiler's Issues

Initialize flask profiler

It will be nice if like other flask extensions we were able to create an instance of the profiler and either bootstrap it in __init__ or init_app. This help when the app is designed with factories and you initialize extensions in separate modules. I will be glad to make a PR with this.

loading different saved profiler.sql files in the dashboard

Is it possible, maybe with some trickery, to load saved flask_profiler.sql files, and view the results in the dashboard? For some reason, accessing the profile page from my production website does not work. However, SQLite profile files are being created. I just cannot view them. So I'm considering downloading the files locally and trying to view them in a local web-app running the profiler in debug mode. Is something like this possible, or would the files become overwritten?

changing url route of flask-profiler

How can we modify the url route for the flask-profiler? Currently it automatically routes to the main, e.g.

localhost:5000/flask-profiler/

This is fine for debug mode or for production mode where you are not behind a subdomain on a larger server. How can I set the url route for flask-profiler to be behind a subdomain so it works in production mode? E.g. my webapp site is at

www.domainname.com/myapp/

and I would like to access the flask-profiler at

www.domainname.come/myapp/flask-profiler/

Both of these

www.domainname.come/myapp/flask-profiler/
www.domainname.com/flask-profiler/

fail and do not recognize the name. This question is related to #38.

My config for flask-profiler is configured for production mode

    # Setup the profile configuration
    app.config["flask_profiler"] = {
        "enabled": True,
        "storage": {
            "engine": "sqlite"
        },
        "basicAuth": {
            "enabled": True,
            "username": "admin",
            "password": "admin"
        }
    }

Has anyone actually successfully used this in a production environment and not in debug mode?

Importing database dump

The process of dumping database was covered in #32 wherein we take the dump in JSON format.

How about an option to take a file and put it back into the DB i.e. importing of a dump.

Let me know your thoughts on this. Thanks

Optimizing for production profiling like XHGui

Is there a feature where we can limit the number of samples probabilistic and use profile in production with huge load ?

If such a feature is not present, we can create one like :-

This is done in one of php profilers named XHGui | https://github.com/perftools/xhgui

The following example configures XHGui to profile 1 in 100 requests, excluding requests with the /blog URL path:

Authentication not working

I keep getting the error WARNING:root:flask-profiler authentication failed when I have flask-profiler and auth enabled. Here's my code

app.config["flask_profiler"] = {
    "enabled": True,
    "storage": {
        "FILE": "/var/www/TestApp/flask-profiler.sql",
        "engine": "sqlite"
    },
    "basicAuth": {
        "enabled": True,
        "username": "admin",
        "password": "admin"
    },
    "ignore": [
        "^/account/.*"
    ]
}

The only way to get into flask-profiler is if I disable basicAuth - enabled but I obviously don't want the rest of the world to see my stats. I can verify that the file /var/www/TestApp/flask-profiler.sql is writable as it is about 9MB now.

How do I troubleshoot what is wrong here? The logs don't say much.

performance benchmark

Hi, do you have any performance benchmark for this flask-profiler ? how much overhead does it add to Flask ?

allow to view timeline plot over all time or a range

The timeline graph should have an option to view over the full range or a filterable range of time, rather than just the last 30 days. At least, I cannot find a way to do this in the current dashboard.

Does the database auto-delete entries that are older than 30 days? My production app has been running for longer than that, yet the sqlite file only contains back to mid-January?? Or does the database sqlite file reset everytime the Flask web app is restarted?

Sampling function in config should be callable

The README.md suggests that sampling_function is an expression but it should be a function.
If an expression is supplied to sampling_function, it leads to bool is not callable exception.

Both these issues are addressed in #53

Exception: key 'Com.Android.Browser.Application-Id' must not contain '.'

flask-profiler seems not able to handle http requests with fields containing dot. probably need to sanitize the key for mongo before insertion.

Exception: key 'Com.Android.Browser.Application-Id' must not contain '.'
On Request: {'data': '', 'stream': <cStringIO.StringO object at 0x7fd978db7e30>, '_cached_data': '', 'headers': EnvironHeaders([('Total-Route-Time', u'0'), ('Accept', u'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8,UC/145'), ('User-Agent', u'Mozilla/5.0 (Linux; Android 8.0.0; BLA-AL00 Build/HUAWEIBLA-AL00) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Mobile Safari/537.36'), ('Connection', u'close'), ('Host', u'www.waitfreesln.com'), ('X-Request-Start', u'1519071553385'), ('Com.Android.Browser.Application-Id', u'com.android.mms'), ('Accept-Language', u'en-US'), ('X-Forwarded-For', u'172.58.41.145'), ('X-Forwarded-Port', u'443'), ('X-Request-Id', u'64ef5219-bc8e-4f07-b09b-cbdf2b40ebc5'), ('Accept-Encoding', u'gzip, deflate, br'), ('Connect-Time', u'0'), ('X-Forwarded-Proto', u'https'), ('Via', u'1.1 vegur')]), 'cookies': {}, 'shallow': False, 'url_rule': <Rule '/t/' (OPTIONS, GET, HEAD) -> frontend.catch_all>, 'base_url': u'https://www.waitfreesln.com/t/05c180', 'args': ImmutableMultiDict([]), 'form': ImmutableMultiDict([]), 'files': ImmutableMultiDict([]), 'environ': {'SERVER_SOFTWARE': 'gunicorn/19.6.0', 'wsgi.errors': <gunicorn.http.wsgi.WSGIErrorsWrapper object at 0x7fd97909c290>, 'HTTP_TOTAL_ROUTE_TIME': '0', 'REMOTE_ADDR': '10.142.224.246', 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8,UC/145', 'wsgi.version': (1, 0), 'RAW_URI': '/t/05c180', 'wsgi.input': <gunicorn.http.body.Body object at 0x7fd97a696890>, 'HTTP_USER_AGENT': 'Mozilla/5.0 (Linux; Android 8.0.0; BLA-AL00 Build/HUAWEIBLA-AL00) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Mobile Safari/537.36', 'HTTP_CONNECTION': 'close', 'wsgi.multithread': False, 'wsgi.url_scheme': 'https', 'HTTP_HOST': 'www.waitfreesln.com', 'SERVER_NAME': '0.0.0.0', 'HTTP_X_REQUEST_START': '1519071553385', 'wsgi.file_wrapper': <class 'gunicorn.http.wsgi.FileWrapper'>, 'HTTP_COM.ANDROID.BROWSER.APPLICATION_ID': 'com.android.mms', 'HTTP_ACCEPT_LANGUAGE': 'en-US', 'werkzeug.request': <Request 'https://www.waitfreesln.com/t/05c180' [GET]>, 'HTTP_X_FORWARDED_FOR': '172.58.41.145', 'gunicorn.socket': <socket._socketobject object at 0x7fd974dc6750>, 'HTTP_X_FORWARDED_PORT': '443', 'SERVER_PORT': '6057', 'REMOTE_PORT': '31415', 'QUERY_STRING': '', 'HTTP_X_REQUEST_ID': '64ef5219-bc8e-4f07-b09b-cbdf2b40ebc5', 'PATH_INFO': '/t/05c180', 'wsgi.multiprocess': True, 'SCRIPT_NAME': '', 'HTTP_ACCEPT_ENCODING': 'gzip, deflate, br', 'HTTP_CONNECT_TIME': '0', 'HTTP_X_FORWARDED_PROTO': 'https', 'REQUEST_METHOD': 'GET', 'SERVER_PROTOCOL': 'HTTP/1.1', 'wsgi.run_once': False, 'HTTP_VIA': '1.1 vegur'}, 'view_args': {'path': u'05c180'}}
Traceback: Traceback (most recent call last):
File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functionsrule.endpoint
File "/app/.heroku/python/lib/python2.7/site-packages/flask_profiler/flask_profiler.py", line 137, in wrapper
return wrapped(*args, **kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/flask_profiler/flask_profiler.py", line 116, in wrapper
collection.insert(measurement.json())
File "/app/.heroku/python/lib/python2.7/site-packages/flask_profiler/storage/mongo.py", line 89, in insert
result = self.collection.insert(measurement)
File "/app/.heroku/python/lib/python2.7/site-packages/pymongo/collection.py", line 2212, in insert
check_keys, manipulate, write_concern)
File "/app/.heroku/python/lib/python2.7/site-packages/pymongo/collection.py", line 535, in _insert
check_keys, manipulate, write_concern, op_id, bypass_doc_val)
File "/app/.heroku/python/lib/python2.7/site-packages/pymongo/collection.py", line 516, in _insert_one
check_keys=check_keys)
File "/app/.heroku/python/lib/python2.7/site-packages/pymongo/pool.py", line 244, in command
self._raise_connection_failure(error)
File "/app/.heroku/python/lib/python2.7/site-packages/pymongo/pool.py", line 372, in _raise_connection_failure
raise error
InvalidDocument: key 'Com.Android.Browser.Application-Id' must not contain '.'

incorporating cProfile output into the page

I really like the webpage output of the endpoint statistics. It would be great to include the output from cProfile for each out as well, perhaps even in a visual way. I can see the option of clicking on one of the routes on the main page, and instead of simply displaying the detailed header info, it takes you to a new page that has that header info plus the detailed output from cProfile. How feasible is this?

Flask has a built-in profiler that you can run, which I'm currently using. It would be great to combine these two somehow. http://werkzeug.pocoo.org/docs/0.11/contrib/profiler/

The relevant code lines are

from werkzeug.contrib.profiler import ProfilerMiddleware, MergeStream

if logdirs:
logpath = os.path.join(logdirs, 'profile.log')
file = open(logpath, 'w')
stream = MergeStream(sys.stdout, file)
else:
stream = None

app.config['PROFILE'] = True
app.wsgi_app = ProfilerMiddleware(app.wsgi_app, restrictions=[30], stream=stream)
app.run(debug=True, port=args.port)

Flask crashes when static files are served (ProgrammingError: Recursive use of cursors not allowed)

When I enable the flask-profiler, the app crashes when files from the static folder are served:

Traceback (most recent call last):
  File "flask\app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "flask\app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "flask\app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "flask\app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "flask\app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "flask\app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "flask\app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "flask\app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "flask_profiler\flask_profiler.py", line 91, in wrapper
    return wrapped(*args, **kwargs)
  File "flask_profiler\flask_profiler.py", line 72, in wrapper
    collection.insert(measurement.__json__())
  File "flask_profiler\storage\sqlite.py", line 128, in insert
    name))
ProgrammingError: Recursive use of cursors not allowed.
Traceback (most recent call last):
  File "flask\app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "flask\app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "flask\app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "flask\app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "flask\app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "flask\app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "flask\app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "flask\app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "flask_profiler\flask_profiler.py", line 91, in wrapper
    return wrapped(*args, **kwargs)
  File "flask_profiler\flask_profiler.py", line 72, in wrapper
    collection.insert(measurement.__json__())
  File "flask_profiler\storage\sqlite.py", line 128, in insert
    name))
ProgrammingError: Recursive use of cursors not allowed.

I first though that this is, because requests to /static are not mentioned in the app routes, but also with a route with return app.send_static_file('index.html') I get the same error. All other routes (none are serving files from static) run fine.

I am using:

  • Python 2.7.5
  • Flask 0.10.1
  • Flask-Profiler 0.5

I am happy to provide more information if needed.

i get the following error: TypeError: Incorrect padding

2016-04-11 11:47:34 - ERROR - app: Exception on /hipchat/authorize [GET]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functionsrule.endpoint
File "/usr/local/lib/python2.7/dist-packages/flask_profiler/flask_profiler.py", line 112, in wrapper
return wrapped(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/flask_profiler/flask_profiler.py", line 86, in wrapper
raise e
TypeError: Incorrect padding

Error using python3 mongodb

File "./manage.py", line 20, in
flask_profiler.init_app(app)
File "/Users/sibelius/.virtualenvs/condomnio/lib/python3.4/site-packages/flask_profiler/flask_profiler.py", line 177, in init_app
collection = storage.getCollection(CONF.get("storage", {}))
File "/Users/sibelius/.virtualenvs/condomnio/lib/python3.4/site-packages/flask_profiler/storage/init.py", line 7, in getCollection
from .mongo import Mongo
File "/Users/sibelius/.virtualenvs/condomnio/lib/python3.4/site-packages/flask_profiler/storage/mongo.py", line 3, in
from base import BaseStorage
ImportError: No module named 'base'

sqlite3.OperationalError: database is locked

From time to time when the package is using sqlite, it locks the db and by then any endpoint is accesible. Here the traceback.

Traceback (most recent call last):
  File "/home/lwserver/env/lib/python3.4/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/lwserver/env/lib/python3.4/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/home/lwserver/env/lib/python3.4/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/lwserver/env/lib/python3.4/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/home/lwserver/env/lib/python3.4/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/lwserver/env/lib/python3.4/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/lwserver/env/lib/python3.4/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/lwserver/env/lib/python3.4/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/home/lwserver/env/lib/python3.4/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/lwserver/env/lib/python3.4/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/lwserver/env/lib/python3.4/site-packages/flask_profiler/flask_profiler.py", line 137, in wrapper
    return wrapped(*args, **kwargs)
  File "/home/lwserver/env/lib/python3.4/site-packages/flask_profiler/flask_profiler.py", line 116, in wrapper
    collection.insert(measurement.__json__())
  File "/home/lwserver/env/lib/python3.4/site-packages/flask_profiler/storage/sqlite.py", line 128, in insert
    name))
sqlite3.OperationalError: database is locked

Failed to load webpage when configuring MongoDB as storage backend

flask_profiler throw

TypeError: string indices must be integers

when executing such code:

series[i["startedAt"].strftime(dateFormat)] = i["count"]

This error occurs when flask_profiler try to access aggregated result from mongodb via

result = self.collection.aggregate(...)

and assume result is a list of aggregated data. But what mongodb return actually like this

{u'ok': 1.0, u'result': [{u'count': 216, u'startedAt': datetime.datetime(2016, 1, 22, 10, 28, 17, 119000), u'_id': {u'month': 1, u'day': 22, u'year': 2016}}]} 

Therefore, we should access the aggregation result by result = self.collection.aggregate(...)['result'].
I'm using MongoDB 2.6.3, not sure if MongoDB has changed the aggregation result format ever or in later release.

profile for api running in subthread

Hi. Can I use flask-profiler for api running in subthread? I tried it like the following. However I got ValueError: signal only works in main thread error. Can you please help me to solve it?

  • files
├── server.py
└── api.py
  • server.py
import time
import api

api_server = api.APIServer()
api_server.daemon = True
api_server.start()

while True:
  time.sleep(1)
  • api.py
import threading
from flask import Flask
import flask_profiler

class APIServer(threading.Thread):

  def __init__(self):
    threading.Thread.__init__(self)

  def run(self):
    app = Flask(__name__)
    app.config["DEBUG"] = True

    app.config["flask_profiler"] = {
      "enabled": app.config["DEBUG"],
      "storage": {
        "engine": "sqlite"
      },
      "basicAuth":{
        "enabled": True,
        "username": "admin",
        "password": "admin"
      },
        "ignore": [
          "^/static/.*"
        ]
      }

    @app.route('/products', methods=['GET'])
    def listProducts():
      return "suppose I send you product list..."

    flask_profiler.init_app(app)

    app.run(host="127.0.0.1", port=5000, threaded=True)
  • command
$ python3 server.py
 * Serving Flask app "api" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/home/xxx/flask-profiler/api.py", line 35, in run
    app.run(host="127.0.0.1", port=5000, threaded=True)
  File "/home/xxx/.local/lib/python3.5/site-packages/flask/app.py", line 943, in run
    run_simple(host, port, self, **options)
  File "/home/xxx/.local/lib/python3.5/site-packages/werkzeug/serving.py", line 812, in run_simple
    reloader_type)
  File "/home/xxx/.local/lib/python3.5/site-packages/werkzeug/_reloader.py", line 267, in run_with_reloader
    signal.signal(signal.SIGTERM, lambda *args: sys.exit(0))
  File "/usr/lib/python3.5/signal.py", line 47, in signal
    handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread

use flask-profiler in wsgi production environment

Hi, for my flask application https://github.com/zimmerst/DmpWorkflow, i started using your brilliant profiler -- first off, thanks a lot for doing all the hard work for me. But I seem to be unable to use the flask profiler in the wsgi-production environment but only when i initialize the application by hand typing "python core/manage.py runserver"

For reference, here's the relevant code piece that includes the profiler:

from DmpWorkflow.config.defaults import cfg
import flask_profiler
from DmpWorkflow import version
from socket import getfqdn
kind = cfg.get("global", "installation")

if kind == 'server':    
    from flask import Flask
    from flask.ext.mongoengine import MongoEngine
    app = Flask(__name__)
    app.config.update(LOGGER_NAME="core")
    app.config['MONGODB_DB'] = cfg.get("database", "name")
    app.config['MONGODB_USERNAME'] = cfg.get("database", "user")
    app.config['MONGODB_PASSWORD'] = cfg.get("database", "password")
    app.config['MONGODB_HOST'] = cfg.get("database", "host")
    app.config['MONGODB_PORT'] = int(cfg.get("database", "port"))
    # make connection numbers unbound
    app.config['MONGODB_MAXPOOLSIZE'] = None 
    # time out after 100ms
    app.config['MONGODB_WAITQUEUETIMEOUTMS'] = 100
    app.config["DEBUG"] = True
    app.config["flask_profiler"] = {
        "enabled": app.config["DEBUG"],
        "storage": {
            "engine": "sqlite",
            "FILE": "/tmp/flask-profiler.sql"
        },
        "basicAuth":{
            "enabled": True,
            "username": "myuser",
            "password": "mypassword"
        }
    }

    db = MongoEngine(app)
    
    def register_blueprints(app):
        # Prevents circular imports
        from DmpWorkflow.core.views import jobs
        from DmpWorkflow.core.admin import admin
        app.register_blueprint(jobs)
        app.register_blueprint(admin)
    
        if app.config['flask_profiler']['enabled']: app.logger.info("started flask profiler")
    register_blueprints(app)
    flask_profiler.init_app(app)
    
    def main():
        app.logger.info("started DmpWorkflow Server Version: %s on %s",version,getfqdn())
        app.run()
else:
    def main():
        pass

if __name__ == '__main__':
    if kind == 'server':
        main()

and yes, when running the wsgi application, it runs fine, except that the profiler seems to be unable to handle multiple connections to the sql database:

[Thu Nov 17 09:49:24.257431 2016] [wsgi:error] [pid 21519] started flask profiler
[Thu Nov 17 09:49:24.257440 2016] [wsgi:error] [pid 21519] --------------------------------------------------------------------------------
[Thu Nov 17 09:49:48.635678 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512] mod_wsgi (pid=21519): Exception occurred processing WSGI script '/var/www/flask_dev/workflow.wsgi'.
[Thu Nov 17 09:49:48.635867 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512] Traceback (most recent call last):
[Thu Nov 17 09:49:48.635979 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]   File "/opt/virtualenvs/DAMPE/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
[Thu Nov 17 09:49:48.636587 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]     return self.wsgi_app(environ, start_response)
[Thu Nov 17 09:49:48.636651 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]   File "/opt/virtualenvs/DAMPE/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
[Thu Nov 17 09:49:48.636709 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]     response = self.make_response(self.handle_exception(e))
[Thu Nov 17 09:49:48.636737 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]   File "/opt/virtualenvs/DAMPE/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
[Thu Nov 17 09:49:48.636828 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]     reraise(exc_type, exc_value, tb)
[Thu Nov 17 09:49:48.636844 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]   File "/opt/virtualenvs/DAMPE/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
[Thu Nov 17 09:49:48.636864 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]     response = self.full_dispatch_request()
[Thu Nov 17 09:49:48.636878 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]   File "/opt/virtualenvs/DAMPE/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
[Thu Nov 17 09:49:48.636898 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]     rv = self.handle_user_exception(e)
[Thu Nov 17 09:49:48.636913 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]   File "/opt/virtualenvs/DAMPE/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
[Thu Nov 17 09:49:48.636932 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]     reraise(exc_type, exc_value, tb)
[Thu Nov 17 09:49:48.636965 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]   File "/opt/virtualenvs/DAMPE/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
[Thu Nov 17 09:49:48.636986 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]     rv = self.dispatch_request()
[Thu Nov 17 09:49:48.637000 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]   File "/opt/virtualenvs/DAMPE/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
[Thu Nov 17 09:49:48.637018 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]     return self.view_functions[rule.endpoint](**req.view_args)
[Thu Nov 17 09:49:48.637033 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]   File "/opt/virtualenvs/DAMPE/lib/python2.7/site-packages/flask_profiler/flask_profiler.py", line 113, in wrapper
[Thu Nov 17 09:49:48.637121 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]     return wrapped(*args, **kwargs)
[Thu Nov 17 09:49:48.637140 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]   File "/opt/virtualenvs/DAMPE/lib/python2.7/site-packages/flask_profiler/flask_profiler.py", line 92, in wrapper
[Thu Nov 17 09:49:48.637162 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]     collection.insert(measurement.__json__())
[Thu Nov 17 09:49:48.637176 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]   File "/opt/virtualenvs/DAMPE/lib/python2.7/site-packages/flask_profiler/storage/sqlite.py", line 128, in insert
[Thu Nov 17 09:49:48.637288 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512]     name))
[Thu Nov 17 09:49:48.637321 2016] [wsgi:error] [pid 21519] [remote 129.194.54.242:512] OperationalError: attempt to write a readonly database

is there a way to make sure that the profiler can access the db for my multiple instances of the wsgi app?
Thanks!

needed an option to store also api responses, not only requests

能否记录下返回的结果呢,因为这个插件在用于测试API接口的时候需要这个功能,这样子便于调试。

Can record the result returned, because this plug-in when used to test the API interface to need this functionality, it easy to debug.

Front-end code is not maintainable

The main.js and main.css are minified which block further work on them, please provide the dev versions and use gulp or grunt (with travis-ci) to minify code. Right now one just can't extend the library to include more data since it has no way to display them.

Feature request: Prometheus support

Hi,
I'm starting to monitor my stack with Prometheus 2.0 and I wanted to ask if there are any thoughts in supporting Prometheus.

Support means that there is an url/port endpoint with simple Prometheus compatible data. So the flattened current data of the profiler web page would work theoretically. An example how this data can look is here in the README:
https://github.com/sbarratt/flask-prometheus

Maybe I will try to help with this feature in my spare time (I'm really new to Prometheus at the moment).

flask-profiler work in production mode?

Does the flask-profiler only work in debug mode? While this is great for debugging, I'd like to collect statistics while users are navigating my Flask app in a real production environment. Is this possible? If so, can you provide an example of the config info that needs to be set?

Image asset being counted on each page

Whenever we load one of our pages, our logo (located at /static/assets/img/logo.png) is being counted in the flask-profiler as a GET with a name of "/static/path:filename" and is throwing off our Method distribution graph.

Is there a way to remove this or make flask-profiler ignore this?

screen shot 2017-02-15 at 2 24 22 pm

Allow deletion of profiling from UI and/or profiling session

Are you considering adding some functionality to start a profiling session. As information marked without it is difficult to process ?

Also, one easy way of doing that would be to have buttons like :-

  1. delete all data (to start over)
  2. Save and delete all data (to save a session or all data so far and start over)

If you guys agree on this, I can start working on a PR for the same :)

Thanks

filtering page's paging does not work

when I click a link on the dashboard and go automatically to the filtering page, list paging does not work and looks like it fetches all the date at a time.

Column "name" is not filled

I set up flask-profiler as described in the README, and it works fine so far. However, column "name" is always empty:

dashboard

Apart from that, everything works find - it's just a bit inconvenient to only see the HTTP method.

I am using:

  • Python 2.7.5
  • Flask 0.10.1
  • Flask-Profiler 0.5

I am happy to provide more information if needed.

Is it possible to specific the life duration of the sampling data

I plan to use flask-profiler in an already running production environment.

However, I have a concern that, after running sometime, the collected sampling data will make the disk full.

Is there a way to tell flask-profiler, only keep the sampling data for last 14 days?

Thanks.

Feature for granular profiling

While using flask-profiler in production (via https://github.com/muatik/flask-profiler#sampling)

I noticed that I miss a feature, while I am able to detect the slower endpoints, I had to go manually to profile the slower end points line by line, which involved going through multiple function calls.

How about integrating something like https://github.com/rkern/line_profiler into flask-profiler itself.

Just throwing an idea here for discussion, in my opinion there are two ways to go about implementing this, one way would be to make this a core feature in flask-profiler and let this be enabled via config, find a way to show this on UI.

Another way would be to not make this as a feature but allowing saving extra data with each request (by fixing a key). So, whenever someone will need to profile code line by line he will import this as a dependency and piggyback the data to flask-profiler manually. This way can also help us solve #42

Thinking in terms of simplicity vs feature, I think if done after a lot of thought we can still manage to keep the code simple enough and add this granular profiling feature there by making flask-profiler even more helpful.

What is your opinion on all of this ?

Thanks

BaseStorage should have method truncate

truncate method is used by the route <your-app>/<flask-profiler-endpoint>/db/deleteDatabase introduced in #41 but it is not defined in the BaseStorage class.

The following method should be present in storage/base.py
This will provide an interface for supporting any new database, there by, making the code more robust.

def truncate(self):
        raise Exception("Not imlemented Error")

Web interface not accessible if host matching is enabled

If my app uses host matching,

app.url_map.host_matching = True

then the flask-profiler web interface cannot be accessed as the URL will always return 404 since it isn't mapped to a host. I would like to suggest an optional config option to set the host to use for the web interface.

changelog?

Is there a changelog somewhere indicating the differences between versions? I'd like to know the features/bug fixes/changes different between 1.5 and 1.6. If there isn't one, I think it would be helpful to include.

unable to open database file on production

I am having issue accessing the database file on Linux production server. I know the problem is associated with permission given to apache user, however, nothing seems to work. it only works when I run application using sudo. Any ideas what might be the reason ?

Minor release for new features and fixes

Hi,

Could we make a minor release here, want to use last changes on a project, my issue is I register all extensions and then all API resources using an app factory and I cannot do it reverse, with last feature implementation no matters where profiler is initialized.

Add dynamodb as database system

This a storage backend that requires almost zero maintenance, cheap and scalable. If I continue using flask profiler I'll try to contribute to this but opening an issue in case it's relevant for someone else.

python3 support

As of now, I think python3 is not supported. I am getting the error

Divyendus-MacBook-Pro:examples divyendusingh$ python3 app.py
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with fsevents reloader
 * Debugger is active!
Traceback (most recent call last):
  File "app.py", line 62, in <module>
    app.run(host="127.0.0.1", port=5000)
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 772, in run
    run_simple(host, port, self, **options)
  File "/usr/local/lib/python3.5/site-packages/werkzeug/serving.py", line 633, in run_simple
    application = DebuggedApplication(application, use_evalex)
  File "/usr/local/lib/python3.5/site-packages/werkzeug/debug/__init__.py", line 249, in __init__
    if self.pin is None:
  File "/usr/local/lib/python3.5/site-packages/werkzeug/debug/__init__.py", line 259, in _get_pin
    self._pin, self._pin_cookie = get_pin_and_cookie_name(self.app)
  File "/usr/local/lib/python3.5/site-packages/werkzeug/debug/__init__.py", line 159, in get_pin_and_cookie_name
    get_machine_id(),
  File "/usr/local/lib/python3.5/site-packages/werkzeug/debug/__init__.py", line 94, in get_machine_id
    _machine_id = rv = _generate()
  File "/usr/local/lib/python3.5/site-packages/werkzeug/debug/__init__.py", line 70, in _generate
    match = re.match(r'"serial-number" = <([^>]+)', dump)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/re.py", line 163, in match
    return _compile(pattern, flags).match(string)
TypeError: cannot use a string pattern on a bytes-like object

I think we should be moving on to support python3 as well. Should I work on a PR adding python3 support and automating the test cases to check both python 2, python 3 support ?

Thanks

Securing the metrics

Can you tell me how to secure the metrics?

What I mean is I dont want to keep the metrics urls like /flask-profiler/ open to anybody. How can I add a security system?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.