eveprosper / prosperapi Goto Github PK

https://pypi.python.org/pypi/pytest-pylint/0.7.0

Add pylint validation to pytest test suite.

Improve performance by making sure ujson is integrated everywhere in the stack it should be.

• incoming requests
• tinydb connections
• pandas to_json
  • pandas.io.json.dumps = ujson.dumps
  • pandas.io.json.loads = usjon.loads + precise_float=True in pd.read_json() args
• skip Flask.jsonify by building response directly from pandas

ESI calls are failing ~1%. Add retry

[ERROR:crest_utils.py--fetch_market_history:354]
ERROR: unable to fetch market history from CREST
    type_id: 39
    region_id: 10000002
Traceback (most recent call last):
  File "/opt/venvs/prosper-api/lib/python3.5/site-packages/publicAPI/crest_utils.py", line 343, in fetch_market_history
    config=config
  File "/opt/venvs/prosper-api/lib/python3.5/site-packages/publicAPI/crest_utils.py", line 303, in fetch_esi_endpoint
    req.raise_for_status()
  File "/opt/venvs/prosper-api/lib/python3.5/site-packages/requests/models.py", line 909, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://esi.tech.ccp.is/latest/markets/10000002/history/?datasource=tranquility&type_id=39
[ERROR:forecast_utils.py--fetch_extended_history:201]
ERROR: trouble getting data from CREST
    region_id=10000002
    type_id=39
Traceback (most recent call last):
  File "/opt/venvs/prosper-api/lib/python3.5/site-packages/publicAPI/crest_utils.py", line 343, in fetch_market_history
    config=config
  File "/opt/venvs/prosper-api/lib/python3.5/site-packages/publicAPI/crest_utils.py", line 303, in fetch_esi_endpoint
    req.raise_for_status()
  File "/opt/venvs/prosper-api/lib/python3.5/site-packages/requests/models.py", line 909, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://esi.tech.ccp.is/latest/markets/10000002/history/?datasource=tranquility&type_id=39

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/prosper-api/lib/python3.5/site-packages/publicAPI/forecast_utils.py", line 194, in fetch_extended_history
    logger=logger
  File "/opt/venvs/prosper-api/lib/python3.5/site-packages/publicAPI/crest_utils.py", line 361, in fetch_market_history
    mode.name
publicAPI.exceptions.CRESTBadMarketData```

Use intelpython docker package: https://software.intel.com/en-us/articles/docker-images-for-intel-python

All deployed apps should report their version. Move version into publicAPI/_version.py and add a /CREST/info/version REST endpoint to check it

Right after logger.info() at the start of each get(), a record should be saved of the following:

• Which endpoint was requested
• Time requested
• args of request

This should be saved somewhere other than cache to avoid steamrolling when installing. /var/ or /opt/ might be the place. In testmode, could save to ROOT level of project. Add path to app.cfg

Be able to roll readthedocs.org docs for project. Also, allow for test failure if docstrings aren't acceptable

EMD requests are causing major issues:

self.chunk_left = int(line, 16)
E           ValueError: invalid literal for int() with base 16: b''

https://travis-ci.org/EVEprosper/ProsperAPI/builds/482103441#L703

Update system to take advantage of new packaging on ProsperCommon.

Though we are covered for exceptions with req.raise_for_status() we are not capturing all the error information.

• Add header/req.text capture to log messages
  • Be careful of 1000 char limit on WebHook log messages
• Fix double-counted HTTP exceptions spamming discord

In an effort to subdue crawlers, add a robots.txt to publicAPI/static

http://stackoverflow.com/questions/14048779/with-flask-how-can-i-serve-robots-txt-and-sitemap-xml-as-static-files

Notes:

• need to add publicAPI/static to MANIFEST.in
• may need to add publicAPI/static/* to package_data in setup.py
• Any new endpoints will need to add tests to test_crest_endpoint.py

PLEX 29668 is splitting on May 9th

Database will need to backfill the split, and handle the type conversion across split-date.

Expected behavior

• typeid=29668: newplex * 500
• typeid=44992: oldplex / 500

Config/Logger utilities were overhauled in v0.2.x to v0.3.x. Need to upgrade/update calls to config and how logger is created

For parallel containers, a shared cache is desired.

• Switch all TinyDB connections to tinymongo ones
• Add pymongo connection
• Default to tiny if/when db connection fails