xtream1101 / cutil Goto Github PK

Download Error: [Errno 104] Connection reset by peer

Try download once more

Currently only support http proxies

If they are not cought those links will be skip and the content not saved.

Solution:

1. If a 503 error occurs, go into a waiting state of x mins then try again
2. Stop the scrape and print why it stopped. Will pick up where it left off on next run

On 403 http erros, auto change proxy and retry the request
Be able to pass in auto_change=False to not switch proxies on error

Right now it will keep roating through the proxy list and trying the same ones over and over. If all proxies have been tried, then exit and let the user know that all proxies are dead/banned

Needs to be updated/reformatted

Postgres has a json type.
When using the upsert function with a value that is a dict/json it fails with the error

psycopg2.ProgrammingError: column "foo1" is of type json but expression is of type text
HINT:  You will need to rewrite or cast the expression.

Make an extra_requires (probably called postgres that adds the psycopg2 dependency).

When using the upsert function with a single dict and a key has a None value, the upsert will fail with the error:
psycopg2.ProgrammingError: could not determine polymorphic type because input has type "unknown"

This is an issue with using unnest when the array only contains null values.

Examples:

fails_1 = [{'id': 1,
            'val1': None,
            'val2': 'bar',
            }]
fails_2 = [{'id': 1,
            'val1': None,
            'val2': 'bar',
            },
           {'id': 2,
            'val1': None,
            'val2': 'bar',
            }]

# The whole array is not None for a single field
success_1 = [{'id': 1,
              'val1': None,
              'val2': 'bar',
              },
             {'id': 2,
              'val1':'foo',
              'val2': 'bar',
              }]

HTTPConnectionPool(host='<host>', port=<port>): Max retries exceeded with url: <url> (Caused by ProxyError('Cannot connect to proxy.', gaierror(11001, 'getaddrinfo failed')))

Try another proxy if it has, else exit script and let the user know that the proxy does not work