othernet-project / artexin Goto Github PK

The whole site will be accessed via SSH tunnel, so no need for convoluted auth.

Clean up fixtures from the repository (and don't use them in the tests).

http://rg3.github.io/youtube-dl/

Simply use nginx's autoindex on for now.

At very least, we should track URLs and their status ('added', 'collected', 'broadcast'). Including some metadata about the URLs is optional.

Do not convert to soup object every time. Not very efficient.

When image URL is passed, generate a HTML page that contains a single image with whatever metadata is available to us.

Use either lower-case-lettters-only (a-z) or mix of lower- and upper-case letters (a-zA-Z).

This is potentially expensive, though.

Add a basic UI that takes a list of URLs and starts collecting them into zip files.

The software should check for available space before starting the operation, and should consider each URL as taking up 3MB of space in total during processing. The software should warn the user when sufficient space is not available.

Rationale for 3MB per URL storage capacity comes from the average size of a web page today of 1.5MB and the additional storage needed to create the zip file.

The maximum size per page should be configurable, and there should be a safety margin as well (i.e., X megabytes less than total free space should be taken by the collecting).

To prevent race conditions, there should be a lock file which the web UI should create before attempting to reserve space for its operation. If web UI finds a lock file, it should wait for its removal before attempting to calculate the available storage space.

There should be a file, called 'reservations', that contains the total space used by all processes (rounded up to nearest integer). Each process should:

• install the lock file first
• read the file contents
• calculate physically available space
• take into account the space registered in the reservations
• add to reservations the space it needs
• overwrite the reservations file with updated data
• release the lock

If the thread cannot reserve enough space, it should release the lock immediately, and inform the user.

Ideally, we should be able to do something like bundle.py http://whatever.com/or/the/other.html and get an unencrypted zipball. This tool would be used by site owners to test ArtExIn output. It should also generate enough debug data so that meaningful bug reports can be made. Also provide .exe version for Windows users.

TODO: Add some notes about desired behavior here

Page titles very commonly unify site name with article title using separators like dash, dot, bar, etc. We need an intelligent way of stripping away the site name.

Example page: https://docs.python.org/3.4/library/argparse.html#metavar

Instead of processing URLs in child processes, we need a proper job queue (like Celery).

ArtExIn should provide an UI for monitoring the queue and reporting on success or failure.

User that creates a new task should receive email notification on success, and there should be an ability to send out notification to admins as well.

We could use this info when deciding whether to broadcast it or not, and whether to put it in priority lane or not.

The tools may or may not include a web-based UI, but local use should be assumed.

The tools are plural, each handling a specific use case, rather than trying to be a complete tool shed.

Readability seems to think article begins quite low inside the page. Actual 'article' is contained within a div#title-overview-widget.

Example page: http://www.imdb.com/title/tt0087803/

1. Type in email and password
2. Receive login link via email
3. Click on the link to access the site

The link should contain a one-time code that expires after 3 minutes.

Sessions should expire when browser window is closed (ideally). Opt-in 'remember me' feature should extend the session for a maximum of 14 days.

The web UI should allow the admin to see a listing of zip files, removal of individual zip files or multiple zip files at once.

Each channel should have a human-readable name, and a folder name (single word consisting of alphanumerics and underscores). Each channel folder should be created when a new channel is created in the UI, and should contain a .name file that contains the human readable name as single-line string.

Each piece of content should be assigned a channel when processed in a batch. There should be an UI for assigning the channel to each URL as well as choosing the default channel for the entire batch.

We probably want more flexibility in choosing what to do with each URL.

The UI should present a multi-select box for each URL where user can choose what pre- and post- processor to run (including clean pass-through).

The simple template on Blogger has post title outside the DIV which contains the article text. This needs to be handled at least for *.blogspot.com URLs.