taroved / pol Goto Github PK

View Code? Open in Web Editor NEW

375.0 375.0 87.0 1.74 MB

RSS generator website

License: MIT License

Python 22.40% CSS 0.84% JavaScript 14.41% HTML 6.85% Dockerfile 0.51% Shell 0.42% Less 52.86% SCSS 1.70%

pol's People

Contributors

Stargazers

Watchers

Forkers

duuroov schulzpin fmogollonr augustgerro povilaskud immortalin icydevx vezycash jreming85 nianglao smylk developerxd traverseda blindslug materaspovic freetoplayir ecirkucuksille muslimconditions vicentehenrique marlluslustosa ninjayoto fakegit 5l1v3r1 anjneyap jamesknox 37dev saeed-parsaee chasekidder amblerkr ayush-kr-gupta tosunkaya fl4min noamelf daniel-illi albinfalk amilcarotero gerrampard trendingtechnology xfrose lauravpf jpverkamp hazemk537 githork d68fbe50 beluckydaf getrss kiyn yesunewh kuraeko eloisetwaine cbrianso slozenger randrienko markcco billmetangmo launacloud symnok sasagr ibrahimkhan2004 majortomde mrjaroslavik gorvis ahmgam shabazz33 lsr00ter ramykatour allenpengk zhangjun1996 oknow23 roadkingvrod poloskovic 712346867 qupb theresaqwq niceguy695c ratulbaba7 ernstki yrmuq tietu-rs coffiner miracle2050 kotpaslyon archiveproject shivy23

pol's Issues

Load Scripts option not displayed in self-hosted PolitePol

While I'm in https://politepol.com I can check the option "Load scripts" for a page.
However, when I self hosted the code avaliable here, this option is missing.

Am I doing something wrong?

Docker error: Service 'politepol' failed to build:

I am running the Docker installation instructions and am getting the following error:

ERROR: Service 'politepol' failed to build: Error parsing reference: "ubuntu:bionic as builder" is not a valid repository/tag: invalid reference format

Any idea what's causing this?

update frequency

Hats up for awesome design and framework. May I know the update frequency the politepol uses to generate the feeds? Is it possible to manually change the frequency?

a news site i want to make a feed out of requires javascript to load images, but when I check the load scripts option I get a gdpr notce that i can't get past to pick the title, image and description. some way to either interact with the site or remove elements from the preview would be nice, or something like the "I dont care about cookies" or "never consent" extension could be integrated if possible.

Не загружаются следующие сайты

http://austria.ved.gov.ru/ru/news
https://www.kiosk.at/web/de/ausgabe/VERKEHR/20180601/archiv

Они теоретически совместимы с PolitePol или лучше искать другой метод создания фидов?

Restarted the server but..

I restarted the server but can't make it work properly now.. The fronted doesn't work at all as it shows 502 bad gateway.. To keep the previous feeds up, I'm running downloader.py on console..

What can I do at the moment to fix everything?

Request: Option to select date and image when creating feed?

It would make this perfect if there was an option to select a date and image element.

Feature request, "Next" button

I'd like to use this to archive forum threads. In order to do that I need it to be able to click the "next" button, and always go to the latest page.

Bad Gateway

I just build the docker image using the compose file and I am getting 502 bad gateway when I try a url..

politepol      |   Applying frontend.0003_field_required... OK
politepol      |   Applying frontend.0004_feed_edited... OK
politepol      |   Applying frontend.0005_auto_20180131_1949... OK
politepol      |   Applying frontend.0008_auto_20180215_1445.../usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py:124: Warning: Duplicate index 'frontend_post_feed_id_75a47426eda13de_idx' defined on the table 'politepol.frontend_post'. This is deprecated and will be disallowed in a future release.
politepol      |   return self.cursor.execute(query, args)
politepol      | 
politepol      |  OK
politepol      |   Applying sessions.0001_initial... OK
politepol      |   Applying sites.0001_initial... OK
politepol      | Installed 3 object(s) from 1 fixture(s)
politepol      | Traceback (most recent call last):
politepol      |   File "./downloader.py", line 2, in <module>
politepol      |     from pol.server import Server
politepol      | ImportError: No module named pol.server
politepol      | [14/Mar/2020 23:25:17] "GET /en/ HTTP/1.0" 200 6490
politepol      | [14/Mar/2020 23:25:18] "GET /static/frontend/stylesheets/application.css HTTP/1.0" 200 1372
politepol      | [14/Mar/2020 23:25:18] "GET /static/frontend/stylesheets/wizard.css.css HTTP/1.0" 200 1575

docker composer file

version: "3"

services:
  politepol:
    build:
      context: .
    environment:
      DB_NAME: 'politepol'
      DB_USER: 'rooooooooooot'
      DB_PASSWORD: 'toooooooooooor'
      DB_HOST: 'dbpolitepol'
      DB_PORT: '3306'
      WEB_PORT: '8088'
      TIME_ZONE: 'America\/Fortaleza'
    image: politepol:latest
    depends_on:
      - 'dbpolitepol'
    #command: ["./wait-for-it.sh", "dbpolitepol:3306", "--", "/bin/bash", "./frontend/start.sh"]
    command: ["./wait-for-it.sh", "dbpolitepol", "/bin/bash", "./frontend/start.sh"]
    container_name: politepol
    restart: unless-stopped
    networks:
      - politepol-network
    ports:
      - '8088:8088'

  dbpolitepol:
    image: mysql:5.7
    container_name: dbpolitepol
    restart: unless-stopped
    environment:
      MYSQL_DATABASE: 'politepol'
      MYSQL_USER: 'rooooooooooot'
      MYSQL_PASSWORD: 'toooooooooooor'
      MYSQL_ROOT_PASSWORD: 'rootpass'
    networks:
      - politepol-network
    volumes:
      - ./mysql:/var/lib/mysql

networks:
  politepol-network:
    driver: 'bridge'

Front end and download server is getting closed automatically

How to run the front end and download server in background? once i quit the terminal, both are getting terminated.

Фиды не работают

Фиды, которые были созданы с помощью PolitePol, на данный момент не работают. Вы в курсе?

Official Docker hub image

Hello.

I wanted to try self-hosting this on my server.

I already have a docker-compose setup for all my self-hosted software, and would rather not have to clone your repo inside mine, or use it as a submodule, just so that I can build the image.

Could you publish an official Docker Hub image that your users can pull?

Admin and sign in/up pages in Docker

I feel this issue is not related to docker but at least I could try it out this way thanks to #32

When I started up the project, I noticed that the menu is different from the online politepol.com. Namely, there are no Sign in/Sign up buttons and there is no admin page as well (https://politepol.com/en/admin/login/?next=/en/admin/). When I manually enter the URLs in the local env, Django outputs error pages which mean that debug mode is also enabled.

How can one edit the added feeds in the local env without signin or admin pages? How could I make them work, do I miss a configuration somewhere that's also need to be added to the Dockerfile or start.sh?

Installation

Hey Alex. I'm having problems with installing politepol. Is it alright with you if I give you root access to the vps for you to install the politepol?

compatibility with raspberry Debian Buster

I tried to build it in the raspberry pi4 and got the following message during pulling dbpolitepol...

no matching manifest for linux/arm/v7 in the manifest list entries

did u plan to add compatibility to raspberry architecture?

Xpath doesn't accept some symbols.

Hi, tried to use "contains" function with russian symbols inside and '€' symbol.
For both page returned error. <h1>Server Error (500)</h1>

Docker install borks :(

Debian testing x64

docker-compose up -d --build

 Downloading Scrapy-1.4.0-py2.py3-none-any.whl (248 kB)
Collecting django-pipeline==1.5.4
  Downloading django_pipeline-1.5.4-py2.py3-none-any.whl (29 kB)
Collecting mysqlclient==1.3.7
  Downloading mysqlclient-1.3.7.tar.gz (79 kB)
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-TnjTQr/mysqlclient/setup.py'"'"'; __file__='"'"'/tmp/pip-install-TnjTQr/mysqlclient/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-TnjTQr/mysqlclient/pip-egg-info
         cwd: /tmp/pip-install-TnjTQr/mysqlclient/
    Complete output (10 lines):
    sh: 1: mysql_config: not found
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-TnjTQr/mysqlclient/setup.py", line 17, in <module>
        metadata, options = get_config()
      File "setup_posix.py", line 44, in get_config
        libs = mysql_config("libs_r")
      File "setup_posix.py", line 26, in mysql_config
        raise EnvironmentError("%s not found" % (mysql_config.path,))
    EnvironmentError: mysql_config not found
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Service 'politepol' failed to build: The command '/bin/sh -c pip install -r requirements.txt' returned a non-zero code: 1

Endless Loading while creating feed

After I select title & desc and click the next button

It is stuck on a endless spinning circle.

Logs show

"POST /setup_create_feed HTTP/1.0" 500 12604

in the frontend app

and

[twisted.web.client._HTTP11ClientFactory#info] Stopping factory _HTTP11ClientFactory(<function quiescentCallback at 0x7fecd2a06aa0>, )

in the backend

Any ideas?

Request: A marking tool to omit objects on the page when creating rss feed

A marking tool for n00bs who don't know how html works to omit objects that you don't want to pop up on the page when creating a rss feed (that usually produce stuff like empty lines and other unwanted stuff in your feed).

Simpler Architecture (without database)

you could make it work without a database at all. just put everything into a link like:

http://politepol.com/en/setup?url=https%3A//github.com/rssowl/RSSOwl/issues&title=a.link-gray-dark&description=span.opened-by

the link can now be used in any rss aggregator to get the rss and you save yourself many potential problems with a database.

Не работает с russian.rt.com

На страницах

https://russian.rt.com/inotv/catalog/country/at
https://russian.rt.com/inotv/tag/%D0%90%D0%B2%D1%81%D1%82%D1%80%D0%B8%D1%8F

нельзя выбрать заголовок и описание.

Compiler error - Fixnum is deprecated

Request Method:	GET
http://127.0.0.1:8000/en/
1.8.6
CompilerError
/var/lib/gems/2.5.0/gems/sass-3.4.22/lib/sass/util.rb:1109: warning: constant ::Fixnum is deprecated
/usr/local/lib/python2.7/dist-packages/pipeline/compilers/init.py in execute_command, line 102
/usr/bin/python
2.7.15
['/root/pol/frontend', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']
Wed, 18 Sep 2019 19:59:30 +0000

Provide image link in RSS feed

deploying on heroku with clearDB?

Is it possible to deploy it on heroku with clearDB guys? is anyone try it to share the details wirt us? I've been googling after this for quite some time now and I didn't get anything useful

307 Infinite redirection detected

PolitePol says: "Something wrong"

Try to refresh page or contact us by email: [email protected]
(Help us to improve our service with your feedback)

Scary mantra: [<twisted.python.failure.Failure twisted.web.error.InfiniteRedirection: 307 Infinite redirection detected to https://blog.path.net/>]

Looks like that site have anti bot protection

Facebook Loading stops when Javascript is activated

I don't know if its a premium feature or its available to free accounts, but when activating javascript for facebook webpage the frame loads blank page

Allowing for remote access

Howdy,

Just wanted to make sure I'm not going insane. Is there a setting that prevents remote hosts from getting the page on port 8000?

If so where is there a config file somewhere I can edit to allow remote access?

how do I see list all my feeds?

I built it on ubuntu and all seems working fine. I created my first feed and it works (/feed/1). I created another one (/feed/2) and also ok but how do I see a list of them? How can I manage them (edit, delete, preview,..)?
I tried with localhost:8088/myfeeds but it does not work

How do I make this work?

Hi all,

I have a small google cloud vps where I want to run this, after install the website works, but creating feed always stops at:
[13/Mar/2018 17:39:46] "POST /setup_create_feed HTTP/1.0" 500 13972

Возможность переименовать фиды

Было бы удобно иметь возможность переименовать отдельные фиды в списке фидов.

RSS Feed overview and management

Is it possible to have a feed overview and management page, like on politepol.com?

MariaDB instead of mysql-server

How about MariaDB instead of mysql-server? It's fully mysql compatible and very lightweight.

blank scrap on diaspora site

blank scrap on diaspora site but mastodon ok. i try this url

https://diaspora.club/people/

Add script loading and image selection

Hi. Downloading scripts and adding pictures is available on the http://politepol.com/. Will this feature appear here?

limiting the rss results

is it possible to limit the feed results created by the politepol on a given feed?

setup_create_feed(_ext) can't be loaded

I setup the POL project on my VServer but if I try to create a new feed, I only see the loading circle.

In the nginx access-log I see that the page "setup_create_feed" can't be loaded.

root@VSERVER:~# tailf /var/log/nginx/access.log
11.11.11.11 - - [06/Jul/2018:03:19:13 +0200] "GET /static/frontend/js/google-code-prettify/prettify.js HTTP/1.1" 304 0 "http://DOMAIN.TLD:8080/en/setup?url=https://netzpolitik.org/category/datenschutz/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.183 Safari/537.36 Vivaldi/1.96.1147.52"
11.11.11.11 - - [06/Jul/2018:03:19:13 +0200] "GET /static/frontend/js/setup-tool.js HTTP/1.1" 200 15395 "http://DOMAIN.TLD:8080/en/setup?url=https://netzpolitik.org/category/datenschutz/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.183 Safari/537.36 Vivaldi/1.96.1147.52"
11.11.11.11 - - [06/Jul/2018:03:19:13 +0200] "GET /static/frontend/js/setup-tool-ext.js HTTP/1.1" 304 0 "http://DOMAIN.TLD:8080/en/setup?url=https://netzpolitik.org/category/datenschutz/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.183 Safari/537.36 Vivaldi/1.96.1147.52"
11.11.11.11 - - [06/Jul/2018:03:19:13 +0200] "GET /static/frontend/js/help.js HTTP/1.1" 304 0 "http://DOMAIN.TLD:8080/en/setup?url=https://netzpolitik.org/category/datenschutz/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.183 Safari/537.36 Vivaldi/1.96.1147.52"
11.11.11.11 - - [06/Jul/2018:03:19:13 +0200] "GET /static/frontend/images/apple-touch-icon-precomposed.png HTTP/1.1" 304 0 "http://DOMAIN.TLD:8080/en/setup?url=https://netzpolitik.org/category/datenschutz/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.183 Safari/537.36 Vivaldi/1.96.1147.52"
11.11.11.11 - - [06/Jul/2018:03:19:13 +0200] "GET /static/frontend/images/target48.png HTTP/1.1" 304 0 "http://DOMAIN.TLD:8080/en/setup?url=https://netzpolitik.org/category/datenschutz/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.183 Safari/537.36 Vivaldi/1.96.1147.52"
11.11.11.11 - - [06/Jul/2018:03:19:13 +0200] "GET /static/frontend/images/wrench48.png HTTP/1.1" 304 0 "http://DOMAIN.TLD:8080/en/setup?url=https://netzpolitik.org/category/datenschutz/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.183 Safari/537.36 Vivaldi/1.96.1147.52"
11.11.11.11 - - [06/Jul/2018:03:19:13 +0200] "GET /static/frontend/images/glyphicons-halflings-white.png HTTP/1.1" 304 0 "http://DOMAIN.TLD:8080/static/frontend/stylesheets/bootstrap_and_overrides.css.css" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.183 Safari/537.36 Vivaldi/1.96.1147.52"
11.11.11.11 - - [06/Jul/2018:03:19:13 +0200] "GET /downloader?url=https%3A%2F%2Fnetzpolitik.org%2Fcategory%2Fdatenschutz%2F HTTP/1.1" 200 23314 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.183 Safari/537.36 Vivaldi/1.96.1147.52"
11.11.11.11 - - [06/Jul/2018:03:19:13 +0200] "GET /static/frontend/images/favicon.ico HTTP/1.1" 200 1150 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.183 Safari/537.36 Vivaldi/1.96.1147.52"
11.11.11.11 - - [06/Jul/2018:03:19:21 +0200] "POST /setup_get_selected_ids HTTP/1.1" 200 212 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.183 Safari/537.36 Vivaldi/1.96.1147.52"
11.11.11.11 - - [06/Jul/2018:03:19:23 +0200] "POST /setup_get_selected_ids HTTP/1.1" 200 298 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.183 Safari/537.36 Vivaldi/1.96.1147.52"
11.11.11.11 - - [06/Jul/2018:03:19:25 +0200] "POST /setup_create_feed HTTP/1.1" 500 11899 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.183 Safari/537.36 Vivaldi/1.96.1147.52"

Installed all components exactly as described in the readme file. Only differences:

Installed sqlparse in additional (needed for python manage.py migrate)
The downloader and frontend server are running as systemd service
Nginx is running on port 8080

System is a Ubuntu:16.04. Same happend in a docker container.

Will politepol.com come back again?

politepol.com is currently down. Do you plan to offer your service for free again?

New feature. Edit link ability.

I would like to be able to edit the source link.

That is useful, for example, when you want to add some filters, to request url.
Now you should create a new feed and delete the old one, that will change feed url, so you need also to update it on all devices.

embedded iframe 404 errors

Hi, I've been working on hosting politepol via docker on my server and have been able to get Django serving the appropriate pages. However, I am getting 404 errors in the embedded iframe when I attempt to setup a new feed. Here is a screenshot of the Django error: https://imgur.com/a/pcFMSRW . I think it might be due to something with nginx, but I don't know enough about it to be sure. I would appreciate any insight you could give me on this.

_mysql_exceptions.OperationalError

Not sure where does it go wrong...no error running Initialise database script.
Please help, thank you very much.

MySQL version: 5.7.22-0ubuntu0.16.04.1

After created the Feed, it shows:

Something wrong. Reload the page or contact us by email

When I click on the Feed link, it shows:

web.Server Traceback (most recent call last):
_mysql_exceptions.OperationalError: (1054, "Unknown column 'f.name' in 'field list'")

Error log in Backend:

2018-06-26T00:19:57+0800 [twisted.web.server.Request#critical]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/twisted/protocols/basic.py", line 572, in dataReceived
why = self.lineReceived(line)
File "/usr/local/lib/python2.7/dist-packages/twisted/web/http.py", line 2054, in lineReceived
self.allContentReceived()
File "/usr/local/lib/python2.7/dist-packages/twisted/web/http.py", line 2145, in allContentReceived
req.requestReceived(command, path, version)
File "/usr/local/lib/python2.7/dist-packages/twisted/web/http.py", line 890, in requestReceived
self.process()
--- ---
File "/usr/local/lib/python2.7/dist-packages/twisted/web/server.py", line 197, in process
self.render(resrc)
File "/usr/local/lib/python2.7/dist-packages/twisted/web/server.py", line 257, in render
body = resrc.render(self)
File "/usr/local/lib/python2.7/dist-packages/twisted/web/resource.py", line 250, in render
return m(request)
File "/root/pol/pol/server.py", line 340, in render_GET
res = self.feed.getFeedData(feed_id)
File "/root/pol/pol/feed.py", line 160, in getFeedData
where f.id=%s""", (feed_id,))
File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 226, in execute
self.errorhandler(self, exc, value)
File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorvalue
_mysql_exceptions.OperationalError: (1054, "Unknown column 'f.name' in 'field list'")

Error log in Frontend:

[25/Jun/2018 16:19:52] "POST /setup_get_selected_ids HTTP/1.0" 200 773
/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py:124: Warning: Field 'owner_id' doesn't have a default value
return self.cursor.execute(query, args)

[25/Jun/2018 16:19:53] "POST /setup_create_feed HTTP/1.0" 200 13
[25/Jun/2018 16:19:54] "GET /en/preview/1 HTTP/1.0" 200 7168

[Feature-Request] Fill more rss parts manually

title (exists)
desciption (exists)
link
author
creation-date
update-date
image
guid (chosen manual to be link, title, ...)
etc (whatever else rss or atom allows)

see fetchrss.com for the same approach as you have but with more options

im trying to rssify github issues like: https://github.com/taroved/pol/issues

_mysql_exceptions.OperationalError: (1366, "Incorrect string value

Hey there, Thanks for this great app. I have been getting this error when I tried to setup my first feed. How can I resolve it ?

Downloader error: (1366, "Incorrect string value: '\\x96!a@\\xA0\\xBE...' for column 'md5sum' at row 1")
Traceback: Traceback (most recent call last):
  File "/home/damion/.pyenv/versions/pol_rss/lib/python2.7/site-packages/twisted/web/_newclient.py", line 1306, in _bodyDataFinished_CONNECTED
    self._bodyProtocol.connectionLost(reason)
  File "/var/www/pol_rss/pol/client.py", line 76, in connectionLost
    self.deferred.callback(b''.join(self.dataBuffer))
  File "/home/damion/.pyenv/versions/pol_rss/lib/python2.7/site-packages/twisted/internet/defer.py", line 460, in callback
    self._startRunCallbacks(result)
  File "/home/damion/.pyenv/versions/pol_rss/lib/python2.7/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File "/home/damion/.pyenv/versions/pol_rss/lib/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/var/www/pol_rss/pol/server.py", line 214, in downloadDone
    self.writeResponse(sresponse)
  File "/var/www/pol_rss/pol/server.py", line 225, in writeResponse
    response_str = self.prepare_response_str(sresponse.selector, sresponse.headers, sresponse.body_as_unicode(), sresponse.url, ip)
  File "/var/www/pol_rss/pol/server.py", line 244, in prepare_response_str
    [response_str, post_cnt, new_post_cnt] = self.feed.buildFeed(selector, page_unicode, self.feed_config)
  File "/var/www/pol_rss/pol/feed.py", line 129, in buildFeed
    new_post_cnt = self.fill_time(feed_config['id'], items)
  File "/var/www/pol_rss/pol/feed.py", line 81, in fill_time
    self.save_post(conn, cur_time, feed_id, item)
  File "/var/www/pol_rss/pol/feed.py", line 36, in save_post
    values (unhex(%s), %s, %s)""", (post_fields['md5'], created, feed_id))
  File "/home/damion/.pyenv/versions/pol_rss/lib/python2.7/site-packages/MySQLdb/cursors.py", line 226, in execute
    self.errorhandler(self, exc, value)
  File "/home/damion/.pyenv/versions/pol_rss/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorvalue
_mysql_exceptions.OperationalError: (1366, "Incorrect string value: '\\x96!a@\\xA0\\xBE...' for column 'md5sum' at row 1")

HTTP 500 while creating new feed: Field 'owner_id' doesn't have a default value

I set up POL on my Ubuntu
And tried to create feed for my test page:
https://music.slavikf.com/logs/test2.php

And I got HTTP 500 for /setup_create_feed page

Request body:

{"html":["html",{"tag-id":"1"},[["body",{"tag-id":"2"},[["br",{"tag-id":"3"},[]],["br",{"tag-id":"4"},[]],["br",{"tag-id":"5"},[]],["br",{"tag-id":"6"},[]],["br",{"tag-id":"7"},[]],["br",{"tag-id":"8"},[]]]]]],"names":{"title":"2","description":"2"},"url":"https://music.slavikf.com/logs/test2.php"}

Response:

IntegrityError at /setup_create_feed
(1364, "Field 'owner_id' doesn't have a default value")

Request Method: POST
Request URL: http://192.168.0.203:8000/setup_create_feed
Django Version: 1.8.6
Python Executable: /usr/bin/python
Python Version: 2.7.12
Python Path: ['/var/www/pol/frontend', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']
Server time: Sun, 30 Jun 2019 06:02:36 +0000
Installed Applications:
('django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.sites',
 'pipeline',
 'frontend')
Installed Middleware:
('django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.middleware.locale.LocaleMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.security.SecurityMiddleware')

Traceback:
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py" in get_response
  132.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/var/www/pol/frontend/frontend/views.py" in setup_create_feed
  129.         feed_id = _create_feed(url, xpathes)
File "/var/www/pol/frontend/frontend/views.py" in _create_feed
  102.     feed.save()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py" in save
  734.                        force_update=force_update, update_fields=update_fields)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py" in save_base
  762.             updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py" in _save_table
  846.             result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py" in _do_insert
  885.                                using=using, raw=raw)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py" in manager_method
  127.                 return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py" in _insert
  920.         return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py" in execute_sql
  974.                 cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py" in execute
  79.             return super(CursorDebugWrapper, self).execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py" in execute
  64.                 return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py" in __exit__
  98.                 six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py" in execute
  64.                 return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py" in execute
  124.             return self.cursor.execute(query, args)
File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py" in execute
  226.             self.errorhandler(self, exc, value)
File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py" in defaulterrorhandler
  36.         raise errorvalue

Exception Type: IntegrityError at /setup_create_feed
Exception Value: (1364, "Field 'owner_id' doesn't have a default value")
Request information:
GET: No GET data

POST: No POST data

FILES: No FILES data
...

Same page works fine on public site

Feature Request: Website Authentication to get data behind login screen

Is it possible to implement website authentication OR custom cookie feature, to fetch a site that's behind a website login screen? Some websites protect their login form with recatcha2, so I think a custom cookie feature would be more easy to login. Please!

If it won't get implemented, can someone recommended an alternative to politepol that has this feature? This is really necessary feature.

Can "title" and "description" come from different pages?

I want to create a RSS feed with "title" coming from a list of articles in the site's home, and the description coming from inside the article (no information given in the home). Is it possible for the description to follow the "title's" link and get the description xpath from there? If not, this feature would be great

Is it possible to add this functionality?

Is it possible to add functionality so that politepol extracts all the links from the given webpage if the URL contains /news/ or /article/ or anything like that..

e.g target page http://www.example.com/ilovelife.php
in the target page there are 50 links which have /article/ in their URL..

Is it possible to make politepol...

Is it possible to make politepol run javascript on pages that we try to create feed from?

Review

It's been almost a week that I've been using PolitePol... I have to note that the script is very efficient. I haven't done any reboots so this is the current memory status (there isn't even swap for just in case moments)

root@vultr:~# free -h
total used free shared buff/cache available
Mem: 996M 334M 61M 10M 599M 484M
Swap: 0B 0B 0B
root@vultr:~#

more...

Tasks: 85 total, 1 running, 84 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.3 us, 0.7 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

Overall after almost a week of non-stop usage on a low-end vps with 1GB ram just 334MB of ram is being used and 99% of the CPU is in idle state..

P.S: Each "feed" created by politepol is being pinged by another script on another server every 15 minutes to get the latest updates.

I believe that from the perspective of an end-user like me the results are superb at the moment. Hopefully, I'll be adding performance updates in a weekly manner for another 3 weeks.

dockerized version

there is the possibility of a dockerized version of politepol?

Feeds aren't available

The last days I see this when creating or previewing a feed:

Something wrong. Reload the page or contact us by email: [email protected]

Are you aware of that?