nerdyator / feedjack Goto Github PK

hi Tabo..
I noticed feedjack was complaining when some of the strings were already in
UTF-8
eg...
 'Perennial Lamb\xe2\x80\x99s Ears'

this fixes that issue with a sledgehammer ;-) .. you might have a better
approach.

def encode(tstr):
    """ Encodes a unicode string in utf-8
    """
    if not tstr:
        return ''
    try:
        return tstr.encode('utf-8', "xmlcharrefreplace")
    except UnicodeDecodeError:
        # it's already UTF8.. sigh
        return tstr.decode('utf-8').encode('utf-8')

hi Tabo..
I noticed feedjack was complaining when some of the strings were already in
UTF-8
eg...
 'Perennial Lamb\xe2\x80\x99s Ears'

this fixes that issue with a sledgehammer ;-) .. you might have a better
approach.

def encode(tstr):
    """ Encodes a unicode string in utf-8
    """
    if not tstr:
        return ''
    try:
        return tstr.encode('utf-8', "xmlcharrefreplace")
    except UnicodeDecodeError:
        # it's already UTF8.. sigh
        return tstr.decode('utf-8').encode('utf-8')

On 'Add feed' in the admin:

If this is disabled, *ths* feed will not be further updated.

Also it shouldn't say this twice, "If disabled" would be enough.

In the home page, under "Configuration", you say: "Now that you have
everything set up, just run the feedjackupdate.py" , but the script is
called feedjack_update.py instead of feedjackupdate.py. 

And in that same page it is not sayed that in order to have a planet
running you have to include 'feedjack' in settings.py under INSTALLED_APPS. 

Greetings

It has been requested that Feedjack should have filters both positive and
negative for posts to appear in a site.

A positive filter means that only posts that meet a certain criteria should
appear in a site.

A negative filter means that if a post meets a certain criteria, it should
not appear in the site.

These filters should exist both at the site and at the subscriber level.

It has been suggested in
http://code.google.com/p/feedjack/issues/detail?id=3  to use regular
expressions. Filters could be done this way and/or with hooks written in
python.

This issue affects the feedjack_update.py script.  It uses the deprecated
entries[i].categories rather than entries[i].tags which causes the tags for
atom feeds to show up as integers rather than terms.  See
http://feedparser.org/docs/reference-entry-tags.html for more information
about the deprecation.

Very minor issue.  Looks like version 0.6a10 is not available at 
http://cheeseshop.python.org/packages/2.3/s/setuptools/

I think a change like this should do the trick in ez_setup.py:

DEFAULT_VERSION = "0.6c6"
#DEFAULT_VERSION = "0.6a10"

What steps will reproduce the problem?
1. Run the install (python setup.py install)

What is the expected output? What do you see instead?
404 Error when downloading setuptools

What version of the product are you using? On what operating system?
Linux (on dreamhost.com shell) + Feedjack-0.9.9.tar.gz

Please provide any additional information below.
Minor issue and easy workaround (really just giving an FYI)

In the "def get_tags(entry, tagdict)" function in feedjack_update.py file
in Feedjack 0.9.7, the lines 51-52:

while '  ' in tagname:
     tagname.replace('  ', ' ')

should instead read:

while '  ' in tagname:
     tagname = tagname.replace('  ', ' ')

otherwise any tagname string that contains '  ' will cause an infinite loop.

What steps will reproduce the problem?
1. Install FJ
2. Enable default template
3. View in Internet Explorer

There's a problem with the width property in IE that makes the post content
too narrow and destroys the layout.

Italian l10n attached. To be added in a 'locale/it/LC_MESSAGES/' directory.

Some blog engines provides per-category feeds but that's not always 
enough. Imagine you want to aggregate two categories from a blog, or you 
want to get all entries whose subjects match /foo/.

Being able to filter attributes (description, category, ...) with regular 
expressions would be a really nice addition.

What steps will reproduce the problem?
1. Add
http://www.amazon.com/rss/people/A1V4UQ0XZXLCGC/reviews/ref=cm_rss_member_rev_ma
nlink
 as a feed.
2. Run feedjack_update.py
3. SQL Error.  Data to large for column.

What is the expected output? What do you see instead?
Comments field in the feedjack_post table is not large enough to handle
large comments.  Would expect large comments to be trimmed down to column
size or column size to be increased or friendly error stating feed has
incompatibilities with database table.  Guid and Link may also have this
problem.

What version of the product are you using? On what operating system?
Fedora Core 6
Django-0.96.1-1.fc6
postgresql-server-8.1.9-1.fc6
Feedjack 0.9.10

Please provide any additional information below.
modified my db table to have guid and link be varchar(255) and comments to
a text field.  Works as expected.

What steps will reproduce the problem?
1. feedjack_update.py failed with the problem below, when trying to update
http://www.securityfocus.com/rss/vulnerabilities.xml feed. 
This feed has some items with <pubDate> tag but the pubDate is empty.
You can use the feed above to reproduce the problem.

What is the expected output? What do you see instead?
The error is
['Traceback (most recent call last):\n', '  File
"/usr/lib/python2.5/site-packages/Feedjack-0.9.16-py2.5.egg/EGG-INFO/scripts/fee
djack_update.py",
line 313, in process\n    ret_entry = self.process_entry(entry,
postdict)\n', '  File
"/usr/lib/python2.5/site-packages/Feedjack-0.9.16-py2.5.egg/EGG-INFO/scripts/fee
djack_update.py",
line 217, in process_entry\n    ret_entry = entry.process()\n', '  File
"/usr/lib/python2.5/site-packages/Feedjack-0.9.16-py2.5.egg/EGG-INFO/scripts/fee
djack_update.py",
line 143, in process\n    fcat, comments) = self.get_entry_data()\n', ' 
File
"/usr/lib/python2.5/site-packages/Feedjack-0.9.16-py2.5.egg/EGG-INFO/scripts/fee
djack_update.py",
line 127, in get_entry_data\n    date_modified =
mtime(self.entry.modified_parsed)\n', '  File
"/usr/lib/python2.5/site-packages/Feedjack-0.9.16-py2.5.egg/EGG-INFO/scripts/fee
djack_update.py",
line 55, in mtime\n    return
datetime.datetime.fromtimestamp(time.mktime(ttime))\n', 'TypeError:
argument must be 9-item sequence, not None\n']

What version of the product are you using? On what operating system?
Feedjack-0.9.16 on Linux with Python 2.5 with Django 1.0 stable version.

Please provide any additional information below.

FIX
===
To fix this issue, I changed the 126th line as
if self.entry.has_key('modified_parsed') and self.entry.modified_parsed:

Best regards

This problem should be especially obvious after the first feeds' update
either if posts have "dynamic" timestamps (generated along with the feed)
or don't include these at all, or if the posts' ordering is set to "Date
fetched".

Feed "channels" will get mixed randomly since most posts' should contain
exactly same timestamps and there's no ordering criteria past "date_modified".
If updater is run with the long-enough intervals and works pretty fast, the
problem should persist after the initial update.

As a solution, I've added extra "feed" criteria past "date_modified", so
the posts from the different feeds but with the same date won't get mixed
like that.

Feedjack should allow the visitors to "rate" the content of the site. This
could be done with "stars" (web 1.0) or "up/down" votes a-la digg (web 2.0).

here two patches i am using. enjoy.

1) add group setting to feeds, so you can view a group instead of all sources. 
/group/id/
2) view posts newer than yyyy-mm-dd /newer/yyyy-mm-dd/

Would be great to be able to upload feedjack to gae.
needs a port to appengine-patch (aka django 1.1 on GAE).
Probably, it's not an easy thing as ORM -> gae db.Model (nosql) is painful.

How about adding some i18n hooks in models/templates?

I'ld like to translate feedjack to Serbian

The home page doesn't state that feedjack is open source.  There is one
place, if you scroll down a bit, where it says "License: BSD" -- but
there's no link to the actual license text, so it's not clear which of the
BSD licenses is meant.  Most likely it's the modern, 3-clause version (?),
but it might be better to be explicit (since there are important differences).

http://en.wikipedia.org/wiki/BSD_licenses has more background on the
various versions of the BSD license.

Personally, I think it would be good to have a clear statement on the front
page that "feedjack is open source software".  But just being unambiguous
about the license would also help a lot.

Best,
-Karl

In the clone I have created a new templatetag which allows easy embedding
of a single feed on for example your front page.

Maybe it could be considered moving all pages to this template tag like
structure for much greater flexibility and embeddability. 

There is a space missing in the main template, near "talks about". See the
patch for details.

Now that Django allows apps to create custom command wouldn't it be an
option to run feedjack_update like:

    ./manage.py feedjack_update [OPTIONS]

It seems this shouldn't be too complicated by looking at the source of
runserver:

http://code.djangoproject.com/browser/django/trunk/django/core/management/comman
ds/runserver.py

More here:
    http://docs.djangoproject.com/en/dev/howto/custom-management-commands/

Looking at the code of django-mailer for an example that is quite analogous
to what feedjack is trying to do:

http://github.com/jtauber/django-mailer/blob/master/mailer/management/commands/s
end_mail.py

I've found whole "channel" block faulty in this template, at least for my
feeds-collection.
Looks like it shouldn't be working right at all, and that's probably masked
with the default theme, which doesn't style "channel" blocks.

Right now it looks like this:

{% for item in object_list %}
...
{% ifchanged %}
<!-- {{ item.date_modified|date:"F j, Y" }} -->
<div class="channel">
    ...(feed header)...
{% endifchanged %}

...(entry)...

{% ifchanged %}
<!-- {{ item.date_modified|date:"F j, Y" }} {{ item.feed.link }} -->
<!-- End .channel -->
<br class="clear"/>
</div>
{% endifchanged %}
...
{% endfor %}

Problem is that if two distinct feeds are from one portal, they often have
the same link, so the "channel" block doesn't get closed, producing deep
tag nesting if this feeds' posts mix frequently enough.
For me it also cause major design breakage, but I'm using a bit modified
sinx theme, so it might be not so bad for vanilla one.
Another problem with the whole approach is that the "channel" blocks get
closed in completely wrong order - closing "ifchanged" part gets triggered
at the new "channel" entry, closing just-opened "channel', not the one that
was opened before that. Furthermore, last "channel" don't get closed at all.

The solution is to either use feed.id as the "ifchanged" key, since it's
guaranteed-unique between the feeds, or add feed.link to the tag-opening
key, if such aggregation is intentional (although I doubt that, since whole
thing just looks like a faulty template logic).

I've changed it like this:

{% for item in object_list %}
...
{% with item.date_modified|date:"F j, Y" as date_key %}
{% ifchanged item.feed.id date_key %}
{% if not forloop.first %}
    <!-- End .channel -->
    <br class="clear"/>
</div>
{% endif %}
<div class="channel">
    <h2><a href="{{ item.feed.link }}" title="{{ item.feed.title }}">{{
item.feed.shortname }}</a></h2>
        ...
{% endifchanged %}
{% endwith %}

...(entry)...

{% if forloop.last %}
    <!-- End .channel -->
    <br class="clear"/>
</div>
{% endif %}
...
{% endfor %}

That way, the "ifchanged" content doesn't have to contain some special
html-comment magic and can actually contain any random stuff, yet still be
echoed only where it's intended to. Channels should close in the correct
order and last one should be properly closed as well.
Whole "with ... as date_key" wrapping is necessary because of this django
bug: http://code.djangoproject.com/ticket/5756

Add a text field for each subscriber, so they can send in an intro to be
used in the planet.

Django trunk has merged with UnicodeBranch and for feedjack works with last
trunk revision, you need make the code compatible. Read this:
http://code.djangoproject.com/wiki/UnicodeBranch#PortingApplicationsTheQuickChec
klist

Thanks!

I can't run syncdb. See:

$ python2.4 manage.py syncdb
Error: Couldn't install apps, because there were errors in one or more models:
feedjack: unsupported operand type(s) for +: '__proxy__' and 'str'


I uses Django trunk revision 5625, python2.4 and last feedparser. My
feedjack version is 0.9.10.

What steps will reproduce the problem?
1.
2.
3.

What is the expected output? What do you see instead?


Please use labels and text to provide additional information.

Support for Grouping Subscribes together (with subgroups) would be cool.

A simple model like this should do:
class Group:
  parentgroup=models.ForeignKey(Group,blank=True,null=True)
class Subscriber:
  group=models.ForeignKey(Group, blank=True, null=True)

and a url /group/groupname for getting group-feed-entries

test

Feedjack should allow visitors to add comments in any entry.

Some antispam system will be needed (probably the python hook for akismet).

Allow importing feeds from an OPML file. It would be a time saver to
populate initial feeds when setting up feedjack.

What is the expected output? What do you see instead?
When reading posts of a given tag, the context should also provide a list
of the other users who have also written posts about the tag.

What version of the product are you using? On what operating system?
0.9.7

The ObjectPaginator utility has been removed completely from the most
modern versions of Django.  In preparation for the official release of
Django 1.0, I would switch to Paginator.  Maybe you could do something like...

{{{
try:
    from django.core.paginator import Paginator
except ImportError:
    from django.core.paginator import ObjectPaginator as Paginator
}}}

...to help maintain a little backwards compatibility.

regarding http://code.google.com/p/feedjack/issues/detail?id=16 

This code was not tested before being submitted.  One of the changes is in
the actual structure of what a tag is, and it is different from a
(deprecated) category.  Right now it's broken.

That would be a killer-feature.

Aggregated content get to the moderated category until it is either 
approved or deleted.

I don't think that would be too hard to do but I'm not really familiar 
with feedjack code.

I'd like to have a per-subscriber option of removing URLs and only
displaying the first N characters from each article.  Readers would have to
click through to the blog to read the complete article, get the links, etc.

This could be handled via the requested regex support.  I'd like to keep it
more simple than that.

I'm currently working on this functionality (for planetscm.org), and can
submit the code changes if anyone is interested.

What steps will reproduce the problem?
1. Try to fetch a livejournal.com's feed with non-english content.

What is the expected output? What do you see instead?

Just this traceback. I don't reveal the wrong feed because it's not mine, 
and I don't know if the blog's author want to be know.

[17] ! -------------------------
['Traceback (most recent call last):\n', '  File "/usr/lib/python2.4/site-
packages/Feedjack-0.9.16-py2.4.egg/EGG-INFO/scripts/feedjack_update.py", 
line 378, in process_feed_wrapper\n    ret_feed, ret_entries = 
pfeed.process()\n', '  File "/usr/lib/python2.4/site-packages/Feedjack-
0.9.16-py2.4.egg/EGG-INFO/scripts/feedjack_update.py", line 305, in 
process\n    postdict = dict([(post.guid, post)\n', '  File 
"/usr/lib/python2.4/site-packages/django/db/models/query.py", line 179, in 
_result_iter\n    self._fill_cache()\n', '  File "/usr/lib/python2.4/site-
packages/django/db/models/query.py", line 612, in _fill_cache\n    
self._result_cache.append(self._iter.next())\n', '  File 
"/usr/lib/python2.4/site-packages/django/db/models/query.py", line 269, in 
iterator\n    for row in self.query.results_iter():\n', '  File 
"/usr/lib/python2.4/site-packages/django/db/models/sql/query.py", line 206, 
in results_iter\n    for rows in self.execute_sql(MULTI):\n', '  File 
"/usr/lib/python2.4/site-packages/django/db/models/sql/query.py", line 
1720, in execute_sql\n    return list(result)\n', '  File 
"/usr/lib/python2.4/site-packages/django/db/models/sql/query.py", line 
1714, in <lambda>\n    result = iter((lambda: 
cursor.fetchmany(GET_ITERATOR_CHUNK_SIZE)),\n', '  File 
"/usr/lib/python2.4/site-packages/django/db/backends/util.py", line 53, in 
typecast_date\n    return s and datetime.date(*map(int, s.split(\'-\'))) or 
None # returns None if s is null\n', "OperationalError: Could not decode to 
UTF-8 column 'title' with text '-\xa4\xd5'\n"]
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/Feedjack-0.9.16-py2.4.egg/EGG-
INFO/scripts/feedjack_update.py", line 378, in process_feed_wrapper
    ret_feed, ret_entries = pfeed.process()
  File "/usr/lib/python2.4/site-packages/Feedjack-0.9.16-py2.4.egg/EGG-
INFO/scripts/feedjack_update.py", line 305, in process
    postdict = dict([(post.guid, post)
  File "/usr/lib/python2.4/site-packages/django/db/models/query.py", line 
179, in _result_iter
    self._fill_cache()
  File "/usr/lib/python2.4/site-packages/django/db/models/query.py", line 
612, in _fill_cache
    self._result_cache.append(self._iter.next())
  File "/usr/lib/python2.4/site-packages/django/db/models/query.py", line 
269, in iterator
    for row in self.query.results_iter():
  File "/usr/lib/python2.4/site-packages/django/db/models/sql/query.py", 
line 206, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/usr/lib/python2.4/site-packages/django/db/models/sql/query.py", 
line 1720, in execute_sql
    return list(result)
  File "/usr/lib/python2.4/site-packages/django/db/models/sql/query.py", 
line 1714, in <lambda>
    result = iter((lambda: cursor.fetchmany(GET_ITERATOR_CHUNK_SIZE)),
  File "/usr/lib/python2.4/site-packages/django/db/backends/util.py", line 
53, in typecast_date
    return s and datetime.date(*map(int, s.split('-'))) or None # returns 
None if s is null
OperationalError: Could not decode to UTF-8 column 'title' with text '-¤Õ'
[17] ! -------------------------
Traceback (most recent call last):
  File "/usr/bin/feedjack_update.py", line 5, in ?
    pkg_resources.run_script('Feedjack==0.9.16', 'feedjack_update.py')
  File "/usr/lib/python2.4/site-packages/setuptools-0.6c8-
py2.4.egg/pkg_resources.py", line 448, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.4/site-packages/setuptools-0.6c8-
py2.4.egg/pkg_resources.py", line 1166, in run_script
    execfile(script_filename, namespace, namespace)
  File "/usr/lib/python2.4/site-packages/Feedjack-0.9.16-py2.4.egg/EGG-
INFO/scripts/feedjack_update.py", line 505, in ?
    main()
  File "/usr/lib/python2.4/site-packages/Feedjack-0.9.16-py2.4.egg/EGG-
INFO/scripts/feedjack_update.py", line 489, in main
    disp.add_job(feed)
  File "/usr/lib/python2.4/site-packages/Feedjack-0.9.16-py2.4.egg/EGG-
INFO/scripts/feedjack_update.py", line 370, in add_job
    self.process_feed_wrapper(feed)
  File "/usr/lib/python2.4/site-packages/Feedjack-0.9.16-py2.4.egg/EGG-
INFO/scripts/feedjack_update.py", line 394, in process_feed_wrapper
    prints(u'[%d] Processed %s in %s [%s] [%s]%s' % (
  File "/usr/lib/python2.4/site-packages/Feedjack-0.9.16-py2.4.egg/EGG-
INFO/scripts/feedjack_update.py", line 397, in <generator expression>
    u' '.join(u'%s=%d' % (self.entry_trans[key],
KeyError: 0

What version of the product are you using? On what operating system?
$ django-admin.py --version
1.0-final-SVN-unknown
and
feedjack 0.9.16

Please provide any additional information below.

"manage.py syncdb" fails with
Error: Couldn't install apps, because there were errors in one or more 
models:
src.feedjack: unsupported operand type(s) for +: '__proxy__' and 'str'

Feedjack uses the deprecated md5 library: this causes a warning to be
raised on import on Python 2.6.

The attached patch avoids the warning and is compatible with previous
Python versions.

A few translation hooks are missing. See patch.

What steps will reproduce the problem?
1. Take a feed whose content contains non-ascii characters. 
2. Run feedjack_update.py for this feed.

I'm receiving a warning:

/bin/feedjack_update.py:158: UnicodeWarning: Unicode unequal comparison
failed to convert both arguments to Unicode - interpreting them as being
unequal
>  if tobj.content != content or (date_modified and

The feedjack.org page doesn't currently alert users to the fact that the
app doesn't work correctly with the latest Django trunk. Two changes that
need to be accounted for:

http://code.djangoproject.com/wiki/BackwardsIncompatibleChanges#Refactoreddataba
sebackends

The fix for this one is simply to replace "backend.quote_name" with
"connection.ops.quote_name" in fjlib.py.

http://code.djangoproject.com/wiki/BackwardsIncompatibleChanges#Auto-escapingint
emplates

The quick-fix for this is to wrap base templates in {% autoescape off %}
... {% endautoescape %}.

Steps to reproduce:
1. add a feed that contains articles with tags longer than 50 characters (eg. 
http://www.digital-copyright.ca/node/feed )
2. run feedjack_update.py

Script fails with multiple python tracebacks, the important part being:

Traceback (most recent call last):
  File
"/usr/local/lib/python2.3/site-packages/Feedjack-0.9.12-py2.3.egg/EGG-INFO/scrip
ts/feedjack_update.py",
line 247, in process_feed
    process_entry(entry, fpf, feed, postdict, options)
  File
"/usr/local/lib/python2.3/site-packages/Feedjack-0.9.12-py2.3.egg/EGG-INFO/scrip
ts/feedjack_update.py",
line 116, in process_entry
    (link, title, guid, author, author_email, content, date_modified, fcat, \
  File
"/usr/local/lib/python2.3/site-packages/Feedjack-0.9.12-py2.3.egg/EGG-INFO/scrip
ts/feedjack_update.py",
line 105, in get_entry_data
    fcat = get_tags(entry)
  File
"/usr/local/lib/python2.3/site-packages/Feedjack-0.9.12-py2.3.egg/EGG-INFO/scrip
ts/feedjack_update.py",
line 67, in get_tags
    cobj.save()
  File
"/usr/local/lib/python2.3/site-packages/Feedjack-0.9.12-py2.3.egg/feedjack/model
s.py",
line 147, in save
    super(Tag, self).save()
  File "/usr/local/lib/python2.3/site-packages/django/db/models/base.py",
line 261, in save
    ','.join(placeholders)), db_values)
  File
"/usr/local/lib/python2.3/site-packages/django/db/backends/postgresql/base.py",
line 47, in execute
    return self.cursor.execute(smart_str(sql, self.charset),
self.format_params(params))
ProgrammingError: ERROR:  value too long for type character varying(50)

INSERT INTO "feedjack_tag" ("name") VALUES ('getting open source logic into
government (gosling)')