glynnbird / birdreader Goto Github PK

Adding a feed URL via the web form will return an error.

By example

Could not add feed for http://feeds.mashable.com/Mashable

I've gotten the reader to run, but when I hit the page in my browser I get:

TypeError: /Users/mike/dev/birdreader/views/index.jade:13
    11|         h1 #{title}
    12|         p 
  > 13|         p #{articles.length} #{title} articles
    14|         p 
    15|         div.spacer
    16|       

Cannot read property 'length' of undefined
at eval (eval at (/Users/mike/dev/birdreader/node_modules/jade/lib/jade.js:176:8), :307:40)
at /Users/mike/dev/birdreader/node_modules/jade/lib/jade.js:181:12
at Object.exports.render (/Users/mike/dev/birdreader/node_modules/jade/lib/jade.js:216:14)
at View.exports.renderFile as engine
at View.render (/Users/mike/dev/birdreader/node_modules/express/lib/view.js:75:8)
at Function.app.render (/Users/mike/dev/birdreader/node_modules/express/lib/application.js:504:10)
at ServerResponse.res.render (/Users/mike/dev/birdreader/node_modules/express/lib/response.js:753:7)
at /Users/mike/dev/birdreader/birdreader.js:48:11
at /Users/mike/dev/birdreader/includes/article.js:16:5
at Object.exports.handle (/Users/mike/dev/birdreader/node_modules/nano/node_modules/errs/lib/errs.js:198:5)

When I look at my local CouchDB in Futon, I din't see an birdreader db's, so I created a "feeds" and an "articles" db, did an import of the sample xml file (nothing imported), and even inserted your sample docs from the .md file....but still the same error.

I'm running on a Mac.

Oh, one other dependency I learned - I needed to install 'make' before the npm install would work, until then I got errors when (I think it was) extractor-js wanted to run make.

Thanks,
Mike

any chance of being able to filter the articles list when clicking on the feed name in the article list to only show articles for that feed only - similar to when you click on a tag.

Having installed on my local server, I load port 3000. All I see is the menu bar, and the terminal return the following:

    10|     div.span10
    11|       div.feedName

> 12|         a.feedlink(href="/"+type+"/byfeed/"+escape(article.feedName.replace("/","%2F"))) @#{article.feedName}
>     13|         if article.icon != null
>     14|           img.miniicon(src=article.icon)
>     15|         span.tags.ml10

Cannot call method 'replace' of undefined
    at eval (eval at <anonymous> (/var/www/birdreader/node_modules/jade/lib/jade.js:176:8), <anonymous>:66:70)
    at /var/www/birdreader/node_modules/jade/lib/jade.js:181:12
    at Object.exports.render (/var/www/birdreader/node_modules/jade/lib/jade.js:216:14)
    at View.exports.renderFile [as engine](/var/www/birdreader/node_modules/jade/lib/jade.js:243:13)
    at View.render (/var/www/birdreader/node_modules/express/lib/view.js:75:8)
    at Function.app.render (/var/www/birdreader/node_modules/express/lib/application.js:504:10)
    at ServerResponse.res.render (/var/www/birdreader/node_modules/express/lib/response.js:753:7)
    at /var/www/birdreader/birdreader.js:312:9
    at /var/www/birdreader/node_modules/async/lib/async.js:227:13
    at /var/www/birdreader/node_modules/async/lib/async.js:107:25```

I tried to install birdreader, I've configured my cloudant db correctly, i've created the feeds, articles and views dbs, bur when I finally try to launch birdreader using node birdreader.js, it throws me an error

/var/lib/stickshift/51f282bd5004461ccc000049/app-root/data/569723/includes/cloudant.js:11                                                                                                                       
var feeds = nano.db.use('feeds');                                                                                                                                                                               
                    ^                                                                                                                                                                                           
TypeError: Cannot call method 'use' of undefined                                                                                                                                                                
    at Object.<anonymous> (/var/lib/stickshift/51f282bd5004461ccc000049/app-root/data/569723/includes/cloudant.js:11:21)                                                                                        
    at Module._compile (module.js:449:26)                                                                                                                                                                       
    at Object.Module._extensions..js (module.js:467:10)                                                                                                                                                         
    at Module.load (module.js:356:32)                                                                                                                                                                           
    at Function.Module._load (module.js:312:12)                                                                                                                                                                 
    at Module.require (module.js:362:17)                                                                                                                                                                        
    at require (module.js:378:17)                                                                                                                                                                               
    at Object.<anonymous> (/var/lib/stickshift/51f282bd5004461ccc000049/app-root/data/569723/birdreader.js:3:16)                                                                                                
    at Module._compile (module.js:449:26)                                                                                                                                                                       
    at Object.Module._extensions..js (module.js:467:10) 

I've tried to reinstall nano, but it didn't worked
Thanks in advance if someone have any ideas.

Sorry I seem to be creating a few issues for you - after running for a while i get a crash as follows:

Listening on port 3000

Checking articles database
Checking views

/usr/home/mike/birdreader/node_modules/async/lib/async.js:22
if (called) throw new Error("Callback was already called.");
^
Error: Callback was already called.
at /usr/home/mike/birdreader/node_modules/async/lib/async.js:22:31
at /usr/home/mike/birdreader/node_modules/async/lib/async.js:224:17
at /usr/home/mike/birdreader/node_modules/async/lib/async.js:507:34
at /usr/home/mike/birdreader/includes/feed.js:71:7
at FeedParser. (/usr/home/mike/birdreader/includes/feed.js:62:9)
at FeedParser.EventEmitter.emit (events.js:95:17)
at /usr/home/mike/birdreader/node_modules/feedparser/main.js:967:15
at process._tickCallback (node.js:415:13)

Currently, the update sequence number for my feeds database is 2.5 million. This is maybe more than it needs to be, and it degrades performance by growing the database file way too much. The un-beefy machine I am running the database on also can't afford the disk space :)

On my machine it also seems that the daily compaction (https://github.com/glynnbird/birdreader/blob/master/birdreader.js#L60) as well as the automatic compaction configured in my CouchDb installation doesn't really prevent the feeds database file from growing up into GBs of space, when 1.2 MB is sufficient. I don't know why that is. Maybe it just grows that much during a normal day?

I suspect that the culprit is this call here: https://github.com/glynnbird/birdreader/blob/master/includes/feed.js#L156

Am I correctly interpreting this to mean that all the documents in the feeds database are written every time feeds are fetched?

It seems that a better solution would be to only write the feeds documents that have changed.

I can probably implement this some day, but I wanted to check if I have understood correctly and if you agree :)

Apparently, you are supposed to pass the URL to a web page which has a <link> to an RSS feed.

This is all well and good, but it should also accept URLs that go directly to RSS feeds, since that is after all what it requires in the end. It might even be necessary to disambiguate when a web page has several RSS links.

As it happens, I've been trying to add RSS URLs for a long while and just sighed and walked away when I inevitably got a nondescript error message (See also #21)

If the database stores an invalid feed URL, it takes down the whole process.

info: socket.io started
Checking feeds database
Listening on port 3000
Checking articles database
Checking views

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: Invalid URI "files/feed.xml"
    at Request.init (/home/mag/prog/traume-roat/deploy/birdreader/node_modules/request/request.js:186:24)
    at new Request (/home/mag/prog/traume-roat/deploy/birdreader/node_modules/request/request.js:105:8)
    at request (/home/mag/prog/traume-roat/deploy/birdreader/node_modules/request/index.js:53:11)
    at fetchFeed (/home/mag/prog/traume-roat/deploy/birdreader/includes/feed.js:47:3)
    at /home/mag/prog/traume-roat/deploy/birdreader/includes/feed.js:123:11
    at /home/mag/prog/traume-roat/deploy/birdreader/node_modules/async/lib/async.js:508:21
    at /home/mag/prog/traume-roat/deploy/birdreader/node_modules/async/lib/async.js:224:13
    at replenish (/home/mag/prog/traume-roat/deploy/birdreader/node_modules/async/lib/async.js:176:21)
    at /home/mag/prog/traume-roat/deploy/birdreader/node_modules/async/lib/async.js:188:33
    at /home/mag/prog/traume-roat/deploy/birdreader/node_modules/async/lib/async.js:226:17

When adding an URL with a <link> element with a relative href attribute, birdreader assumes that this link is absolute.

Instead, it should be resolved as relative to the current document, for example using url.resolve(htmlUrl, feedUrl).

Example of problematic page: http://jimbenton.com/page14/page14.html

Excerpt: <link rel="alternate" type="application/rss+xml" title="Comics" href="files/feed.xml"/>

Notice that the href attribute is relative, and the absolute URL to the RSS feed is therefore http://jimbenton.com/page14/files/feed.xml

IF I am adding a feed
AND BirdReader fails to find the RSS url in the page supplied
THEN I should get some feedback to say that it failed.

I have left birdreader running for a few days with about 80 feeds.

Initially the db size is < 1mb but after its left for a few days it has grown to about 30 mb.

A new schedule which runs daily to compact the feeds db automatically would stop the feeds db growing too large.

At the moment I'm using tt-rss with multiple users, but the web interface is not really nice and far from being responsive or at least useable from a mobile device.

I'd like to have a nodejs reader, but I need multi-user support...

if an article doesnt contain a description i get the following crash:

TypeError: Cannot call method 'replace' of undefined
at processArticles (/usr/home/mike/birdreader/birdreader.js:92:42)
at /usr/home/mike/birdreader/birdreader.js:110:20
at /usr/home/mike/birdreader/node_modules/async/lib/async.js:227:13
at /usr/home/mike/birdreader/node_modules/async/lib/async.js:107:25
at /usr/home/mike/birdreader/node_modules/async/lib/async.js:24:16
at /usr/home/mike/birdreader/node_modules/async/lib/async.js:224:17
at /usr/home/mike/birdreader/node_modules/async/lib/async.js:507:34
at /usr/home/mike/birdreader/includes/article.js:18:5
at Request._callback (/usr/home/mike/birdreader/node_modules/nano/nano.js:296:11)
at Request.self.callback (/usr/home/mike/birdreader/node_modules/nano/node_modules/request/main.js:120:22)

adding "description": "" to any article docs will fix any articles which cause this

the temp view below will help find any docs with out a description:

function(doc) { if(!doc.description){emit(doc); } }

Do you have instructions for running birdreader on server startup?

If i try to remove a feed the address bar changes to:

javascript:removeFeed('f771ac4b0f352ddb82acfc2683028f2e')

and the document window contains "false"

I get a similar issue when deleting tags from a feed and the address bar becomes:

javascript:removeTag('f771ac4b0f352ddb82acfc268300bb98','something')

again with false as the page content

I'm using FireFox

add date/time of when article was published to article list in title for a bit more information

I've done npm i and tweaked just the cloudant bits in config.json as well as setup an API key/secret with _admin privs on both the articles and feeds databases.

But I get this when running:

$ node birdreader.js
   info  - socket.io started
Checking feeds database
Listening on port 3000
Database error - server_admin access is required for this request

I've double checked the _security objects and confirmed that the API key is setup as an _admin.

Is it hitting a different database that needs to be writeable via this key?

Cheers!
🎩

I think this crash is happening while feeds are being read without me using the app in a browser.

Checking feeds database
Listening on port 3000
Checking articles database
Checking views

crypto.js:176
this._binding.update(data);
^
TypeError: Not a buffer
at Hash.update (crypto.js:176:17)
at FeedParser. (/usr/home/mike/birdreader/includes/feed.js:42:16)
at FeedParser.EventEmitter.emit (events.js:95:17)
at /usr/home/mike/birdreader/node_modules/feedparser/main.js:967:15
at process._tickCallback (node.js:415:13)

Most/all feed readers have a sidebar showing the subscriptions. In birdreader this would be a listing of the feed tags, and perhaps subsidiary to each feed tag would be the feeds that have that tag.

By having this list it would become easy to navigate to showing the articles for a given feed tag.

I have made a small change to the article.js to allow for searching for normal couchdb setups via elasticsearch and the couchdb river
https://github.com/elasticsearch/elasticsearch-river-couchdb

Once the river plugin is installed setup the river for articles:

curl -XPUT 'http://elasticsearch_host:9200/_river/articles/_meta' -d '{
"type" : "couchdb",
"couchdb" : {
"host" : "couchdb_host",
"port" : 5984,
"db" : "articles"
}

Install elasticsearch node client

npm install --save elasticsearch

Changes to: includes/article.js

var articles = require('./cloudant.js').articles;

var moment = require('moment');
var async = require('async');
var mc = require('./mc.js');
var elasticsearch = require('elasticsearch');

Replace the search part from line 322

var search = function (keywords, callback) {
var client = new elasticsearch.Client( { host: 'elasticsearch_host:9200', log: 'info' } );
client.search({
q: keywords, index: "articles", size: 99
}).then(function (body) {
retval = [];
for (i = 0; i < body.hits.hits.length; i++) {
retval.push(body.hits.hits[i]._source);
}
callback(false, retval);
}, function (error) {
console.trace(error.message);
});
};

Hi there,

For some reason I'm not able to save any feeds using 'add' button. I receive 'Successfully added feed' response, but the feeds is actually not added.

Here is the link to the deployment: http://aggr-mhlab.rhcloud.com/

Any advice would be appreciated.

Many thanks,
Max

if there are only 3 or 4 unread articles (eg after filtering by feed) is it possible to have an expand all button which opens all of the articles listed and marks them as read

Following the instructions in README.md:
npm install

This throws the error:
Unsupported
npm ERR! Not compatible with your version of node/npm: [email protected]
npm ERR! Required: {"node":">= 0.8.0"}
npm ERR! Actual: {"npm":"1.1.4","node":"0.6.12"}

OS: Ubunto 12.04
node -v v0.6.12
npm -v 1.1.4

Thanks.

i installed mocha and tried to run the tests but it failed, below is the log..

    throw err;
    ^

Error: Cannot find module 'should'
    at Function.Module._resolveFilename (module.js:485:15)
    at Function.Module._load (module.js:437:25)
    at Module.require (module.js:513:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (H:\Projects\newExperiments\birdreader\test\api.js:5:12)
    at Module._compile (module.js:569:30)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)
    at Module.require (module.js:513:17)
    at require (internal/module.js:11:18)
    at C:\Users\salus\AppData\Roaming\nvm\v8.0.0\node_modules\mocha\lib\mocha.js:231:27
    at Array.forEach (native)
    at Mocha.loadFiles (C:\Users\salus\AppData\Roaming\nvm\v8.0.0\node_modules\mocha\lib\mocha.js:228:14)
    at Mocha.run (C:\Users\salus\AppData\Roaming\nvm\v8.0.0\node_modules\mocha\lib\mocha.js:536:10)
    at Object.<anonymous> (C:\Users\salus\AppData\Roaming\nvm\v8.0.0\node_modules\mocha\bin\_mocha:573:18)
    at Module._compile (module.js:569:30)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)
    at Function.Module.runMain (module.js:605:10)
    at startup (bootstrap_node.js:158:16)
    at bootstrap_node.js:575:3

I personally need to be able to tag each article ... and to browse articles by tag.

The API should have related methods to retrieve articles by article tags.

Have glanced through the code and see it would be easy to add this to the schema. Then a dialog for adding tags, ..etc..

Hi - greader convert here - very happy to have found birdreader - thanks.

I have noticed a minor issue with the feed fetching only happening once at .15 past the hour - I think the schedule rule should be:

rule.minute = [0,15,30,45]

in order for the feeds to be updated every 15 minutes.

This maybe a setup issue but I installed CouchDB on my RaspberryPi and I'm able to access the admin panel it via my server's external IP.

I've followed your instructions to the letter and everything is setup, running and listening but I can't see anything at http://myserversip:3000
Here's is my config - (server named here as "myserverip"):

{
"cloudant": {
"server": "http://myserverip",
"port": 5489,
"username": "",
"password": "",
"secure": true
},
"authentication": {
"on": false,
"username": "",
"password": ""
},
"purgeArticles": {
"on": false,
"purgeBefore": 15
}
}