srid / slownews Goto Github PK

View Code? Open in Web Editor NEW

113.0 8.0 11.0 1.52 MB

🐢 Bringing slow news for the nervous among us

Home Page: https://slownews.srid.ca

License: BSD 3-Clause "New" or "Revised" License

Haskell 91.32% Nix 8.68%

haskell reddit hackernews reflex-frp reflex obelisk

slownews's Introduction

slownews

A web app that aggregates top news during the last week from sites like reddit, Lobste.rs and Hacker News. Written in Haskell and using Reflex FRP (via Obelisk).

Running locally

Install obelisk and run:

ob run

Visit http://localhost:8000/

Deploying

To deploy to a remote (NixOS) server, use ob deploy (see Obelisk docs for details).

To create a release for running locally, run:

nix-build -A exe

and then to run that:

(cd result && ./backend -p 3001)

slownews's People

Stargazers

Watchers

Forkers

alex-marmot oeohomos rrevanth dnnx kgisl steshaw elgusy tarsbase baig xtr4mbor59 desonedex996

slownews's Issues

Make the scheduler reliable

Broken http requests (like the one below) leads to the scheduler stop working. Probably need to handle the exceptions reliably.

backend: HttpExceptionRequest Request {
  host                 = "hn.algolia.com"
  port                 = 80
  secure               = False
  requestHeaders       = [("User-Agent","haskell wreq-0.5.1.0")]
  path                 = "/api/v1/search"
  queryString          = "?tags=story&filters=num_comments%3E2%20AND%20created_at_i%3E1514851200&hitsPerPage=3&query=%22haskell%22"
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 ResponseTimeout

Upgrade to Elm 0.17

https://github.com/elm-lang/elm-platform/blob/master/upgrade-docs/0.17.md

Elm component refactoring

Refactor the Elm client to use a component model (like chronicle) before implementing upcoming client-side features.

Is there a builtin library/framework (more than start-app)?

Upgrade obelisk

ob thunk unpack .obelisk/impl
cd .obelisk/impl
git checkout develop && git pull
cd -
ob thunk pack .obelisk/impl

And then apply code changes necessary to use the new Obelisk. ob run should give the compilation errors for this.

Highlight recently added links

Sometimes a 3-day old link (it got popular suddenly?) would appear today. It makes sense to put this link at top, or highlight somehow in UI of its newness.

Handle timeouts in the Haskell backend

.. and other http errors. Just retry.

backend: HttpExceptionRequest Request {
  host                 = "www.reddit.com"
  port                 = 443
  secure               = True
  requestHeaders       = [("User-Agent","haskell wreq-0.5.1.0")]
  path                 = "/r/zerocarb/top/.json"
  queryString          = "?sort=top&t=week&limit=3"
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 (ConnectionFailure Network.Socket.getAddrInfo: does not exist (nodename nor servname provided, or not known))

SlowNews does not render without JavaScript

I disable JavaScript by default, and SlowNews seems to render everything in JS client-side.

Would it be possible to render things server-side at first? That would be likely much faster, as a bonus. :)

Add Github Trending repositories

https://github.com/trending?since=weekly

Interesting, I was just thinking about a slow news idea like this. In particular I thought it would be cool if I could get these kind of links emailed to me weekly. I understand that this may be way out of scope for your project but thought I'd mention it.

Save articles offline

Perhaps do it as separate project. A Haskell clone of wallabag (PHP, ughhh).

Icons for sites

For visual distinguishing.

HackerNews 'Active Threads'

https://news.ycombinator.com/active

A lot of HN threads with active discussion get suppressed from the front page (but made available in the active page). Figure out a way to get 'weekly best' active threads in SlowNews. Currently we get threads with top votes only.

Request timeouts

Oct 05 21:58:23 slownews heroku/router:  at=error code=H12 desc="Request timeout" method=GET path="/data" host=slownews.ca request_id=d3b49ebe-bc3f-4469-b8af-074822153c83 fwd="207.102.213.120" dyno=web.1 connect=1ms service=30000ms status=503 bytes=0

obviously Go is not enough. Go Elixir!

Oct 10 16:35:22 slownews-trot heroku/router:  at=info method=GET path="/data" host=slownews-trot.herokuapp.com request_id=6a975b5b-d658-444c-845c-11d3e10b497e fwd="207.102.213.120" dyno=web.1 connect=6ms service=13ms status=200 bytes=7571 
Oct 10 16:35:24 slownews-trot app/web.1:  {"Kernel pid terminated",application_controller,"{application_start_failure,slownews,{{shutdown,{failed_to_start_child,'Elixir.Slownews.Crawler',{#{'__exception__' => true,'__struct__' => 'Elixir.HTTPoison.Error',id => nil,reason => timeout},[{'Elixir.Slownews.Site.Reddit.Client','request!',5,[{file,\"lib/slownews.site.reddit.ex\"},{line,21}]},{'Elixir.Slownews.Site.Slownews.Site.Reddit',fetch,1,[{file,\"lib/slownews.site.reddit.ex\"},{line,11}]},{'Elixir.Slownews.Crawler.Util',fetchAndStore,1,[{file,\"lib/slownews.crawler.ex\"},{line,58}]},{'Elixir.Enum','-map/2-fun-0-',3,[{file,\"lib/enum.ex\"},{line,1043}]},{'Elixir.Enum','-reduce/3-lists^foldl/2-0-',3,[{file,\"lib/enum.ex\"},{line,1385}]},{'Elixir.Enum',map,2,[{file,\"lib/enum.ex\"},{line,1043}]},{'Elixir.Slownews.Crawler',init,1,[{file,\"lib/slownews.crawler.ex\"},{line,28}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,306}]}]}}},{'Elixir.Slownews',start,[normal,[]]}}}"}  
Oct 10 16:35:26 slownews-trot app/web.1:  Crash dump was written to: erl_crash.dump  
Oct 10 16:35:26 slownews-trot app/web.1:  Kernel pid terminated (application_controller) ({application_start_failure,slownews,{{shutdown,{failed_to_start_child,'Elixir.Slownews.Crawler',{#{'__exception__' => true,'__struct__' => 'Elixir.HTT