zaza / olx-rss Goto Github PK

View Code? Open in Web Editor NEW

7.0 2.0 2.0 63 KB

Fluent Java API and RSS feed generator for OLX search results

License: MIT License

Java 73.86% HTML 26.14%

rss rss-generator rss-feed olx api

olx-rss's People

Contributors

Stargazers

Watchers

Forkers

swiftgeek oleiva

olx-rss's Issues

Verify provided location before searching for ads

See https://www.olx.pl/sitemap/regions/

Support searching in a price range

It's an easy way to remove "1PLN" spam at the very least.

unread feeds number missing in feedly

OlxOfferTest.offerWithNoPhoto() can time out

Timed out twice already:

' offerWithNoPhoto(com.github.zaza.olx.OlxOfferTest) Time elapsed: 14.197 sec <<< ERROR!

java.net.SocketTimeoutException: Read timed out

at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)

at java.net.SocketInputStream.read(SocketInputStream.java:171)

at java.net.SocketInputStream.read(SocketInputStream.java:141)

at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)

at sun.security.ssl.InputRecord.read(InputRecord.java:503)

at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)

at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)

at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)

at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)

at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)

at java.io.BufferedInputStream.read(BufferedInputStream.java:345)

at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)

at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)

at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)

at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)

at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)

at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)

at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:655)

at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:628)

at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:260)

at org.jsoup.helper.HttpConnection.get(HttpConnection.java:249)

at org.jsoup.Jsoup.parse(Jsoup.java:183)

at com.github.zaza.olx.OlxScrapper.getDocument(OlxScrapper.java:88)

at com.github.zaza.olx.OlxScrapper.getNextPageElements(OlxScrapper.java:81)

at com.github.zaza.olx.OlxScrapper.getNextPageUrl(OlxScrapper.java:76)

at com.github.zaza.olx.OlxScrapper.getOffers(OlxScrapper.java:58)

at com.github.zaza.olx.OlxOfferTest.offerWithNoPhoto(OlxOfferTest.java:100)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)

at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)

at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)

at org.junit.rules.RunRules.evaluate(RunRules.java:20)

at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)

at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)

at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)

at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)

at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)

at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)

at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)

at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)

at org.junit.runners.ParentRunner.run(ParentRunner.java:363)

at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)

at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)

at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)

at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)

at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)

at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)

at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)'

Price is a number, is-negotiable is a boolean

Consider enabling search with negotiable prices only

Multiple tests failing, the library is broken

support searching in category

https://www.olx.pl/sitemap/
https://github.com/jonmiles/bootstrap-treeview

upgrade to the latest heroku stack

From the latest heroku build/deployment:

! Warning: You are running on a deprecated stack.
! Please upgrade to the latest stack by following the instructions on:
! https://devcenter.heroku.com/articles/upgrading-to-the-latest-stack

olx-rss stopped working on 28-07-2018

No queries are working since 28-07-2018

build #18 failed: expected an offer with a photo

add a search option to return only offers with photos [1] and use that in the test

[1] https://www.olx.pl/oferty/q-sprzedam-opla/?search%5Bphotos%5D=1

Offline test samples are outdated

Replace sysouts with proper logging

all feeds display in feedly, with "Unread only"

even if the "Unread only" option is selected all feeds are displayed in feedly. some are greyed out, as they have been read, still shouldn't be shown at all

Retry temporary network errors e.g. 502

https://travis-ci.org/github/zaza/olx-rss/builds/732815332

use offer creation date as pub date of the feed

Upgrading jsoup breaks tests

upgrading jsoup to 1.10.3 causes some tests to fail with:

org.jsoup.select.Selector$SelectorParseException: Did not find balanced maker at class=link pageNextPrev {page:2} at org.jsoup.select.QueryParser.parse(QueryParser.java:44) at org.jsoup.select.Selector.<init>(Selector.java:86) at org.jsoup.select.Selector.select(Selector.java:108) at org.jsoup.nodes.Element.select(Element.java:322) at com.github.zaza.olx.OlxScrapper.getNextPageElements(OlxScrapper.java:91) at com.github.zaza.olx.OlxScrapper.hasNextPage(OlxScrapper.java:81) at com.github.zaza.olx.OlxScrapper.getOffers(OlxScrapper.java:58) at com.github.zaza.olx.OlxScrapperTest.assertHasOffers(OlxScrapperTest.java:46) at com.github.zaza.olx.OlxScrapperTest.hasOffers(OlxScrapperTest.java:16) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:89) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:41) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:541) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:763) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:463) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:209)