loklak / loklak_jlib_api Goto Github PK

In TwitterScrapper class (search method):

1. Images are scraped using class="media media-thumbnail twitter-timeline-link media-forward is-preview", as the relevant tags for images are changed to, class="AdaptiveMedia-photoContainer" with key data-image-url, the scraper can't extract images from html text.
2. The same goes for videos and GIFs, the provided tags and keys in source code doesn't work. Required video details are in <video class="visible with key src (the link of video is prefixed with _glob: , trying to open that link in a new tab in browser doesn't redirect to the video, it says twitter can't find it). GIFs have also the same <video class="visible and key src (link is not prefixed with glob: , and we can open the link in a new tab, it shows the GIF).
3. The thumbnail of videos and GIFs can be obtained using div class="poster-image and key is style (the value of key needs to be processed to obtain the url, example of value is ... background-image: url("https://pbs.twimg.com/tweet_video_thumb/C7VwFXsU4AAmpE9.jpg"); display: none; ).

Now the problem with videos and GIFs is that they are being loaded by Javascript, so we don't get the required details in GET request made by HttpsUrlConnection's object (we only get the things provided by html, in this case, details of images).

@Orbiter What do you suggest ?

Create an Interface where loklak API endpoint methods are defined, based on which the request URL will be generated and the request (GET/POST) can be called using static methods in scr/org/loklak/tools/NetworkIO.java. The defined methods are implemented using java.lang.reflect.Proxy. Sample implementation is mentioned in GSoC proposal.

This way adding a new API endpoint and managing existing API in case of a change is lot easier, we don't need to write the repeating code for creating the request URL, just defining the method in the Interface does the work.

(Check if done) Documentation, Code Readability and Unit-tests of the following packages in org.loklak:

• client
• harvester
• objects
• tools

Add TravisCI to the project to check build failures.

Add pom.xml file for Maven, adding maven should not disturb the ant build as the library is currently not present in Maven repositories.

This is in reference to this issue. fossasia/loklak_wok_android#16 . This is the current user agent we are using.
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2

And according to twitter.

If you log in to twitter.com on the following browsers, you will be directed to see mobile.twitter.com.
Internet Explorer, version 10 or earlier
Safari, version 6 or earlier
Chrome, version 9 or earlier
Firefox, version 9 or earlier

The code in TwitterScrapper.java file is written according to twitter.com and not mobile.twitter.com. Thus the app is crashing.

@sudheesh001 Does this seem valid to you?

Publish loklak_jlib_api in MavenCentral repository.

TODOs for publishing the library:

• Making loklak_jlib_api similar to other Loklak API libraries, by removing the scraping(tweet-harvesting) code. The scraping part in the library is implemented by classes in harvester and objects packages.
• The client package have API endpoints implementation using Java reflect and proxy (implemented by LoklakAPI.java and APIGenerator.java) and the old API implementations i.e. PushClient, SearchClient and SuggestClient can be removed.
• Removing ant build as Maven has already been added. Including a new library (which has dependencies) to provide ant support will increase a lot of complexity (need to manually download jar files and provide it in lib directory).
• Add a markdown file, that gives examples to use Loklak API endpoints.

loklakj_lib uses the json.org JSON library. Android also uses json.org: https://developer.android.com/reference/org/json/package-summary.html

Unfortunately both versions are not compatible. The version used in loklakj_lib knows JSONObject.put(String, Collection) which is used in org.loklak.objects.MessageEntry for example. The version contained in Android does not know this method.

This seems to be the reason for the Android-Wok not being able to push data at the moment:

failed synchronous push to backend, attempt 4 java.lang.NoSuchMethodError: org.json.JSONObject.put at org.loklak.objects.MessageEntry.toJSON(Unknown Source) at org.loklak.objects.Timeline.toJSON(Unknown Source) at org.loklak.client.PushClient.push(Unknown Source) at org.loklak.android.wok.Harvester$PushThread.doInBackground(Harvester.java:263) at org.loklak.android.wok.Harvester$PushThread.doInBackground(Harvester.java:244) at android.os.AsyncTask$2.call(AsyncTask.java:288) at java.util.concurrent.FutureTask.run(FutureTask.java:237) at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:231) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587) at java.lang.Thread.run(Thread.java:841)

Unfortunately I think the only solution is to replace json.org.