nasa-jpl-memex / geoparser Goto Github PK

Each file in menu will show results underneath, if the result is long it goes off screen.
Each file/result should be collapsable as well as be able to scroll up and down

As we are indexing millions of records I can see lot of issues with solr.

We initially did lot of handholding editing data types and using encoding but it's now constantly failing with OutOfMemoryError. I have tried increasing mem upto 1.5 gigs but it only gives us some extra time.

At this moment I am trying to create a new schema which addresses below limitations

• Instead of one single huge document we scale points across multiple documents
• We must be able to produce search results for a location. Query will be a LOCATION

I am planning below -

• one core in geoparser solr instance for every index
• one document in geoparser solr instance for each document in index
• one domain can cover multiple cores. core syntax will be -> domain_[id]

I am right now doing it only for indexed data on a new branch. As uploaded files thing works with no issues.

@MBoustani

An input box on top left for file upload.

@smadha Please update the menu as below:

Usually crawled data are being indexed to either Solr or Elasticsearch.
GeoParser should be able to get the URL to either of these to indexing machines and domain name, scan whole indexes and geoparse them.
The result (location name and point) will be stored in Solr internally along side with path to crawled data.

https://github.com/MBoustani/GeoParser/blob/master/geoparser_app/views.py#L261

Setup cherrypy to server static files and template HTML file.

We are decided to use CherryPy for server instead of Flask.
CherryPy is more stable and also as we are going to join with other Memex geo projects we should be using same technologies they use.

As of now index.html is called once and stored as template object. To reflect any change in index.html we had to restart server.

Server should start the Geoparsing process as soon as file/s being uploaded to server and send the status back to front-end.

After each file uploaded, name of file will appear under "upload file" section.
@smadha Can you please put a remove icon by each file or check with @lawongsta about how to remove a file and maybe send a request to server to remove a file?
@smadha should we use REST URL to send the remove command to server for each file?

Progress bar should show the status of file being uploaded as well as file being geoparsed with status text underneath it.

Basic map accessible via index page.

need to fix the issue with not being able to upload files using Django

-Add a black bar with heading on top of page
-Use Bootstrap icons and create a navigation switch
-Remove temp images used

Adding exposed = True as per https://github.com/Kitware/minerva/blob/master/server/loader.py

Uploaded files will appear as soon as they get uploaded, but if page get refreshed they will not be shown any more.

Solution:
Have server look up in uploaded folder and send list of files to front-end.

Using Solr for storing and retrieving data from.
One collection called "Uploaded_Files" will store all geoparsed uploaded files.
Schema could be something close to:
file_name = <file_name>
extracted_text =
location_names = [list of locations name]
lat/lon = [list of location dictionary(key: location name, value: lat/long)]

**This schema may change as we learn more about Solr Spatial query

Delete button in front of each uploaded file should remove file from server and Solr.

When the GeoParser app loads, have Girder to login to be able to use Girder.
Username: girder
Password: girder

Base 64 encode: Z2lyZGVyOmdpcmRlcg==

I am using this page:
http://tools.cherrypy.org/wiki/DirectToDiskFileUpload
to setup upload file capability.

Need to show more data on popups to analyze individual bubble.

• "title" : "Title of doc fetched as configured",
• "descr" : "Short description fetched as per configuration"
• "url" : "Link to html/image"
• "solr_url" : "Link to solr_doc"

We have already made collection and a folder called "uploaded_files" within collection.
As user upload a file, client can create and item with in "uploaded_files" using Girder REST API.

@smadha please review Grider docs REST API to see how you can create item with in folder.

As we are replacing current CherryPy serve to Girder, we need to define a REST URL to POST uploaded file to Girder Filesystem

@lawongsta Any ideas?

Fetching all the coordinates from solr and transforming them to csv is taking lot of time.
This can be done asynchronously to save time of main thread.

Some text return from Tika has some characters that cannot be indexed to Solr.

We have decided to replace the current CherryPy server with Girder (it is using CherryPy as server by itself)

Geoparser plugin for girder can have multiple jobs running using Girder.
Each job can be called using REST URL and will return the results as JSON.

We need to hide the last side menu when the points are in map to be able to view map in full screen.
Now the menu covers 1/4 of screen.

Below APIs need to be mocked

[API signature], [Method],
[Sample Response]

1. /upload POST
   Response code - 200
2. /status/%file_id% GET
   {
   "name":"File Name",
   "status":"Message to be displayed to user",
   "stepCount":4,
   "parsedInfo":[
   {
   "lat":-34.6037232,
   "lon":-58.3815931,
   "name":"Aires Argentina",
   "refCount":10,
   "refContext":"Test line 4 in file uploaded",
   "refUrl":"https://geo1.ggpht.com/cbk?panoid=wkEz-Hwmc44EnMsE7SuXBw&output=thumbnail"
   },
   {
   "lat":19.4302678,
   "lon":-99.1373136,
   "name":"Mexico City, Mexico",
   "refCount":2,
   "refContext":"Test line 42 in file uploaded",
   "refUrl":"https://geo0.ggpht.com/cbk?output=thumbnail&thumb=2&panoid=3DKyddof6dWPw3tx5BULbQ&w=96&h=64&yaw=176"
   }
   ]
   }
3. /search/index/%keyword% GET
   {
   "name":"File Name",
   "status":"Message to be displayed to user",
   "stepCount":4,
   "parsedInfo":[
   {
   "lat":-34.6037232,
   "lon":-58.3815931,
   "name":"Aires Argentina",
   "refCount":10,
   "refContext":"Test line 4 in file uploaded",
   "refUrl":"https://geo1.ggpht.com/cbk?panoid=wkEz-Hwmc44EnMsE7SuXBw&output=thumbnail"
   },
   {
   "lat":19.4302678,
   "lon":-99.1373136,
   "name":"Mexico City, Mexico",
   "refCount":2,
   "refContext":"Test line 42 in file uploaded",
   "refUrl":"https://geo0.ggpht.com/cbk?output=thumbnail&thumb=2&panoid=3DKyddof6dWPw3tx5BULbQ&w=96&h=64&yaw=176"
   }
   ]
   }

https://drive.google.com/open?id=1ASR0j0lzT8GqifZ0ep6WMBV9SaAOENPHUIqUrrR7dbo

@smadha I have update the "return_points" url to work with both uploaded files and crawled data.
Now you can call return_points to get point from both.
However, you need to update the code you have now that called return_points, here is the update:

For uploaded files: http://localhost:8000/return_points/<file_name>/uploaded_files
Example: http://localhost:8000/return_points/5c0024-25.pdf/uploaded_files

For crawled data: http://localhost:8000/return_points/<solr_url>/<core_name>
Example: http://localhost:8000/return_points/http://crawl.dyndns.org/solr/domain

Please update both.

Here we are going to compare GeoTopicParser with Geograpy combined with GeoPy ruing over same text document.
@smadha can you please take over GeoTopicParser 1 and print the result here.
I am going to use Geograpy + GeoPy over same text and print the result here 2

Files:

TAP1500-Active-Probe-Datasheet-6.pdf
5c0024-25.pdf

simplify text like we do for indexed data before indexing file text

After user types domain name and indexed URL put the result under save button that user can come back and on/off point on map.

1. Commit l-g-g jar from maven central. https://oss.sonatype.org/content/groups/public/edu/usc/ir/lucene-geo-gazetteer/0.2/lucene-geo-gazetteer-0.2-jar-with-dependencies.jar
2. Commit tika-server jar from mirror. http://www-us.apache.org/dist/tika/tika-server-1.12.jar

Start solr,l-g-g,tika geotopic server through manage.py

@MBoustani @chrismattmann any thoughts on this?

Pleas change the map engine from GeoJS to OpenLayer.

For example 5989-9131EN.pdf returns many location twice thrice

Quick
Quick
Quick
Data
Data
U.S.

GeoParser at this stage is using two servers as backend.
1- CherryPy as web server
2- Grider for file system and running jobs

Beside running two servers at same time could increase the chance of application failure there is problem if calling Girder on different port by CherryPy in different port (cross-domain issue), therefore these two can be merged to one and have Girder take care of everything.

Make one single file for JS and CSS plugin.
Minimize both app and plugin JS and CSS.

In CherryPy there is a configuration for statics file such as CSS, JS and imgs.

After the Front-end send files to Server, Girder should create Collection and Folder if they are not already created.