emir-munoz / hive-mrc Goto Github PK

See how well HIVE scales

(noted by Kevin C)
HIVE doesn't build on a Unicode-based system because the character encoding of 
the source files is Windows-1252 (and the stemmer files choke in the build).  
To build on a unicode-based system, the build.xml file needs to have encoding 
attribute added to its javac:

<javac srcdir="${src.dir}" destdir="${build.dir}" encoding="Windows-1252"
  source="1.5" target="1.5" debug="off" includeAntRuntime="false">

RDF import assumes RDFXML. Include support for other RDF encodings.

Ensure HIVE starts when karya is rebooted

Some documentation in the trunk/doc directory duplicates content in the wiki. 
Move everything out of the doc directory and onto the wiki (except the javadocs)

When the indexer presents results, it should be possible to select each term -- 
clicking on a term highlights that term, and clicking again turns off the 
highlight.

There should be a button that copies all highlighted terms to the clipboard 
(with newlines separating them).

Every vocabulary in HIVE have to be synchronized with the original source when 
a RSS source or similar is available

In concept browser, allow users to compare terms from two different 
vocabularies. Design a new page layout that allows 2 or more) search results to 
be selected and compared.

Look through the code that accesses low-level indexes. Ensure that items are 
only kept in memory when they are needed to enhance performance, and objects 
are not re-created unnecessarily.

Classes include: 
SKOSSearcherImpl
IndexAdministrator
SKOSSchemeImpl

Based on sample code (and comments) in SkosServerImpl

Clean up the package structure of HIVE.

NOTE: since this is a major reorganization of the code, create an svn tag 
first, and send a notice to the hive-dev list about what you're doing, in case 
anyone was depending on this structure.

Hive-rs and hive-core have slightly different package structures. They should 
be parallel, and simple. (edu.unc.hive is much better than edu.unc.ils.mrc.hive)

The class edu.unc.ils.mrc.hive.api.impl.elmo.SKOSSearcherImpl is in a package 
that implies Sesame-only functionality, but in reality it uses both Sesame and 
Lucene functionality.

A search for "wood" does not return the NBII term "wood", because this term
is not ranked highly enough. The entire list of matching terms should be
returned.

1. Perform indexing on the Indexing page
2. Click on a result term.
3. The detailed display for the term can overlap the page footer, especially if 
the term has many related terms.

The most recent version of AGROVOC includes multiple languages. Add support in 
HIVE to import a specific locale.

The blue header bar at the top of the HIVE web UI is useless. Remove it.

Write Javadocs for all high-level APIs

Need to be able to translate HTML and GWT strings.

Terms returned from the indexer often break across lines. Each term should 
appear as a whole, without intervening line breaks.

Add support for conversion of EMBNE (MARC Authority) to SKOS

Move HIVE application out of Jose's directory on Karya.

Add Apache-CLI (modeled after Admin vocabularies)
-f input file of URLs to be crawled
-o output directory where text will be dumped
-n number of hops (default 0 - first page only)
-m number of terms (default 10)
-d enable differencing

What steps will reproduce the problem?
1.plan schedule
2.
3.

What is the expected output? What do you see instead?


Please use labels and text to provide additional information.

Do we really need Lucene? We're only using very simple functionality for term 
searching.... maybe we could do all of this in Sesame?

add Hudson to the HIVE setup so we can more easily update the deployed version

Add conversion from NCBI export format to SKOS (viruses only)

The concept browser consistently rates exact matches lower than partial 
matches. Review the algorithm that performs matching, and adjust it

add Hudson to the HIVE setup so we can more easily update the deployed version

Need to be able to specify the minimum phrase occurrence during model creation 
(KEA and MAUI) and indexing.  

the word "add" beside the vocabulary listing should be clickable to add a new 
vocabulary

In the concept browser, immediately after performing a search, the "View in 
SKOS" button does not return any results. After the user clicks on one of the 
search results, the "View in SKOS button performs correctly.

HIVE import is not using UTF-8 readers for RDF or for H2 initialization for KEA>

Clean up documentation of installation/deployment process

Find a good place to add these links and add them.

Make MeshConverter actually use command line arguments. 

The architecture diagram in the docs directory is outdated, and should be 
replaced with the architecture diagram that is part of the developer workshops.

The HIVE project currently depends on GWT 1.7.1. GWT version 2.1.1 is 
downloaded and installed by default with the GWT plugin for Eclipse. While it 
is possible to download and install the 1.7.1 SDK for later plugin versions, it 
may be worth updating. The update process will require a set of test cases for 
the current GWT interface.

Allow users to select whether to enabled/disable indexing of only differences 
between pages in URL-based indexing process.

BioPortal has an autocomplete widget that people can add into their web forms, 
backed by a particular vocabulary. It would be great to have a similar widget 
in HIVE.

There is a method ConceptsResource.tagDocument() that supposedly performs 
tagging for hive-rs. Determine whether it actually works (fix it if it 
doesn't), and add appropriate documentation to the hive-rs pages.

On initial glance, I'm not certain whether it is doing anything, because it 
relies on a file path coming in to the method, but this path isn't connected to 
a URL parameter. 

It probably isn't possible to support a file-based submission in the http GET 
call, but we could allow a URL to be passed as a parameter in the GET call.

* Add indexing algorithm selector to hive-web indexer page.
* Add Maui tagger implementation to hive-core.
* Add Maui model initialization to AdminVocabularies class.

Add callback parameter support to autocomplete and autosuggest servlets

In the concept browser, users can see both the browsing list for the full 
vocabulary and the list of search results. This can be confusing. The easiest 
solution is to move the search box directly above the search results box. But 
the entire page needs to be redesigned.

Add the JAX-RS/Jersey RESTful Web Services HIVE-CORE interface that was 
developed at LTER. As per the videoconference call on 8/23/2010, the new 
software will be added in a sub-tree named 'hive-rs' which will be a sibling 
directory to the 'hive-core' and 'hive-web' directories.

(Additional notes from the videoconference call are included below.)

-----

Dryad/HIVE/LTER Meeting Notes, 8/23/2010:

A summary of my meeting notes from today's Dryad/HIVE/LTER meeting:

* In the remaining weeks, equal priority should be placed on:
  1. Committing the JAX-RS implementation of hive-core restful web services (source code, documentation, and unit tests) into the 'hive-mrc' repository.
  2. Completing at least one of the two prototypes. (EML keyword enrichment prototype; LTER Data Portal enhanced search prototype)

* The JAX-RS implementation of hive-core restful web services will be committed 
into a separate sub-tree named 'hive-rs', which will be a new sibling directory 
to the current 'hive-core' and 'hive-web' directories under the 'hive-mrc' 
repository. This will become the primary resource for LTER, which will 
eventually discontinue maintaining a local copy in the LTER repository.

* Ryan has added Duane's Google account ([email protected]) to authorize 
Duane to do commits to the 'hive-mrc' repository in Google Code. A fresh 
checkout will probably be needed. Unsure whether this works with the 
TortoiseSVN client on Windows.

* Todd Vision has asked for a high-level document describing the work that has 
been accomplished. Duane will check with Mark Servilla about composing the 
document.

* James Brunt has approved the release of the LTER Vocabulary for use in Dryad 
and HIVE. Mike Frame has approved the use of the NBII thesaurus by LTER in its 
production systems. (But not necessarily its redistribution.)

* Duane and Jose will arrange a developer meeting on Skype soon to conduct a 
code review of the JAX-RS web services code that has been developed.

Base this on the SkosServerImpl in
hive-core/src/edu/unc/ils/mrc/hive/api/impl/elmo

The config file is given when running SkosServerImpl.main()

All files in the trunk/doc directory should be moved into the wiki if possible. 

Add new properties to hive.properties to support configuration of a proxy 
server that will be used when indexing websites.

The sample searches on the REST page aren't working -- have they changed?
    * http://hive.nescent.org/hive-rs/schemes/nbii/concepts/SKOSFormat?keyword=activity
    * http://hive.nescent.org/hive-rs/schemes/nbii/concepts/concept/SKOSFormat?prefLabel=Eruptions 

We want to analyze how people are using HIVE. More logging is needed to provide 
us with information about what parts of the interface are being used.

The indexer frequently lets multi-word-terms wrap across lines, which is very 
confusing. Ensure that each term appears as a whole object, with no linebreaks.

In the indexer, when the user enters a URL, there should be a checkbox titled 
"multi-hop indexing", with a dropdown beside it that contains the numbers 1 
through 3.

If multi-hop indexing is selected, the page specified by the URL is sent to the 
indexer, AND ALSO the pages that are linked from the original page. The numbers 
1-3 indicate the depth of links to follow. Links are only followed for pages 
within the same domain.