suhailrehman / relic Goto Github PK

detectors are enumerating and evaluating the schema of every pair of artifacts right now. If the Join detector is invoked after PPTs, we are assured that every component has the same schema, or we can even use the clustering info to check for schema compatibility before enumerating all join triples.

RELIC badly needs a code redesign. Ideally should be completed after #4, and is required before we write APIs for the demo.

Some Ideas / Wishlist:

1. Object-Oriented Design based on Ousterhout's Principles
2. Serialize Algorithm's State and Resume
3. Separate Profiling and Score Computation steps from Graph-Building
4. Just the appropriate data structure to store ranked order of edges and coalesce when components are joined
5. Robust tie-breaking mechanism descriptor and implementation
6. Load dataframes on-demand to improve memory
7. Robust logging with trace and debug logging to file
8. Design should allow for quick API hooks into future RELIC webserver
9. Allow easy implementation of sampling/estimation hooks for the future.
10. Allow for threaded operation for compute-intesive tasks such as distance calculation. Examine JIT and threaded operation to verify actual threading via HTOP

RELIC currently expects a ground truth file before the operation, need to remove this in order to evaluate workflows sans ground truth.

Should write unit tests.

Run a code trace to see code callgraph and remove old/redundant code from the code base