squarlhan / snofyre Goto Github PK

I'm attempting to use Snofyre as a proof-of-principle demonstrator of an 
architecture to cope with concept inactivation, where inactive conceptIDs may 
be found in EITHER the EPR data being queried OR in the query specifications 
being run over that data. Demonstrating the solution requires swapping 
different versions of the SNOMED core tables, and particularly the TC table, 
around under Snofyre....

My current ‘test’ case runs as follows:

Query1 = Patients who have 38103000 Atopic rhinitis
Query2 = Patients who have 70076002 Rhinitis

..and these queries are to be executed (or used to generate data) under one of 
three different SNOMED configurations:

Config ‘OLD’:
Concept, relationship and description tables = October 2010 data
Transitive Closure Table = October 2010 data

CONFIG ‘NEW’
Concept, relationship and description tables = April 2011 data
Transitive Closure Table = April 2011 data

CONFIG ‘AUG’
Concept, relationship and description tables = April 2011 data
Transitive Closure Table = April 2011 data augmented with historical 
substitutions

The sequence of events that demonstrates the problem is as follows:

1. Delete all records and purge the TC table and expression library

2. Using Config=OLD (in which concepts table says Atopic Rhinitis is current 
and TC table says Atopic rhinitis subtypeOf Rhinitis)
Generate 100 patients using Query1
Run Query1: returns 100 patients
Run Query2: returns 100 patients

3. Purge the TC table but not the expression library
Swap to Config:NEW (in which concepts table says Atopic Rhinitis is Duplicate 
and so tc table says Atopic rhinitis NOT subtypeOf Rhinitis)
Run Query2: returns zero patients (as expected)

4. Swap back to Config:OLD
Run Query2: returns 100 patients (interesting; Snofyre appears to have sucked 
in the TC table without the expression TC table being purged first…)

5. Swap back to Config:NEW
Run Query2: returns 100 patients (not same result as before; assume expression 
TC table has to be purged to flush old results?)
Purge the TC table but not the expression library
Re-Run Query2: returns zero patients (as expected)
Re-Run Query2: returns zero patients (as expected)

6. Swap back to Config:OLD
Run Query2: returns 100 patients (so expression TC table is extended when new 
TC results are encountered, but never trimmed if existing rows become invalid 
against external main TC table?)

So, at this point everything is going well because I can swap back and forward 
between OLD and NEW and, provided I purge the TC table at each swap, I always 
get the answer I expect for the particular TC table sitting under SnoFyre: 
either zero, or 100 patients.

Where things go strange is if I then do:

7. Purge the TC table but not the expression library
Swap to Config:AUG  (in which Atopic rhinitis is an inactive duplicate, but 
despite this the TC table says it IS a subtype of Rhinitis)
Re-Run Query2: returns zero patients (NOT as expected)

It looks to me as though the sct_concept table is still being used first to 
test each instance of 38103000 that’s in the EPR and, finding that its an 
inactive concept (which it is, according to the sct_concept table in situ in 
either NEW or AUG) it then tries to substitute it. But, if it did do that, it  
*should* end up with 38103000 being turned into 61582004, which is still a 
subtype of the Rhinitis query concept, so I don’t understand why I still get 
zero patients!

Whatever, it looks as though some part of some Snofyre pipelines includes a 
step to logically replace any inactive ConceptIDs encountered with an active 
one. 

Could this be optionally switched off? 

Proposal for a way to import data into SNOFyre rather 
than generate the date within it.

This would presumably require not only a data import method but one which 
conducted at least the minimum data validation.

What is the proposed new feature or improvement?
Snomed Expression model classes do not have a factory implementation. They are 
being instantiated directly. All instantiations should be through a factory 
implementation!

Version Information:
 -- Snofyre version      : 1.8

SNOMED database name and column names hard-coded in DAO class

What steps will reproduce the problem?

1. Define query 'All pts who have 123397009|Injury of anatomical site|'
2. Try to change querydef:
   a. Double-click on existing criterion
   b. Click Edit button on Author Component Query dialog
   c. Enlarge Author expression dialog to reveal Expression Editor pane
   d. Expand the role group
   e. Double-click on the Associaed morphology = Damage line
   f. In Concept Hierarchy dialog, expand the 'Damage' line
   g. Double-click on 'Dislocation'

What is the expected output? What do you see instead?

Querydef changed to 'All pts who have Injury of anatomical site, with 
morphology of Dislocation'

Please use labels and text to provide additional information.

'All pts who have Injury of anatomical site, with morphology of Inferior 
dislocation'

Widget behaviour appears to be that, if you double-click on an unexpanded node 
in the selection tree, then item actually selected is first list-
order descendent of node clicked on, if the tree were to be expanded. If tree 
already IS expanded, then item selected is same as that double-clicked on.

Version Information:
 -- Snofyre version       : 1.8 build guy-fawkes-4d5fd02decc8+
 -- Operating System      : Window 7 64-bit
 -- MySQL Server version  : 5.5

Would like to be able to specify a postcoorinated concept as a querydef where 
the axes of postcoordination go beyond the existing defining and qualifying 
concepts in order to include relationships permitted by the MRCM.

e.g. to be able to ask for (and generate from)

patients who have any 271737000|Anaemia|:
   47429007|Associated with|=118948005|Disorder of abdomen)|

What steps will reproduce the problem?
1. Generating data for query with above concept
2. Snofyre runs out of memory even when patient number set to 100

What is the proposed new feature or improvement?
Add ability to launch scripting console from menu action

Version Information:
 -- Snofyre version      : 1.8

Using 20110131 version of installer app over a 10Mb broadband connection...

TRUD download of the 225M data pack takes around 30 minutes. But the installer 
app has no progress bar and doesn't warn the user that it will sit there doing 
apparently nothing for a very long time. The savvy user might check the 
download progress by monitoring the current size of the downloading file in the 
SnoFyre install directory, but it would help to tell everybody to do this and 
also that the file being downloaded is about 225M, so they can work out for 
themselves that this is a long job.

Probably also a mistake to suppress any of the FTP session responses; they're 
evidence that at least the login has been successfully made. If the FTP 
responses were displayed, then the script could include an ls -a command to 
display the file size.

(Also...about the 'prom off' directive in the FTP script: the 'off' part 
appears to be redundant, at least using the MS FTP command line client. 
'prompt' or 'prom' on its own simply toggles the interactive status between on 
and off, and any trailing on/off argument isn't parsed. Two successive 'prom 
off' commands actually leaves interactive mode back on.)

What steps will reproduce the problem?
1. Data generation using refinements on expressions.
2. Throws data too long for column error.

C:\Program Files\Snofyre\bin\queries\allergicDisorder.xml
[DEBUG] 07 Jan 2011 14:55:04 
(uk.nhs.cfh.dsp.srth.desktop.modules.querycreationtreepanel.QueryAuthoringTreePa
nel:459) - Notified model of query change
[ INFO] 07 Jan 2011 14:55:35 
(uk.nhs.cfh.dsp.srth.simulator.engine.impl.QueryBasedDataGenerationEngineImpl:12
6) - Starting data generation from data map
[ WARN] 07 Jan 2011 14:55:35 
(uk.nhs.cfh.dsp.srth.expression.repository.impl.ExpressionMappingObjectDAOImpl:2
87) - Found more than one match. Returning the first match found
[ INFO] 07 Jan 2011 14:55:35 
(uk.nhs.cfh.dsp.srth.simulator.engine.impl.QueryBasedDataGenerationEngineImpl:24
4) - Saved patient with ID : 475
[ WARN] 07 Jan 2011 14:55:36 
(uk.nhs.cfh.dsp.srth.expression.repository.impl.ExpressionMappingObjectDAOImpl:2
87) - Found more than one match. Returning the first match found
[ INFO] 07 Jan 2011 14:55:36 
(uk.nhs.cfh.dsp.srth.simulator.engine.impl.QueryBasedDataGenerationEngineImpl:24
4) - Saved patient with ID : 638
[ WARN] 07 Jan 2011 14:55:37 
(uk.nhs.cfh.dsp.srth.expression.repository.impl.ExpressionMappingObjectDAOImpl:2
87) - Found more than one match. Returning the first match found
[ INFO] 07 Jan 2011 14:55:37 
(uk.nhs.cfh.dsp.srth.simulator.engine.impl.QueryBasedDataGenerationEngineImpl:24
4) - Saved patient with ID : 277
[ WARN] 07 Jan 2011 14:55:38 (org.hibernate.util.JDBCExceptionReporter:357) - 
SQL Error: 0, SQLState: 22001
[ERROR] 07 Jan 2011 14:55:38 (org.hibernate.util.JDBCExceptionReporter:454) - 
Data truncation: Data too long for column 'nfe_cgf' at row 1
[ WARN] 07 Jan 2011 14:55:52 
(uk.nhs.cfh.dsp.srth.desktop.modules.simulator.viewcomponent.actions.DataGenerat
orTask:105) - Error creating data. Nested message is could not insert: 
[uk.nhs.cfh.dsp.srth.expression.repository.om.impl.ExpressionMappingObjectImpl];
 nested exception is org.hibernate.exception.DataException: could not insert: 
[uk.nhs.cfh.dsp.srth.expression.repository.om.impl.ExpressionMappingObjectImpl]N
ested exception is : could not insert: 
[uk.nhs.cfh.dsp.srth.expression.repository.om.impl.ExpressionMappingObjectImpl];
 nested exception is org.hibernate.exception.DataException: could not insert: 
[uk.nhs.cfh.dsp.srth.expression.repository.om.impl.ExpressionMappingObjectImpl]

What steps will reproduce the problem?
1. Download Snofyre 1.9 - SNAPSHOT from trunk and execute mvn clean install as 
per documentation
2. mvn clean install fails for srth-osgi with error message below

What is the expected output? What do you see instead?

See report at 
http://groups.google.com/group/snofyre-dev/browse_thread/thread/125e06d68d7c23

What steps will reproduce the problem?
1. Data generation throws non-unique identifier randomly
2. Looks like a hibernate generated error

What steps will reproduce the problem?
1. attempt to install snofre with the Snofyre-installer.jar
2. installer stalls on step 15 of 16 - where the database connection is 
verified. logging info shows:

Sep 8, 2011 10:22:58 PM
uk.nhs.cfh.dsp.srth.distribution.DBConnectionVerifier verifyConnection
WARNING: Error creating connection. Nested exception is : SELECT
command denied to user 'snofyre'@'localhost' for table 'CONCEPT'
Sep 8, 2011 10:22:58 PM
uk.nhs.cfh.dsp.srth.distribution.JDBCConnectionValidator validateData
INFO: installPath = /home/matt/Applications/Snofyre
Sep 8, 2011 10:22:58 PM
uk.nhs.cfh.dsp.srth.distribution.JDBCConnectionValidator validateData
INFO: varMap = {sappheiros.db.password=[mypassword],
sappheiros.db.schema.name=fakedata2,
sappheiros.db.host.name=localhost, sappheiros.db.port.number=3306,
sappheiros.db.user.name=snofyre} 

What is the expected output? What do you see instead?

Have checked connections in mysql workbench where everything seems fine.  My 
guess is a case sensitivity issue in the installer that may not have been 
picked up on windows.

Version Information:
 -- Snofyre version      : 1.8
 -- Operating System        : Ubuntu 10.04
 -- MySQL Server version  : 5.1


Please provide any additional information below. Also attach the log file
where appropriate. It is located in INSTALLATION_FOLDER/Snofyre/bin/logs

Suggest changing default schema name from 'fakedata2' to 'snofyre'

Some schema table names carry prefixes to indicate broad purpose e.g. epr_, but 
this currently isn't carried through. It could help understanding of the 
backend if it was.

E.g. patient_table -> ehr_master_patient_index
     clinical_entry -> ehr_clincal_entry
     clinical_entity -> ehr_clinical_entity

A naming prefix for those tables that hold the expression repository would be 
useful.

E.g. exp_expression, exp_expression_tc_table, exp_close_to_user_expressions, 
exp_unique_expressions_table

A lot of tables don't appear to be used at all. If they really are legacy of 
earlier stages in the software development, it would help to have a spring 
clean. 

E.g. person_table, parents

What steps will reproduce the problem?
1. Create a querydef
2. Generate 3500 patients against the QueryDef
3. Generate another 3500 patients against the QueryDef
4. Count the number of unique patient IDs geneated so far

What is the expected output? 

7000

What do you see instead?

3500

What steps will reproduce the problem?
1. Generate 4000 patients with condition X
2. Query for patients with X

What is the expected output? 

A list of 4,000 patients

What do you see instead?

A java heap explosion

What steps will reproduce the problem?

1. Load the attached Query1
2. Generate 100 patients with option ADD_NEW_ALWAYS from it
3. Run Query2

What is the expected output? 

A list of either 100 patients, or of no patients.

What do you see instead?

An error

What steps will reproduce the problem?

1. Define query 'All pts who have 123397009|Injury of anatomical site|'
2. Try to change querydef:
   a. Double-click on existing criterion
   b. Click Edit button on Author Component Query dialog
   c. Enlarge Author expression dialog to reveal Expression Editor pane
   d. Expand the role group
   e. Double-click on the Associaed morphology = Damage line
   f. In Concept Hierarchy dialog, expand the 'Damage' line
   g. Double-click on 'Dislocation'

What is the expected output? What do you see instead?

Querydef changed to 'All pts who have Injury of anatomical site, with 
morphology of Dislocation'

Please use labels and text to provide additional information.

'All pts who have Injury of anatomical site, with morphology of Inferior 
dislocation'

Widget behaviour appears to be that, if you double-click on an unexpanded node 
in the selection tree, then item actually selected is first list-
order descendent of node clicked on, if the tree were to be expanded. If tree 
already IS expanded, then item selected is same as that double-clicked on.

Version Information:
 -- Snofyre version       : 1.8 build guy-fawkes-4d5fd02decc8+
 -- Operating System      : Window 7 64-bit
 -- MySQL Server version  : 5.5

Need a UI widget where you can paste in an original concise CTU or LNF CG 
expression that contains no human readable descriptions, and get back a 
pretty-printed (e.g. tabbed indented, attribute names in UCASE and/or different 
colour) version of the expression but with the FSNs inserted.

What is the proposed new feature or improvement?
Create Maven repo for project artefacts. This will allow us to host a Maven 
repo on google code. Developers wont have to check out and do a local install 
of Snofyre code when using Maven.

What is the proposed new feature or improvement?
1. Attribute ids are currently hard-coded. Need to be moved to a config file.
2. Helps portability and also user driven editing of human readable text

Version Information:
 -- Snofyre version      : 1.8 SNAPSHOT

What steps will reproduce the problem?
1. New online installer does not work on Mac OS X
2. Log error WARNING: Nested exception is : java.util.zip.ZipException: archive 
is not a ZIP archive
3. Installation can not be completed!

What is the expected output? What do you see instead?
Expect to see all Snofyre components to be downloaded from TRUD. Instead 
downloads are incomplete. 

Please use labels and text to provide additional information.

What is the proposed new feature or improvement?
Snofyre needs some kind of profiling support and a framework for monitoring 
performance

Version Information:
 -- Snofyre version      : 1.8
 -- Operating System        : Any
 -- MySQL Server version  : Any

Please provide any additional information below.
Adding profiling support for Snofyre services will allow performance to be 
monitored remotely (ideally via JMX)

What steps will reproduce the problem?
1.Search filter are ambiguous
2.Search filters wont work!

What steps will reproduce the problem?
1. Selected Burns morphological abnormality in search results
2. User interface becomes unresponsive
3. Error stack shows illegal attribute for rendering?

What steps will reproduce the problem?
1. Generate 16,000 patients with a subtype of anaemia, in 4 batches of 4,000 
each
2. Query for all patients with some anaemia

What is the expected output? 

A list of 16,000 patient IDs

What do you see instead?

(after a long pause) an error message:

uk.nhs.cfh.dsp.snomed.objectmodel.error.ConceptNotFoundException: No matching 
concept found for concept ID - {116676008

Version Information:
 -- Snofyre version       : patched 1.1.0
 -- Operating System      : Win7 64-bit
 -- MySQL Server version  : 5.5.8 community
 -- SNOMED Release        : October 2010 UK data

What steps will reproduce the problem?
1. Create new query: ALL patients who have 271737000 or ANY of its types
2. Try and generate patient data against the current query

What is the expected output? 
100 generated patients

What do you see instead?

A random number of between 1 and 6 patients are generated before an error 
message is raised:

could not insert: 
[uk.nhs.cfh.dsp.srth.expression.repository.om.impl.ExpressionMappingObjectImpl];
 nested exception is org.hibernate.exception.DataException: could not insert: 
[uk.nhs.cfh.dsp.srth.expression.repository.om.impl.ExpressionMappingObjectImpl]


Version Information:
 -- Snofyre version      : 1.8 guy-fawkes-4d5fd02decc8+
including Jan 7th/10th JAR patches from workshop
 -- Operating System     : XP Pro
 -- MySQL Server version  : 5.1.41 community

Please provide any additional information below.

Possibly due to a corrupton in the underlying tables owing to a previous failed 
query. I'm pretty sure this worked immediately after the original install. I'm 
wondering whether the option to delete all tables and rebuild the DB from 
scratch could be valuable to implement sooner rather than later, otherwise I 
suspect there's room for it all to get permanently confused.

Would be handy to have a widget on the Data menu to report basic metrics of the 
simulated EPR repository, such as:

Number of simulated patients
Number of clinical entries per patient
Number of unique clinical expressions encountered across all EPR instances

What steps will reproduce the problem?
1. Search for concept id 445427006 brings back no results
2. Concept is known to exist in Oct 2010 release of SNOMED CT

What steps will reproduce the problem?
1.Edit an existing query by adding a new query criterion or deleting existing 
one
2.Nodes in query tree rendering collapse

What is the expected output? What do you see instead?
Expect query tree to remain expanded and automatically scroll to selected 
expression or parent (in case of deletion)

Please use labels and text to provide additional information.
Snofyre version : 1.8 - SNAPSHOT

What is the proposed new feature or improvement?
It'll be useful to expose Snofyre's underlying services as ReSTful services to 
applications to plug into. 

Version Information:
 -- Snofyre version      :  1.8

Please provide any additional information below.
While creating my experimental browser for SNOMED CT between jobs, it became 
obvious that it would be cleaner to have a ReSTful API access to Snofyre's 
services. I guess some of this has to do with the fact that I chose to create 
my browser as a web app... 

What is the proposed new feature or improvement?
The current installer packages application binaries and SNOMED CT data into the 
same executable. This is inefficient and will mean that the installer can only 
ever be delivered through TRUD. An installer that can access TRUD and download 
SNOMED CT data, on the other hand can be hosted on Google Code

Version Information:
 -- Snofyre version      : 1.8-SNAPSHOT

What steps will reproduce the problem?
1. Generate 100 patients with MI (add new always)
2. Generate 100 patients with Injury anatomical structure (add new always)
3. Query for pts with MI
4. Query for pts with Injury anatomical structure
5. Query for Injury anatomical structure, with morphology of necrosis

What is the expected output? 

Response to (5) should be less than response to (4)

What do you see instead?

Response to (5) is more than response to (4) 

Patients with MI are not being retrieved by (4), but are by (5)

What steps will reproduce the problem?
1.Checked out code from svn repo
2.Try to build snomed-osgi component with mvn clean install
3.Maven build fails

What is the expected output? What do you see instead?
Expect successful build of snomed-osgi components. 

Instead Maven throws the following error:
[INFO] [buildnumber:hgchangeset {execution: default}]
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Command failed.
EXECUTION FAILED
  Execution of cmd : id failed with exit code: 255.
  Working directory was: 
    /Users/jay/Documents/snofyresvnworkspace/snomed-osgi
  Your Hg installation seems to be valid and complete.
    Hg version: 1.7.1+20101116 (OK)

Please use labels and text to provide additional information.

What steps will reproduce the problem?
1. Tried to change number of patients in Data generation Panel
2. Field reverts back to 100.

What steps will reproduce the problem?
1. Run a query
2. Double click on any row in the returned result set
3. Inspect the contents of the attestation time field in the displayed data

What is the expected output? 

A human readable date

What do you see instead?

The text of a java call that probably would generate a human readable date

e.g. 
Java.util.GregorianCalendar[time=-1246410000000,areFieldsSet=true,areAllFieldsSe
t=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="Europe/London",offset=0,
dstSavings=3600000,useDaylight=true,transitions=242,lastRule=java.util.SimpleTim
eZone[id=Europe/London,offset=0,dstSavings=3600000,useDaylight=true,startYear=0,
startMode=2,startMonth=2,startDay=-1,startDayOfWeek=1,startTime=3600000,startTim
eMode=2,endMode=2,endMonth=9,endDay=-1,endDayOfWeek=1,endTime=3600000,endTimeMod
e=2]],firstDayOfWeek=2,minimalDaysInFirstWeek=4,ERA=1,YEAR=1930,MONTH=6,WEEK_OF_
YEAR=27,WEEK_OF_MONTH=1,DAY_OF_MONTH=4,DAY_OF_YEAR=185,DAY_OF_WEEK=6,DAY_OF_WEEK
_IN_MONTH=1,AM_PM=0,HOUR=0,HOUR_OF_DAY=0,MINUTE=0,SECOND=0,MILLISECOND=0,ZONE_OF
FSET=0,DST_OFFSET=3600000]  

What steps will reproduce the problem?
1. From a SQL command line, issue DROP SCHEMA fakedata2
2. Start SnoFyre

What is the expected output? 

Re-creation of fakedata schema and then population with tables

What do you see instead?

No schema, no tables

What steps will reproduce the problem?
1. Reporting an error opens a link to a Google Form. This needs to point to 
Snofyre web site

What steps will reproduce the problem?
1. Need ability to export query results
2. Needs to be delimited text file at least
3. Needed for QA

What steps will reproduce the problem?
1. SNOMED CT Data currently distributed via TRUD Is Oct 2008 release
2. SNOMED CT data really need to be up to date

In various places clicking on an item in a list does not highlight the 
selected- this useful to have as assurance to the user. 

e.g. 

1. Load the attached query. 
2. In query authoring select Mild to moderate, Mycordial Infarction
3. Clicking on an item in the Excluded terms does not give an indication that 
this has been highlighted, so user not sure whether they can press the delete 
button. 


I would like to see selected items in a list highligted to give visual 
confirmation


Please use labels and text to provide additional information.

What is the proposed new feature or improvement?
Javadocs need to display versioning and add info on method implementation.

What is the proposed new feature or improvement?
We can make Snofyre's installation and configuration a lot more easier if we 
could package the underlying database with appropriate data and user 
permissions. However, we can not do this with MySQL due to other licensing 
restrictions. So its worth investing in migrating the backend from MySQL to 
PostgreSQL or another database with liberal licensing.

Version Information:
 -- Snofyre version      : 1.8

What steps will reproduce the problem?
1. Drop all fakedata2 tables
2. Start SnoFyre with option create tables on
3. Examine schema contents

What is the expected output? 

Lots of tables, recreated

What do you see instead?

Not tables, and a load of mySQL syntax errors in the logfile

Table create statements have 'type=myisam' argument, instead of 
'engine=myisam'. Older 'type=' format is deprecated from mySQL 4.1 and no 
longer works at all in current 5.5

What steps will reproduce the problem?
1. Add functionality to recreate the database
2. Need this for QA and validating data generation... among other things...