How much time does this library take to parse a file?

What if the file is very large? What if there are a lot of errors?

Are there tricks you can do to speed up the analysis the library does? Do other libraries have tricks we can learn from?

I see the rules here, which is great:

https://github.com/intermine/biovalidator/blob/master/src/main/java/org/intermine/biovalidator/validator/fasta/FastaValidator.java#L42

But it's a bit hidden. Can we make it more prominent somehow? We want this available by the command line and in github.

We want to publish this JAR so others can use this library.

Here are where we publish our InterMine JARs: https://bintray.com/intermineorg

When do we want to publish? Maybe first stable version.

Can you add some comments to extractSequenceIdFromHeader()? Specifically because I am wondering why split() wouldn't be appropriate (I actually don't know!). but in looking at the function it's a little hard to understand what's going on.

Thanks!

I get this error message when I don't provide a filetype:

java.lang.IllegalArgumentException: Invalid validator type
	at org.intermine.biovalidator.validator.ValidatorType.of(ValidatorType.java:43)
	at org.intermine.biovalidator.api.ValidatorHelper.validate(ValidatorHelper.java:70)

Instead can we do either (if filetype is NULL):

a. guess
b. give an error message "filetype required: FASTA or GFF3"

We use semantic versioning at InterMine:

https://semver.org/

Thoughts?

I am guessing you plan to update to 1.0 (or 1.0.0!), e.g. no SNAPSHOT, when we are happy with GFF3 and want to publish to Maven.

This is me being lazy and not wanting to iterate! For discussion!!

This is my code snippet:

            List<Message> messages = validationResult.getErrorMessages();
            StringBuilder errorMessage = new StringBuilder();
            for (Message msg : messages) {
                errorMessage.append(msg.getMessage());
            }
            return errorMessage.toString();

I would like to do this instead:

            return validationResult.getErrorMessage();

For the strict setting we'll only have one error message anyway.

We want to give the user enough information to fix the problem.

This goes for all error messages but specifically file not found. Update this error message to include the path that was tried.

Again, when I didn't provide the filetype, I got an illegal argument exception. Instead, I would like an informative error message "filetype required: FASTA or GFF3".

What are fatal exceptions for? They are to stop the process because there is simply no point in going on. Something very bad has happened. In our case, that doesn't really apply. Does that make sense? Something bad has happened (e.g. no filetype), but we don't need to kill the process, we only want to stop what we are doing and return the relevant information to the user.

Exceptions in Java also let you do custom actions for certain exceptions, e.g. close database connections, write to the log etc. We don't have those in our case, we always only ever want to return the error message to the user.

I think your error handling is correct however, it makes using the library more difficult. Look at my code, I have to catch a bunch of exceptions, then check for NULL then check the error messages. Very messy! It's not maintainable -- later on you may change the exceptions, people will forget what exceptions are thrown etc. And I am finding it really hard to write my unit tests.

This is a specific library to do a specific job. These fatal errors don't help do that job. Because this library is part of a big application, your library should catch the errors, and return an informative error message to the user so they can act on them.

Thoughts?

So I checked both so.obo and so-simple.obo and turns out both files do not have an exact match of 'DNaseI_hypersensitive_site' and 'recombination_region' but
1. for type 'DNaseI_hypersensitive_site' -> 'so.obo' and 'so-simple.obo' do have 'DNAseI_hypersensitive_site' here the case of third letter is difference, its capital 'A'.
2. for type 'recombination_region' -> 'so.obo' and 'so-simple.obo' have 'mitotic_recombination_region', 'non_allelic_homologous_recombination_region', etc. 

See intermine/intermine#1828 for details.

This problem has been fixed but there are older files that have the incorrect spelling.

@rachellyne since this is the NCBI only I think we should be case sensitive and this file should fail. What do you think?

1. generate javadoc (./gradlew javadoc)
2. put generated javadoc directory on gh-pages branch
3. push gh-pages branch to github

We have our intermine.org domain mapped to github pages so they should end up here: http://intermine.org/biovalidator/javadoc/

In the app (and in biovalidator) we use enums to represent the file formats.

Here is the spec: openapi.json

This is the Java representation of that: DataFile.java

This is fragile. I guess I see this as a place for future bugs! Right now we are okay with just the handful of file types we have. But in the future?

Ideally we would have a standard that we would use. I don't think there is any "good" answer. But we can mitigate the danger by:

• put a comment in configurator to be sure to check valid types in biovalidator
• Have biovalidator be very permissive with the file type names, e.g. case insensitive, accept "GFF" and "GFF3"

(Just because I am thinking about error messages, and I have NOT tested this.)

What happens when the GFF file has the wrong number of columns. Are we going to get an error message for each bad row? It could be that we have some validation rules that are only enforced once. e.g. check for number of columns each row. but on fail, don't check again.

Ignore if you've handled this!! :)

In the column attributes a value as

ke1=value1;key2;key3=value3
should not pass, I think
Documentation here