Vulnerability

Determine if the findings about  results if an integer multiplication being cast to a long are valid, and determine the potential consequences if they are.

Software Version

Current

💡 Description

Per comments on this this PR, there is way too much use of the generic exception in this code, and overuse of core java exceptions that make no sense when applying to PDS (e.g. ArrayIndexOutOfBounds exception should not be used when we are talking about overlapping fields in a PDS array data object).

Additionally, per the comment on PR, could use a comment when some values are very PDS domain specific.

🐛 Describe the bug

This code in FieldType.java looks wrong:

   ASCII_NUMERIC_BASE16("ASCII_Numeric_Base16", new NumericTextFieldAdapter(), true),

in view of the ASCII_Numeric_Base16 type definitiion.
The NumericTextFieldAdapter includes code like:

    @Override
    public long getLong(byte[] buf, int offset, int length, int startBit, int stopBit) {
        return Long.parseLong(getString(buf, offset, length, startBit, stopBit).trim());
    }

which wil fail (with a java.lang.NumberFormatException) if fed the kind of text expected for Base16.

📜 To Reproduce

Steps to reproduce the behavior:
try to use pds4-jparser to read values from a table with ASCII_Numeric_Base16 data. I have an example data file of this sort, but the provider doesn't want to post it publicly.

📚 Version of Software Used

I'm looking at the source code on master at time of writing, 194b5f9.

Suggested fix

I suggest either to introduce a new RadixTextFieldAdapter for use with ASCII_Numeric_Base16, ASCII_Numeric_Base8 and ASCII_Numeric_Base2 (I have prototyped this and can make a Pull Request on request). Failing that, fall back to the DefaultFieldAdapter for ASCII_Numeric_Base16, as is already done for ASCII_Numeric_Base8 and ASCII_Numeric_Base2.

🦄 Related requirements

⚙️ Engineering Details

Describe the bug
When trying to run mvn test, was getting this error:

java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()

To Reproduce

mvn test

Proposed solution
See https://stackoverflow.com/questions/61267495/exception-in-thread-main-java-lang-nosuchmethoderror-java-nio-bytebuffer-flip to cast ByteBuffer as Buffer

((Buffer) bb).flip();

Expected behavior
Test completes successfully.

Desktop (please complete the following information):

• OS: Mac OS
• Java Version

openjdk version "1.8.0_265"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_265-b01)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.265-b01, mixed mode)

Vulnerability

Determine if the findings about SQL queries containing user input that is not sanitized are valid, and determine the potential consequences if they are.

Software Version

Version

Vulnerability

Determine if the findings about string object comparison issues do exist and determine the potential consequences if they are.

🐛 Describe the bug

Per jetty/jetty.project#3244

Like many other projects (Tomcat, MongoDB, etc.) compiling with JDK 9+ with release 8 produces incorrect bytecode for these ByteBuffer methods:

• position(int)
• limit(int) None
• mark​()
• reset​()
• clear()
• flip()
• rewind​()

Running a Jetty compiled with JDK 9+ with release 8 in a JDK 8 JVM will produce a NoSuchMethodError.

The solution is to cast the ByteBuffer to Buffer when calling those methods:

((Buffer)byteBuffer).position(0);

Is your feature request related to a problem? Please describe.
When validating a Table_Delimited PDS4 product, validate tool takes as much memory as 12 times the size of the data file. This makes difficult to validate products close to the gigabyte (like highgate magnetometer data) as they require to run validate with a large portion of the available memory.

See the following other tickets for more details. Any improvements should be sure the following tickets are also verified as successfully tested.

• NASA-PDS/transform#2
• NASA-PDS/validate#155
• NASA-PDS/validate#189

Debugging this issue I found that most of the memory allocation is in pds4-jparser's gov.nasa.pds.objectAccess.TableReader class, specifically in line 120 which loads the whole data file.
delimitedRecordList = csvReader.readAll();

Describe the solution you'd like
An alternative table delimited read implementation that does not consume so much memory.

Describe alternatives you've considered
Reading the data sequentially using csvReader.readNex(); but the problem of this approach it doesn't consider cases where the current row pointer is reset while reading the file, and that might happen as TableReader declares:
public TableRecord getRecord(int index) throws IllegalArgumentException, IOException public void setCurrentRow(int row)

Additional context
Memory footprint validating a 720MB product:

Memory footprint validating a 720MB product with the alternative described:

Modifications to TableReader used for the second case:
@@ -65,7 +65,7 @@ public class TableReader {

•   private List<String[]> delimitedRecordList;
  

•   //private List<String[]> delimitedRecordList; 
  

@@ -117,7 +117,7 @@ public class TableReader {

•                   delimitedRecordList = csvReader.readAll();
  

•                   //delimitedRecordList = csvReader.readAll();
  

@@ -188,7 +188,7 @@ public class TableReader {

•                   String[] recordValue = delimitedRecordList.get(currentRow-1);
  

•                   String[] recordValue = csvReader.readNext(); 
  

Checked for duplicates

No - I haven't checked

🧑‍🔬 User Persona(s)

Data Engineer

💪 Motivation

...so that I can have the ability to read and write 4D arrays

📖 Additional Details

No response

Acceptance Criteria

Given a 4D array product
When I perform a read() call
Then I expect to be able to read the object into memory

⚙️ Engineering Details

No response

🎉 I&T

No response

💡 Description

We need methods to:

• get the name of the array object
• get the entire 1D array object

Tracking issue for:

• https://github.com/NASA-PDS/pds4-jparser/security/code-scanning/1

💡 Description

@msbentley filed a regression in validate that demonstrated a NullPointerException that appeared in validate-2.4 (and 3.0) that wasn't present in 2.3.

@nutjob4life created a pull request that addressed the issue and made the exception go away. However, he also opined that the fix didn't feel "right" since the code addressed in the PR hasn't really changed in quite some time—and now suddenly it's producing NPEs? 🧐

@jordanpadams requested a new issue for him to go back through the recent and large changes to pds4-jparser and validate and see if there's something else going on—or if PR#63 is indeed appropriate.

🐛 Describe the bug

Project doesn't build due to XJC compilation errors.

📜 To Reproduce

Steps to reproduce the behavior:

git clone [email protected]:NASA-PDS/pds4-jparser.git
cd pds4-jparser
mvn clean package

output:

[xjc] [WARNING] schema_reference.4: Failed to read schema document 'schema-stripped.xsd', because 1) could not find the document; 2) the document
could not be read; 3) the root element of the document is not <xsd:schema>.
[xjc]   line 13 of file:C:%5Cws%5Cgithub%5Cpds4-jparser%5Ctarget%5Cgenerated-sources%5Ctemp%5CPDS4_DISP_1G00.xsd
[xjc]
[xjc] [ERROR] src-resolve: Cannot resolve the name 'pds:ASCII_Integer' to a(n) 'type definition' component.
[xjc]   line 111 of file:C:%5Cws%5Cgithub%5Cpds4-jparser%5Ctarget%5Cgenerated-sources%5Ctemp%5CPDS4_DISP_1G00.xsd
[xjc]
[xjc] [ERROR] src-resolve: Cannot resolve the name 'pds:Units_of_Frame_Rate' to a(n) 'simpleType definition' component.
[xjc]   line 150 of file:C:%5Cws%5Cgithub%5Cpds4-jparser%5Ctarget%5Cgenerated-sources%5Ctemp%5CPDS4_DISP_1G00.xsd
[xjc]
[xjc] [ERROR] src-resolve: Cannot resolve the name 'pds:Units_of_Time' to a(n) 'simpleType definition' component.
[xjc]   line 236 of file:C:%5Cws%5Cgithub%5Cpds4-jparser%5Ctarget%5Cgenerated-sources%5Ctemp%5CPDS4_DISP_1G00.xsd

🕵️ Expected behavior

Builds successfully

📚 Version of Software Used

2.1.0-SNAPSHOT (2021-06-17)

🩺 Test Data / Additional context

🏞Screenshots

🖥 System Info

• OS: Windows 10
• JDK 11

🦄 Applicable requirements

N/A

⚙️ Engineering Details

• removing parent POM reference and adding all that info into the POM appears to work better

The 2.0.0 release of pds4-jparser contains SNAPSHOT dependencies in its pom.xml. This means that this release can not be built using publicly accessible maven repositories. More importantly, downstream projects that depend on pds4-jparsers:2.0.0 also can not be built using public maven repositories. For example, the current master branch of https://github.com/NASA-PDS/validate.
The two SNAPSHOT dependencies are gov.nasa.jpl.ammos.ids.vicario:vicario:48.0.3-SNAPSHOT and com.opencsv:opencsv:5-4-SNAPSHOT.

Potential fix: Release pds4-jparser:2.0.1 with no SNAPSHOT dependencies.

💡 Description

Eclipse environment is unhappy with registry-api and pds4-jparser because the former uses version 4 (antlr4 technically) and latter references but does not use 3. Removing it from POM makes eclipse happy and not having unused dependencies in the POM makes it more robust.

Tracking issue for:

• https://github.com/NASA-PDS/pds4-jparser/security/code-scanning/190
• https://github.com/NASA-PDS/pds4-jparser/security/code-scanning/191
• https://github.com/NASA-PDS/pds4-jparser/security/code-scanning/192
• https://github.com/NASA-PDS/pds4-jparser/security/code-scanning/193
• https://github.com/NASA-PDS/pds4-jparser/security/code-scanning/194
• https://github.com/NASA-PDS/pds4-jparser/security/code-scanning/195
• https://github.com/NASA-PDS/pds4-jparser/security/code-scanning/196
• https://github.com/NASA-PDS/pds4-jparser/security/code-scanning/197
• https://github.com/NASA-PDS/pds4-jparser/security/code-scanning/198
• https://github.com/NASA-PDS/pds4-jparser/security/code-scanning/199

Summary

Per the known vulnerability, we want to upgrade log4j to the latest 2.* version, which would be switching over to using log4j-core and log4-api instead of the current log4j:

Per the POM:

    <dependency>
      <groupId>log4j</groupId>
      <artifactId>log4j</artifactId>
      <version>1.2.16</version>
      <scope>test</scope>
    </dependency>

This is a low priority because the scope when this is used is during the test phase, but we still want to update eventually.

Vulnerability

Describe the vulnerability
Resolves https://github.com/NASA-PDS/pds4-jparser/security/dependabot/1
Resolves https://github.com/NASA-PDS/pds4-jparser/security/dependabot/2
Resolves https://github.com/NASA-PDS/pds4-jparser/security/dependabot/3
Resolves https://github.com/NASA-PDS/pds4-jparser/security/dependabot/4

Software Version

Version
Latest

• https://github.com/NASA-PDS/pds4-jparser/security/code-scanning/2

Is your feature request related to a problem? Please describe.
As of Java 9, Java EE modules have been slowly removed over time until Java11 where JAXB was also removed. I tried numerous additions to the POM to fix this, but I believe the issue is with the ant classpath, but not sure.

https://www.jesperdj.com/2018/09/30/jaxb-on-java-9-10-11-and-beyond/

See NASA-PDS/validate#2 and NASA-PDS/validate#3 for suppression of WARNING and INFO messages from JAXB dependency.