Git Product home page Git Product logo

opendis7-source-generator's Introduction

opendis7-source-generator

This project generates a type-safe Java implementation of the DIS Protocol version 7, IEEE Standard 1278.1-2012 from SISO and IEEE specifications. This project is written for Java developers working on autogenerated source-code production.

This project is not for end users or developers integrating DIS in other application tools. This project produces a fairly complete implementation of the IEEE DIS Protocol and associated enumerations that are integrated in the opendis7-java and related projects. In preparation: opendis7-python.

Recent project products include full Java support for all 72 DISv7 Protocol Data Units (PDUs) and over 22,000 SISO-REF-010 enumerations.

Additional testing and experimentation is performed in the NPS MOVES Networked Graphics MV3500 course.

Work in progress includes similar autogeneration of a complete opendis7-python version.

Background

This work is an updated continuation of the open-dis/xmlpg project created by the late Don McGregor of the Naval Postgraduate School (NPS).

Project goals are twofold:

  1. To provide reference implementations of the DIS protocol network messages in several programming languages.
  2. To do so by means of single XML descriptions of the protocol which are then referenced by individual language generators.

While there exists code in the project to generate source in JavaScript, Python and other languages, that code is legacy, and so far only the Java implementation is complete.

This work is driven by two specifications.

  • IEEE Std 1278.1-2012, IEEE Standard for Distributed Interactive Simulation—Application Protocols
  • SISO-REF-010 Enumerations, Simulation Interoperability Standards Organization (SISO).

The first reference describes the DIS protocol in detail -- specifying application algorithms as well as the precise format of network data. The second reference enumerates specific values for fields within the network data which correspond to actual entities in the real world.

The SISO specification is issued in several file formats. One of these is XML and the latest version of that file is used directly by this project. The IEEE specification is textual and initial work was required to describe its defined data structures in a set of XML files. Both the SISO file and the IEEE-based XML files used as input to this project are found in the xml subdirectory.

String templates for the various output classes are used to define the basic structure of the generated code, and these files are found in the [stringTemplates/edu/nps/moves/dis7/source/generator](https://github.com/open-dis/opendis7-source-generator/tree/master/stringTemplates/edu/nps/moves/dis7/source/generator} directory.

Development Environment

The Ant build.xml is greatly improved and build tasks are now simply performed. The underlying build process for autogenerating complex software library in separate projects like this is quite involved.

The Java language is inherently cross-platform and any OS on any hardware for which a Java run-time is available should support running of this project. However, the configuration used by the initial developer is the following:

  1. Apache Netbeans 20 Integrated Development Environment ("IDE")
  2. Apache Ant Java build tool version 1.10.14
  3. Git version control system (for downloading project; supported in Netbeans)
  4. OpenJdk Java version OpenJdk 21.0.1

Please see Savage Developers Guide to find our current recommended development settings for using each of these tools.

The project is hosted at github.com/open-dis/opendis7-source-generator and the support files which are used to define the project structure are also included. Following the procedure below, a simple download, then a small number of additional steps are all that are required to build the source files for a DIS distribution.

The project does not automatically download run-time dependencies like a Maven-based project. Only one external dependency is used, and that is the Apache Commons-IO library. The jar for that is found in the libs/ directory of the project.

  1. The generated source resides in the src-generated directory.
  2. The generated entity jars reside in the dist directory.
  3. The generated entity javadoc resides in the dist directory.

Project Directory Structure

The initial project directory structure looks like:

|-- images
|-- lib
|-- nbproject
|-- src-autogenerate
|   +-- edu
|       +-- nps
|           +-- moves
|               +-- dis7
|                   +-- source
|                       +-- generator
|                           +-- entityTypes
|                           +-- enumerations
|                           +-- pdus
|-- src-generated
|-- src-specialcase
|-- src-supporting
|-- stringTemplates
|   +-- edu
|       +-- nps
|           +-- moves
|               +-- dis7
|                   +-- source
|                       +-- generator
|                           +-- entitytypes
|                           +-- enumerations
|                           +-- pdus
+-- xml
    +-- xml/SISO
    +-- xml/dis_7_2012

After project execution, the directory tree will also contain:

|-- build
|-- dist
|   +-- javadoc
|-- test
  1. build -- generated directory holding products of the Java compiler
  2. dist -- generated jar files and javadoc, the products of the project
  3. images -- illustrative image files
  4. lib -- third-party Java libraries used by this project
  5. nbproject -- files supporting the Netbeans project structure
  6. src-autogenerate -- generator classes producing source files in Java, Python, etc.
  7. src-generated -- Java source file output from the source generator
  8. src-specialcase -- necessary DIS class files which required some modification after autogeneration
  9. src-supporting -- additional class files satisfying generated source dependencies
  10. stringTemplates -- supporting files, such as string templates
  11. xml -- SISO and IEEE-based XML files which serve as the input to the generator
  12. test -- unit tests for checking correctness are all migrated to populate opendis7-java tests

Products are then copied to the opendis7-java projects for further integration, testing and publication.

Project Internals

There are several logical output types described separately in the specifications.
This project processes them independently, i.e., the input XML is re-read for each type. Those types are:

  1. Protocol Data Units (PDUs)
  2. Enumerations
  3. Object types
  4. Radio Jammer types
  5. Entity types

PDUs and Enumerations are the most commonly used. The number of distinct entity types is large, which translates to a large number of Java classes. The use of the generated entity types is optional in a DIS application, since a DIS programmer managing a small number of entities may choose to manually insert the appropriate values into his/her data structures. For that reason, plus the fact that the entity type classes simply implement an abstract class by supplying 1-4 integer values, the source is not intended to be included in a distribution. The completed class jar files are available in the dist/ directory.

When the project is "run", as described above, the class which serves as the entry point, or "main", is src/edu/nps/moves/dis7/source/generator/Main. As mentioned above, the 5 types of Java classes which are generated are done so independently. To that end, the main entry just listed simply calls similar Java "main" methods in 5 separate classes:

  1. edu.nps.moves.dis7.source.generator.enumerations.GenerateEnumerations -- produces enumerations from the SISO specification
  2. edu.nps.moves.dis7.source.generator.pdus.Main -- produces Pdus and assorted sub-object classes from the IEEE-derived XML inputs
  3. edu.nps.moves.dis7.source.generator.entitytypes.GenerateJammers -- produces radio jammer classes from the SISO specification
  4. edu.nps.moves.dis7.source.generator.entitytypes.GenerateObjectTypes -- produces miscellaneous object classes from the SISO specification
  5. edu.nps.moves.dis7.source.generator.entitytypes.GenerateEntityTypes -- produces entity type classes from the SISO specification

Source Generation Method -- Pdus

This class contains remnants of legacy code which created pdus classes in different languages. The "JavaGenerator" subclass is the only one used in this project (to date).

edu.nps.moves.dis7.source.generator.pdus.Main first reads the IEEE XML files with a SAX parser and produces a map of class names-to-GeneratedClass objects. The GeneratedClass object contains fields which reflect the information contained in the XML for the particular pdu or object. This map is theoretically language-neutral is then used to produce pdus classes in various languages.

edu.nps.moves.dis7.source.generator.pdus.JavaGenerator then processes this map, generating source for each GeneratedClass object encountered. Template files are used so that the standard Java library String class may be used like the following:
String fileContents = String.format(template, value1, value2, value3 ...);

Source Generation Method -- Enumerations

These classes are simpler than Pdus and are created in a simpler way. The enumerated values in the SISO specification are implemented as either java Enumeration or java Bitset classes. (The latter uses an invented "BitField" class as a front end.)

A SAX ("Simple API for XML") Java implementation is used to process the XML input files. The generated uses SAX for operation when the "start" and "end" tags of the following elements are encountered:

  1. enum
  2. enumrow
  3. bitfield
  4. bitfieldrow
  5. dict
  6. dictrow

The enum, bitfield, and dict elements map to separate classes. The enumrow, bitfieldrow, and dictrow map to values within each of those classes.

When a SAX "end" element is encountered, the "current" element is written out as a complete Enumeration or BitField class, with enumerated values contained therein. Template files are used as above.

Source Generation Method -- Entity types, Object types, Jammers

These classes are also simpler than Pdus. Each of these three is hierarchically defined. For instance, a single Entity type is defined by entity, category, subcategory, specific and optional parameters, and not all of the sub parameters are required. Using a SAX parser, the SISO specification is read sequentially.

When a SAX "start" element is encountered, a new Java object is created -- one of EntityElem, CategoryElem, SubCategoryElem, SpecificElem, or ExtraElem. Since it is a hierarchical structure, when, e.g., a SubCategoryElem is encountered, the newly created object is inserted as a child of the previously created EntityElem.

When a SAX "end" element is encountered, the "current" element is written out as a complete EntityType class, with specific category, subcategory, etc., values. Template files are used as above.

Further work:

  • Refactor Java generator classes -- reasonably stable, minor refactoring occurs occasionally
  • Implement other language outputs -- Python work is in progress
  • Improve descriptions / javadoc in XML -- looking pretty good now!
  • Implement information "toString()" methods for classes like EulerAngles, EntityID, EntityKind

opendis7-source-generator's People

Contributors

arttu-salminen-insta-fi avatar blueoak avatar brutzman avatar mimmi-matalamaki-insta-fi avatar ricklentz avatar terry-norbraten avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

opendis7-source-generator's Issues

Wrong unmarshal of Domain in EntityType

Following issue is observable when EntityTypes with domains not equal to PlatformDomain: The unmarhalled domain will always be PlatformDomain (see c'tor of EntityType), unmarshalled with the numerical value of the other enum class.
Here the test:
ByteBuffer buffer = ByteBuffer.allocate(4096);
try {
// marshal/ unmarshal Domain: correct
Domain d = Domain.inst(MunitionDomain.ANTI_ARMOR);
d.marshal(buffer);
buffer.rewind();
d.unmarshal(buffer);
System.out.println(d.getDescription());
buffer.clear();
// marshal/ unmarshal EntityType incorrect for entity types not equal PlatformDomain
EntityType ent = new EntityType();
ent.setEntityKind(EntityKind.MUNITION).setCountry(Country.GERMANY_DEU).
setDomain(Domain.inst(MunitionDomain.ANTI_ARMOR)).setCategory(1).
setSubCategory(1).setSpecific(3);
ent.marshal(buffer);
buffer.rewind();
EntityType et = new EntityType();
et.unmarshal(buffer);
System.out.println(et.toString());
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
which gives following output:
Anti-Armor
EntityType entityKind:EntityKind 2 MUNITION domain:Air country:Country 78 GERMANY_DEU category:1 subCategory:1 specific:3 extra:0
where the MunitionDomain.ANTI_AIR is now PlatformDomain.AIR as PlatformDomain.unmarshal was called despite having EntityKind.Munition.

build.xml copy-generated-source-to-opendis7-java target adds to scr-generated folder files that are not generated by JavaGenerator

target name="copy-generated-source-to-opendis7-java" copies files to dir="${opendis7-java}/src-generated" that are not generated by JavaGenerator.java and are not found on opendis7-source-generator project src-generated folder. (opendis7-source-generator projects' src-generated folder contains only files generated by JavaGenerator.java.)
-> Confusing moments for developers.

Solutions idea: a new ant target to opendis7-source-generator projects' build.xml for copying files from src-specialcase folder to folder that has the same name in opendis7-java project.

Missing padding field in marshalling and unmarshalling ElectronicEmitter

If I understand correctly, I believe the ElectronicEmitter class is missing a 16-bit/2-byte padding field between numberOfBeams and emitterSystem that is missed during (un)marshalling (ref):

ie:

public int unmarshal(DataInputStream dis) throws Exception
{
    int uPosition = 0;
    try 
    {
        systemDataLength = (byte)dis.readUnsignedByte();
        uPosition += 1;
        numberOfBeams = (byte)dis.readUnsignedByte();
        uPosition += 1;
        dis.readUnsignedShort(); // missing this 2-byte padding buffer.
        uPosition += 2;
        uPosition += emitterSystem.unmarshal(dis);
        uPosition += location.unmarshal(dis);
        for(int idx = 0; idx < numberOfBeams; idx++)
.
.
.

... and in here:

public void marshal(java.nio.ByteBuffer byteBuffer) throws Exception
{
   byteBuffer.put( (byte)systemDataLength);
   byteBuffer.put( (byte)beams.size());
   byteBuffer.putShort((short)0); // needs to insert the (empty) padding field of the PDU.
   emitterSystem.marshal(byteBuffer);
   location.marshal(byteBuffer);
.
.
.

... and in the corresponding ByteBuffer methods for marshalling and unmarshalling too.

SISO-REF-010-v30-DRAFT-20220129-d11 enumerations

Testing latest release exposed some new issues, reported to [email protected] mailing list.

Overrides currently in place:

  1. Asked about warnings from empty XML children.

  2. <subcategory value="31" description="Hz 21\24 Fast Bridge" includes a backslash character, which is problematic and needs to be forward slash

  3. <enumrow value="2147483648" description="Rectangular Volume Record 4 value reduced by 1 as 2147483647 in order to remain a valid Java int

Unmarshaling causes NPE when enumerations not found

The current implementation of unmarshaling PDUs performs some enumeration lookups of the extracted values. If the value isn't found, it returns a null. This causes issues later in determining the unmarshalledSize() of the PDU, resulting in a NullPointerException (NPE).

One such scenario is seen in #14. ie: When some of the values are high end shorts (read as negative values), resulting in a failed enumeration lookup (no negative mappings in the enumeration classes), which returns a null for that instance. Later the parent object requests a .getMarshalledSize(), which performs the same thing on all member objects, and as one is a null the method bombs out.

But, as a lot of the Enumerations are filled with holes / incomplete, it's very easy to pull out a value that results in the null.

Example

In BeamStatus.java:getMarshalledSize() :

marshalSize += beamState.getMarshalledSize(); // When `beamState` is not mapped from the enumeration possibilities, this is `null`, resulting in a NPE.

Sure, this particular example should be either on or off, but due to how it is unmarshalled, I've seen other values extracted from the byte. This could be related to the next issue (#7 ) though.

There are other PDUs though, such as the name of an emitter / radio / etc where it may not exist either.

Suggestion

If the mapping is insisted at PDU generation time, perhaps ensure that allgenerated Enumeration classes from the SISO standard (SISO-REF-010-v28 at time of viewing) contain an "unknown"/"invalid"/"other" mapping, and return that instead of null? Some already do, but not all.

IFFPdu improvements

IFFPdu needs modifications so that Layer 3 information can be added to it. Layer 3 contains Mode 5 information.

Current IFFPdu only supports information for Layer 1 and 2. Current IFFLayer2Pdu extends IFFPdu which does not support to adding layer 3 information into IFFPdu.

Solution idea:

We would like to modify IFFPdu so that a layer information can be added directly to it and the layers do not need to extend IFFPdu.

Basically we would create corresponding data classes for layers 1, 2 and 3. (IffPduLayer1Data, IffPduLayer2Data, IffPduLayer3Data) These data classes would extend AbstractIffPduLayerData. Data classes can then be added to IffPdu via addLayer method. Then later when someone wants to create layer 4 and 5 they can simply create IffPduLayerData classes and add them to IffPdu.

Files that need modifications:

  • DistributedEmissionFamilyPdus.xml and JavaGenerator.java
    -- These for IffPdu, AbstractIffPduLayerData, IffPduLayer1Data, IffPduLayer2Data and IffPduLayer3Data
  • PduFactory.class on openddis7-java project

Is this solution okay to implement or do you have any suggestions for us how to improve it?

Project rename as opendis7-source-generator

A "hyphen-ectomy" is planned, we are preparing to rename project from

  • open-dis7-source-generator
    to
  • opendis7-source-generator

This simplification with lead to more consistency in deliverables without reducing capabilities. This only applies to the "7" series of codebases using the XML program generator.

  • opendis7-source-generator (mature)
  • opendis7-java (mature)
  • opendis7-python (in preparation)

Java doesn't have unsigned shorts, bitmasks occur instead

I think there might be an issue with some of the fields being read out / unmarshaled using .readShort(), as this results in high end shorts (>32k) being negative. This could cause a few issues with lookups of enumerations - if they were populated that high.

So far in various simulation tools, I've encountered issues with ElectronicEmitterPdu, EntityStatePdu and the classes used within (EntityID, etc) but I'm sure any part of the standard that references 2-bytes needs to be unsigned, so probably stored internally as an int that's masked (0xFFFF). Some are, but some aren't. Unsure on the lack of uniformity.

Raising this for further discussion on resolution or my use-case. :-)

TransmitterPdu has double extraction of RadioID

Hi again Don @brutzman,

I have noticed the TransmitterPdu class extracts the RadioID field (ref) once inside the header field (see RadioCommsHeader class), and again separately in the radioNumber field. this causes an unnecessary shift of unmarshalling by 2 bytes. Removing it from the RadioCommsHeader class fixes the problem, but so does commenting out the local field to the TransmitterPdu class, and redirecting all calls to get/set of the RadioID to the embedded RadioCommsHeader instance's field. I'm not sure which way you want to go.

Let me know if you're the best POC, or if there's a preferred approach to raising these issues.

Unsafe equalsImpl for object list attributes

Generated equalsImpl methods do unsafe checks for OBJECT_LIST type attributes. Method will throw IndexOutOfBoundsException when other object list is shorter that this object list. Equals method should not throw exceptions, list sizes should be compared before trying to access variables in the list.

Recalibration and testing

Much refactoring is nearly complete. open-dis7-source-generator products are feeding into the open-dis7-java build.

Still needed for release:

  • Enumeration low-level package naming normalized and unified,
  • better description string and uid value for enumerations,
  • better javadoc for enumerations,
  • Emitters enumeration currently truncated at 2000 entries to avoid Java "code too large" error,
  • reverse lookup from string, possibly other utility methods.

This work is being tested as part of a MOVES graduate course at NPS.

Status of project?

Hi!

I used the generated C++ bindings from open-dis-cpp to get started on a project and they work great! But I need some changes. So I was just about to fork XMLPG and start making the changes I need, but then I found this project.

  1. I gather this project intends to replace XMLPG but is still in development. Is that right?
  2. The README in this repository only describes generating Java code, but there exist generators for other languages. Which languages are supported?
  3. What is the state of the project for generating C++? Should I just continue with the XMLPG for now, if I need C++ marshaling / un-marshaling for DIS messages?

Thanks so much, and congrats on the useful libraries!

enum values with leading zeros

There is a nasty issue for enumerations, where the enumeration value is initialzed by an integer with leading zeroes (this will be interpreted as an octal number by the compiler), e.g. for MunitionDescriptorFuse, where MULTIFUNCTION should have the decimal value 100, but the constructor in the source is MULTIFUNCTION(0100, "Multifunction"), which gives you a decimal value of 64. To my knowledge this is also valid for MunitionDescriptorWarhead, I have not searched for other cases, as one has to check the whole SISO-REF-010 enumerations.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.