atlasmap / atlasmap Goto Github PK

Part of epic #248

@mattrpav	2017-09-27	enhancement

Discussed w/ the team the idea of an AtlasGroupStrategy that would be a pre-cursor to conditional logic, expressions and advanced Field Actions (including Math related on non-Collections).

The GroupStrategy would collect source values and have a FieldGroup collection of sorts to pass to FieldActions that support FieldGroup as an input would be responsible for creating the single value output.

Another thought was that the CombineStrategy could be converted to a Group-of-Field field Action and then deprecate the CombineStrategy in the engine.

The UI could then still have a first class mapping action called 'Combine', that would create a group strategy and apply a combine field action

@mattrpav	2017-07-06	housekeeping

Naming convention for stuff.. we have a bit of a mix right now. Probably a good idea to settle on one set of terms and align it b/w the UI, services and the engine.

1. Input & Output
2. Source & Target

Part of epic #248

@igarashitm	2017-10-03	enhancement

• MappingType: COMBINE
• Source field: some Strings, like

{
  "firstname": "foo",
  "lastname": "bar",
  "phone": "111-111-1111"
}

• Target field: String
• User input to the field action: "You have a message from ${firstname} ${lastname}, call ${phone}."
• Expected output: "You have a message from foo bar, call 111-111-1111."

FieldAction name would be 'template' or something.

@igarashitm	2017-08-09

Right now it requires AtlasMappingFormat as an parameter on loadMapping() and saveMappingAsFile(), but it would be better to let AtlasMappingService detect the format so that the invoker doesn't have to worry about the format. If it's File/URL/URI then it's easy to detect from file extension, but if it's Stream/Reader then it has to read the content and check if there's "<" or "{". Also it needs to consider about the character encoding.

TODO : put a real usecase for the type convert field action here

@mattrpav commented on Thu Jul 27 2017

The Xml module uses XmlPathCoordinate and should be migrated to the common AtlasPath handler, or an extension of the AtlasPath class that supports namespaces

@igarashitm	2017-09-29	enhancement

This is required for #78

Some of the field action have COMPLEX as a source/target type, which won't appear in field action list as COMPLEX field is not mappable in UI. Check all the field actions appear for corresponding pair of field types, then process that mapping and verify those field actions work as expected.

@mattrpav	2017-07-07	enhancement

For initial release support UTF-8 and default locales throughout

@igarashitm commented on Tue Oct 10 2017

Right now DefaultAtlasContext doesn't invoke module.processOutputActions(), but UI put output field action for the COMBINE mapping. See if it should be enabled or remove completely.

@mattrpav	2017-07-24	enhancement

1. Classes without a no-arg constructor
2. Classes with multiple no-arg constructors (what/how to inform the user at design time?)
3. Support invoking a method post construction (ie. init()/destroy() style Java classes)

@igarashitm	2017-09-29	enhancement

Implement field actions marked as 'p1' in the list. There're 128 remained as of Sep 29, 2017.

@mattrpav	2017-07-26	housekeeping

The JavaFieldWriter, JsonFieldWriter and XmlFieldWriter classes are very similar and would allow for conversion to be generic.

This will lay the foundation for the modules to be generic as well.

@igarashitm commented on Fri Oct 27 2017

https://github.com/atlasmap/atlasmap-runtime/blob/master/atlas-model/src/main/resources/atlas-model-v2.xsd#L152

Right now JsonField and JsonComplexType doesn't have any constraint for the fieldType in the schema, but JsonComplexType should only be used for 'Complex' and JsonField for others. See if possible to reflect that constraint in the XML schema. XML model as well.

We should enforce it in checkstyle for runtime and camel

@igarashitm commented on Wed Nov 08 2017

It looks Byte related converters are all marked as UNSUPPORTED but shouldn't be, easy to implement. Also NUMBER type conversion must be provided OOTB as it often appears in the JSON schema.

BYTE -> Short
BYTE -> String (string representation of the number)
...

-127 to 127
Int, Short, Long -> BYTE (range concern)
String -> BYTE (range concern and type concern)
Decimal -> BYTE (precision concern.. if 1.2 comes in as a source value throw an exception)

@igarashitm	2017-10-09	housekeeping

I have hit/fixed several bugs caused by the lack of dependency specific to the runtime. They should have been detected by the tests. Introduce runtime tests which run atlasmap spring-boot runtime and test if all the services are working properly, including each modules.

Make sure all the TypeConverter can be utilized from the mapping created via UI.

At least we have a bug in DateConverter, it has COMPLEX as a source or target, which won't be picked up as UI doesn't define COMPLEX as a source/target field type.

@mattrpav	2017-07-07	housekeeping

The atlas-model module includes Jackson JsonDeserializer and JsonSerializer classes for prettifying AtlasMapping json definitions to provide type-free, extensible field action support while maintaining a DSL-like syntax in JSON (say that 10x fast).

In doing so, there is no mechanism to output unsupported or type-o'd field action names. For example, if someone has "puppercase" instead of "Uppercase" in the json, we quietly skip over and continue on.

Option 1: Add slf4j dependency to the model class module

Option 2: Break off the ActionsJsonDeserializer, ActionsJsonSerializer, AtlasJsonModule and AtlasJsonMapper classes to a separate module

"outputField" : [ {
"jsonType" : "io.atlasmap.v2.MockField",
"actions" : [ {
"Uppercase" : null
}, {
"Lowercase" : null
}, {
"SubString" : {
"startIndex" : 2,
"endIndex" : 5
}
}, {
"SubStringAfter" : {
"match" : "a",
"startIndex" : 2,
"endIndex" : 5
}
}, {
"SubStringBefore" : {
"match" : "z",
"startIndex" : 2,
"endIndex" : 5
}
} ],
"value" : "map-output-value",
"fieldType" : "STRING",
"name" : "map-output"
} ]

@igarashitm	2017-09-07	enhancement

In following JSON schema 'ref'-c' refers 'urn:d' before it's defined in schema, which is not supported for JSON schema inspection today.

{
    "$schema": "http://json-schema.org/schema#",
    "type": "object",
    "properties": {
        "ref-a": { "$ref": "urn:a" },
        "ref-b": { "$ref": "urn:b" },
        "ref-c": { "$ref": "urn:d" },
        "def-d": {
            "$id": "urn:d",
            "type": "object",
            "properties": {
                "str-d": { "type": "string" }
            }
        }
....

As long as it's defined in definitions section, order doesn't matter. It's a problem only on a embedded definition.

This could be achieved in Java module by using a root path / for the value. Easier than introducing atlas:core module.

@mattrpav	2017-07-27	enhancement

AtlasMap supports processing repeated items in a Collection. It would be handy to be able to support a mapping where you can map a specific field out of an indexed collection to a target field.

source path: /order/address[1]/email
target path: /contact/primaryEmail

@mattrpav	2017-07-24	enhancement

Java class inspection edge case - field and setter/getter conflict

Scenario:

1. Java class contains an int field named "myInt" with no getter/setter operating on an Integer/int type
2. Java class contains methods named String getMyInt()/setMyInt(String myInt)
3. Need a merge strategy when there are two potential AtlasMap "Fields" that have the same name

Note: An approach could be to use "@" to indicate field vs method

ie..

/Address/@street <-- Java field
/Address/street <-- Java get/set method

Advanced use case:
Extended class implements same field as super class

Result, 2 field definitions:

• /Address/@street <-- Class
• /Address/@street?fieldIsSuperClass=true . <-- SuperClass

Right now the directory names for the runtime is a bit verbose. It doesn't have to be same with maven module name. For example atlas-api could be just api, and atlas-{java,json,xml}-parent could be just java, json, xml. Modules could be even moved into modules subdir without changing maven hierarcy.

@igarashitm	2017-10-19	enhancement

See #89

xref: https://issues.redhat.com/browse/ENTESB-12408

We would be able to:

• UI
• • Show everything wrapped with those rules
• • Show some decoration for the rule
• Runtime
• • Error out if source document violates the rule
• • Error out if target document outcome violates those rule as a result of whole mappings
    But we can't determine if it violates the rule until we get source document instances and process mappings.

An example from camel-salesforce here

{
  "type": "object",
  "id": "urn:jsonschema:org:apache:camel:component:salesforce:dto",
  "$schema": "http://json-schema.org/draft-04/schema#",
  "oneOf": [
    {
      "id": "urn:jsonschema:org:apache:camel:component:salesforce:dto:QueryRecordsMerchandise__c",
      "properties": {
        "done": {
          "type": "boolean"
        },
        "totalSize": {
          "type": "integer"
        },
        "nextRecordsUrl": {
          "type": "string"
        },
        "records": {
          "type": "array",
          "required": true,
          "items": {
            "type": "object",
            "$ref": "urn:jsonschema:org:apache:camel:component:salesforce:dto:Merchandise__c"
          }
        }
      }
    },
    {
      "id": "urn:jsonschema:org:apache:camel:component:salesforce:dto:Merchandise__c",
      "title": "Merchandise",
      "properties": {
        "Id": {
          "type": "string",
          "required": true,
          "readonly": true,
          "title": "Record ID"
        },
        "OwnerId": {
          "type": "string",
          "required": true,
          "readonly": false,
          "title": "Owner ID"
        },
        "IsDeleted": {
          "type": "boolean",
          "required": true,
          "readonly": true,
          "title": "Deleted"
        },
        "Name": {
          "type": "string",
          "readonly": false,
          "title": "Merchandise Name"
        },
        "CreatedDate": {
          "type": "string",
          "required": true,
          "readonly": true,
          "title": "Created Date",
          "format": "date-time"
        },
        "CreatedById": {
          "type": "string",
          "required": true,
          "readonly": true,
          "title": "Created By ID"
        },
        "LastModifiedDate": {
          "type": "string",
          "required": true,
          "readonly": true,
          "title": "Last Modified Date",
          "format": "date-time"
        },
        "LastModifiedById": {
          "type": "string",
          "required": true,
          "readonly": true,
          "title": "Last Modified By ID"
        },
        "SystemModstamp": {
          "type": "string",
          "required": true,
          "readonly": true,
          "title": "System Modstamp",
          "format": "date-time"
        },
        "LastActivityDate": {
          "type": "string",
          "readonly": true,
          "title": "Last Activity Date",
          "format": "date-time"
        },
        "LastViewedDate": {
          "type": "string",
          "readonly": true,
          "title": "Last Viewed Date",
          "format": "date-time"
        },
        "LastReferencedDate": {
          "type": "string",
          "readonly": true,
          "title": "Last Referenced Date",
          "format": "date-time"
        },
        "Description__c": {
          "type": "string",
          "readonly": false,
          "title": "Description"
        },
        "Price__c": {
          "type": "number",
          "required": true,
          "readonly": false,
          "title": "Price"
        },
        "Total_Inventory__c": {
          "type": "number",
          "required": true,
          "readonly": false,
          "title": "Total_Inventory"
        }
      }
    }
  ]
}

@igarashitm commented on Mon Oct 30 2017

https://github.com/atlasmap/atlasmap-runtime/blob/ce498616965254723428641ab5c03ed80cff4d40/atlas-core/src/main/java/io/atlasmap/core/DefaultAtlasConversionService.java#L279-L306

Yes it's definitely TODO. Even DateConverter is not available.

@igarashitm	2017-10-20	enhancement

Check the fields listed in the required property and add required flag in the inspection result. Also see if XML schema inspection does it.

Don't forget to add Labels for each submodules

I suspect Enum field without getter/setter is not handled here. Add a unit test to inspect a class which has a Enum field without getter/setter, and see if it's in the inspection result. If not, implement it.

@mattrpav	2017-09-27	document

AltasSession: Per execution of mapping data
AtlasContext: Per mapping definition
AtlasContextFactory: Global for assembling all Atlas components.. modules, field actions, validations, converters, etc.

@mattrpav	2017-07-12

• Update Class.forName() references to specify class loaders
• Test installation and bundle wiring in an OSGi runtime (such as Karaf)
• Write an integration test that confirms data processing in Karaf or other OSGi runtime

@jbakermd	2017-07-20	enhancement

Character encoding / locale information needs to be added to inspection and mapping.

Add the following two properties:

• characterEncoding (string)
• locale (string)

To the following classes:

• io.atlasmap.v2.Document (json/xml inspection)
• io.atlasmap.java.v2.JavaClass (java inspection)
• io.atlasmap.v2.DataSource (mapping data source info pass through)

Please cc me when this is finished so I can test it on the UI.

Related ticket: #70

@mattrpav	2017-07-23	enhancement

Currently, each read / write instantiates a document. This should be converted to support an overloaded method where the node can be passed in for subsequent calls

@zregvart	2017-10-02	enhancement

Currently there is only support for String in the input of JSON and XML modules, we should support InputStream as input.

See XmlModule.java#L167-L171 and JsonModule.java#L145-L150.

@mattrpav commented on Wed Jul 12 2017

1. Add unit tests to serialize and deserialize reference mappings to and from JSON and XML
2. Validate all fields are processed correctly

@igarashitm	2017-10-15	enhancement

Right now Java field writer only supports concrete class which can be instantiated via no-arg constructor. See createObject() in this class:
https://github.com/atlasmap/atlasmap-runtime/blob/master/atlas-java-parent/atlas-java-module/src/main/java/io/atlasmap/java/module/DocumentJavaFieldWriter.java

One option is to feed a mapping between interface/abstract types and default impl classes from configuration or something.
java.util.List : java.util.ArrayList
java.util.Map : java.util.HashMap
...

And it also should be configurable from mapping file so that the user can define their own types.

so that the camel-atlasmap can synchronize with runtime version, and also centralize dependency managemewnt.

@igarashitm commented on Tue Nov 07 2017

Poor JavaModule, don't look into Constants...

@igarashitm commented on Tue Nov 07 2017

Don't forget to check JSON/XML modules as well

@mattrpav summarizing comments up to 12/19:

Java, Xml and Json Modules should not create validation messages for: ConstantField, PropertyField, LookupField and other common *Field types from atlas-core.

@mattrpav	2017-07-26	enhancement

Required to fix for Stand alone data mapper
And also required to support custom connector in Syndesis

@mattrpav	2017-07-26

1. When specifying a mapped field in Xml, an empty element may be interpreted as ""
   ie. The desired behavior of the value may be "" or null
2. We need a config option in order to tell XmlFieldReader what to do
3. The issue crops up w/ Xml to Json where the "" fields are being written : { myField: "" }

@jpav commented on Thu Nov 02 2017

Code near the end of the method that determines whether a response should be generated indicating a converter is needed to perform a mapping has been left incomplete. We should determine the necessity of this code and either complete its implementation or remove it from the class.

@igarashitm commented on Thu Nov 02 2017

Weird thing is that we sometimes see "converter is not available" warning in UI. I thought it came from this converterCheck. Need to investigate.

@igarashitm commented on Wed Nov 08 2017

I now suspect the converterCheck is not used at all. All the converter validation results I have ever seen come from validate. Make sure UI doesn't invoke it and remove.

@dsimansk	2017-07-17

When trying to do Twitter Mention -> Salesforce Create/Update Contact, I can see the following exception.

Could not load JAVA document. Document is not found: io.syndesis.connector.salesforce.Contact

@apupier	2017-09-19

a warning is reported by Eclipse

the possible issue is that the files on the classpath won't be deleted (on windows) because a lock will be keep on it.

@igarashitm	2017-09-29	bug

Right now it's invoking only per module validations. It should just invoke processValidation() instead.

@mattrpav commented on Wed Jul 19 2017

• DefaultAtlasCombineStrategy (100%)

• DefaultAtlasContext

• DefaultAtlasContextFactory

• DefaultAtlasConversionService (74%)

• DefaultAtlasPropertyStrategy (84%)

• DefaultAtlasSeparateStrategy (100%)

• DefaultAtlasSession (100%)

• DefaultAtlasValidationService (88%)

• StringDelimiter (86%)

• io.atlasmap.actions (85%)

• io.atlasmap.converters (99%)

• io.atlasmap.validators (70%)

@mattrpav	2017-07-20	enhancement

The io.syndesis.connector.salesforce.Contact object is heavily generic, parameterized and extended. This results in a lot of unnecessary inspection on inner member fields that do not need to be presented to the end user.

@apupier	2017-09-19

the default is 1.5 https://maven.apache.org/plugins/maven-compiler-plugin/

it would be nice to upgrade to 1.8 to be able to leverage new java idioms