Git Product home page Git Product logo

mfatihercik / dsm Goto Github PK

View Code? Open in Web Editor NEW
29.0 5.0 5.0 4.84 MB

Declarative Stream Mapping (DSM) is a stream de/serializer library for XML and JSON. DSM allows you to make custom parsing, filtering, transforming, aggregating, grouping on any JSON or XML document at stream time(read only once).

Home Page: https://mfatihercik.github.io/dsm

License: MIT License

Java 86.64% Groovy 13.36%
xml json-parser xml-transformation json-transformation xml-parser xml-stream-parsing json-stream xml-mapping json xml-parsing

dsm's People

Contributors

mfatihercik avatar qxo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

dsm's Issues

Filtering

Firstly, congratulations on the project, it's quite creative. For the filter, let's suppose I have a source.json
//source.json
[
{
"provider": "VODAFONE"
},
{
"provider": "VERIZON"
},
{
"provider": "TEXAS"
}
]

//config
version: 0.1.0
result:
type: array
path: /
filter: $self.data.provider == 'TEXAS' || $self.data.provider == 'VODAFONE'
#filter: $self.data.provider == 'TEXAS' or $self.data.provider == 'VODAFONE' i tried it too
fields:
provider: string

I would like to filter by texas and vodafone, but it is returning only records for the first branch, maybe i lost something.

One more duplicate data exists when converting json array

JSON:

 [

  {
    "id": 8531,
    "category": {
      "id": 0,
      "name": "string"
    },
    "name": "doggie",
    "photoUrls": [
      "string"
    ],
    "tags": [
      {
        "id": 3,
        "name": "string"
      }
    ],
    "status": "available"
  },
  {
    "id": 9009,
    "category": {
      "id": 0,
      "name": "string"
    },
    "name": "doggie",
    "photoUrls": [
      "string"
    ],
    "tags": [
      {
        "id": 2,
        "name": "string"
      }
    ],
    "status": "available"
  }
]

yaml:

sourceSystem: "ASD"
destinationSystem: "ASDF"
params:
  dateFormat: yyyy-MM-dd'T'HH:mm:ss'Z'
result:
  type: array
  path: ..../
  #  filter: $self.index%3==0
  xml:
    path: "/pets/Pet"
  fields:
    category:
      type: object
      fields:
        id:
          dataType: long
        name:
    #          tagId:
    #             type: object
    #             filter: $self.index==3
    #             path: ../tags
    ##             parentPath: ../tags
    #             xml:
    #               path: /pets/Pet/tags/tag
    #             fields:
    #               id: int
    #               name:
    #
    tagIds:
      dataType: long
      type: array
      path: tags/id
      xml:
        path: tags/tag/id

    id:
      dataType: long
    name:
    status:
    photoUrls:
      type: array
      path: photoUrls
      xml:
        path: photoUrls/photoUrls
    tags:
      type: array
      path: tags
      xml:
        path: tags/tag
      fields:
        id:
          dataType: long
        name:
         

Unit Test:

@Test
    public void parsePet() throws IOException {

        Path rootPath = Paths.get(TestUtils.getTestResourcePath(), pathPlace);

        DSMBuilder builder = new DSMBuilder(rootPath.resolve("pet.yml").toFile(), rootPath.toString());
        DSM dsm = builder.setType(DSMBuilder.TYPE.JSON).create(Pet.class);
        File file = rootPath.resolve("pet.json").toFile();

        Object object = dsm.toObject(file);

        List<Pet> list = (List<Pet>) object;
        String s = JSONUtil.toJsonStr(list);
        System.out.println(s);
    }

Result:

[
	{
		"photoUrls": [
			"string"
		],
		"tagIds": [
			3
		],
		"tags": [
			{
				"name": "string",
				"id": 3
			}
		],
		"name": "doggie",
		"id": 8531,
		"category": {
			"name": "string",
			"id": 0
		},
		"status": "available"
	},
	{
		"photoUrls": [
			"string"
		],
		"tagIds": [
			2
		],
		"tags": [
			{
				"name": "string",
				"id": 2
			}
		],
		"name": "doggie",
		"id": 9009,
		"category": {
			"name": "string",
			"id": 0
		},
		"status": "available"
	},
	{
		"photoUrls": [
			"string"
		],
		"tagIds": [
			2
		],
		"tags": [
			{
				"name": "string",
				"id": 2
			}
		],
		"name": "doggie",
		"id": 9009,
		"category": {
			"name": "string",
			"id": 0
		},
		"status": "available"
	}
]

There are two items with the same id "9009". It seems one more data exists.
I researched there is a new version 1.0.5 to fix similar bug. But when I debuged in the code at "case END_ARRAY", found it go to another condition at line 27 as below.
image

Is there any missing configuration for me?
@mfatihercik

java.lang.IllegalArgumentException: URI is not hierarchical at java.io.File.<init>(File.java:418)

Hi.....

I got the same error even i added new File("") in DSMBuilder

this is my sample code

DSMBuilder builder = new DSMBuilder(new File("dsm-config-file.yaml"));
DSM dsm = builder.setType(DSMBuilder.XML).create();
Pet pet = dsm.toObject(new File("path/to/xmlFile.xml"),Pet.class);

i try to solve my problem by using below question.....

java.lang.IllegalArgumentException: URI is not hierarchical..

but i codn't figure out the problem please help me..

if you want i will share my code ...

java.lang.IllegalArgumentException: URI is not hierarchical

Hi, I'm trying to use this library according to documentation but no matter what path I use i get the same error as in title. For example :

DSMBuilder builder = new DSMBuilder("/tmp/targetFile.yaml");
DSM dsm = builder.setType(DSMBuilder.XML).create();

Caused by: java.lang.IllegalArgumentException: URI is not hierarchical
at java.base/java.io.File.(File.java:418)
at com.github.mfatihercik.dsb.resource.FileSystemResourceAccessor.addRootPath(FileSystemResourceAccessor.java:54)
at com.github.mfatihercik.dsb.resource.AbstractResourceAccessor.init(AbstractResourceAccessor.java:39)
at com.github.mfatihercik.dsb.resource.FileSystemResourceAccessor.init(FileSystemResourceAccessor.java:47)
at com.github.mfatihercik.dsb.resource.FileSystemResourceAccessor.(FileSystemResourceAccessor.java:27)
at com.github.mfatihercik.dsb.configloader.AbstractConfigLoader.setResourceAccessor(AbstractConfigLoader.java:89)
at com.github.mfatihercik.dsb.configloader.AbstractConfigLoader.(AbstractConfigLoader.java:32)
at com.github.mfatihercik.dsb.configloader.YamlConfigLoaderStrategy.(YamlConfigLoaderStrategy.java:11)
at com.github.mfatihercik.dsb.DSMBuilder.create(DSMBuilder.java:99)
at com.github.mfatihercik.dsb.DSMBuilder.create(DSMBuilder.java:93)

I don't understand what the problem is, the path is correct and the file is present.

parsing element is not merged when path is null

First:

         METHOD:
         -  parentPath: header
            default: $all.result.data.TYPOLOGY
            transformationCode: METHOD

Second

         METHOD:
         -  parentPath: header
            default: $self.parent.data.TYPOLOGY

second extends to first

dynamic filter - clarification - Urgent

with this library, is it possible to have filter expression to match inputs?
for eg: self.data.salary>10000, is it possible to accept the value to be compared (instead of hard coded value 10000) from input? like from java.

I have a requirement as follows
I have an array of objects as follows
{
"operations" : [
{
"name" : "operationname1",
"version" : "version",
"operationtype" : "single",
"transformationclass" :"fullpackage class"
},
{
"name" : "operationname2",
"version" : "version",
"operationtype" : "single",
"transformationclass" :"fullpackage class"
}
]
}

a) I get operation name and version from input, so i need to load only operation type and transformationClass only for the provided inputs. Is that even possible with DSM library?

b) If thats not possible, I can try loading all the elements always and do normal java Collection Streams operation to filter - In this case what benefit will i get , as it is same as loading using ObjectMapper ( jackson Library).

c) is DSM supported for concurrent operations?

help to check XML node

great stuff!.

how can I map the following when the XML node define the value:

XML Source:
<GigabitEthernet> <name>1/0/5</name> <speed> <auto/> </speed> </GigabitEthernet>

or

<GigabitEthernet> <name>1/0/6</name> <speed> <1g/> </speed> </GigabitEthernet>

Expected result:
{ "name":"1/0/5", "speed":"AUTO" }
or
{ "name":"1/0/6", "speed":"1000" }

Handling arrays of arrays

Trying to write config to handle array of arrays.
Input to be parsed looks like:

[
  {
    "columns": [
      "name1",
      "name2",
      "name3"
    ],
    "events": [
      [
        "value1_1",
        "value1_2",
        "value1_3"
      ],
      [
        "value2_1",
        "value2_2",
        "value2_3"
      ],
      [
        "value3_1",
        "value3_2",
        "value3_3"
      ]
    ]
  }
]

Config (in JSON):

{
  "version": 1.0,
  "result":{
    "type":"object",
    "path":"/",
    "function": "processStuff",
    "fields":{
      "columns": {
        "type": "array"
      },
      "events": {
        "type": "array"
      }
    }
  }
}

The result looks like:
{columns=[name1, name2, name3], events=[value1_1, value1_2, value1_3, null, value2_1, value2_2, value2_3, null, value3_1, value3_2, value3_3, null]}

Looking to get a result like:
columns=[name1, name2, name3], events [[value1_1, value1_2, value1_3], [value2_1, value2_2, value2_3], [value3_1, value3_2, value3_3]]}

Tried a few variations of config but wasn't able to get the result I was looking for. Can DSM handle this kind of input?

dynamic fieldName

Currently, the field name is static. it is not changed. Defining field name by expression will be very efficient

new count type

new type to get the number of tags exists in XML or JSON.

The conversion seems a little problematic when the root node is an array

version:1.0.4

source json:
[
{
"address": {
"se": {
"db": "f"
},
"addressName": "2dsf3",
"addressPwd": "323"
}
}
]

mapping json:
{
"result" : {
"path" : "/",
"type" : "array",
"fields" : {
"address": {
"type": "object",
"path": "/address",
"fields": {
"addressName" :{
"path": "/address/addressName",
"dataType": "string"
},
"addressPwd": {
"path": "/address/addressPwd",
"dataType": "string"
}
}
}
}
}
}
result:
[{address={addressName=2dsf3, addressPwd=323}}, {address={addressName=2dsf3, addressPwd=323}}]

How to parse XML if format in below manner?

@mfatihercik

Consider I have my xml file in below format (its a partial file)

<package name="com/example/configure">
		<class name="com/example/configure/AutoConfigure"
			sourcefilename="AutoConfigure.java">
			<method name="tryItOut;" desc="()V" line="8">
				<counter type="INSTRUCTION" missed="3" covered="0" />
		        </method>
			<method name="sampleController"
				desc="()Lcom/example/controller/SampleController;" line="13">
				<counter type="INSTRUCTION" missed="4" covered="23" />
			</method>
                 </class>
</package>
<package name = "something>
                <class name = 
                       ...

I have to loop through multiple packages which can have multiple class names and which in-turn can have multiple methods inside the class

using DSM how should I capture the package name , class name and method name only if covered !=0
and filter the XML document and convert it to JSON output format

PS:I am new to java and the examples I saw on your website have data inserted between <class>"ClassName"</class> but not as <class name = "ClassName">...</class>

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.