shashisadasivan / edmxtrimmer Goto Github PK

View Code? Open in Web Editor NEW

20.0 5.0 11.0 229 KB

D365 Finance and operations EDMX trimmer

License: MIT License

C# 100.00%

d365fo dynamics-365

edmxtrimmer's Introduction

EDMXTrimmer

EDMX trimmer for OData specifications.

Some OData specifications (i.e. the metadata or .edmx file) can be a quite large. One example is the specification for the Dynamics 365 Finance & Operations (D365FO) ERP system's OData API.

Tools that operate on OData specifications in turn create huge files. The can also show bad performance or just crash. On example for such a tool is the Visual Studio extension OData Connected Service.

EDMXTrimmer is used to remove entities and related objects (entity sets, navigation properties, actions and enums) from the OData specification that are not needed.

For example the D365FO OData specification in version 10.0.29 is a 30 MB .edmx file. If only one entity such as CustomersV3 is required, EDMXTrimmer can reduce the file size to less than 200 KB.

Build

EDMXTrimmer is a .Net 6 console application. It can be built with Visual Studio 2022 or with the .Net 6 SDK.

Command line

EDMXTrimmer can be run from the command line.

dotnet EDMXTrimmer.dll --edmxfile=<your file name here> --entitiestokeep=<entitylist separated by commas>

On Windows, you can also run the executable EDMXTrimmer.exe instead of dotnet EDMXTrimmer.dll.

The following command line arguments are supported:

edmxfile : This is the OData metadata file that you can download off https://<url>/data/$metadata. This parameter is required. Save the output to a file and use it in the command line argument.
entitiestokeep : Enter the entity set names (plural) separated by commas. E.g. CustomersV3,VendorsV2. All other entities and their related objects will be removed.
enttitiestoexclude : Enter the entity set names (plural) separated by commas. E.g. CustomersV3,VendorsV2. These entites and their related objects will be removed. All other entities will be kept.
outputfilename : The name of the output file. If not specified, the output will be written to file Output.edmx in the current directory.
entitiesareregularexpressions : If this parameter is specified, the entity names are treated as regular expressions. This can be used to keep or remove all entities where their names follow a pattern. E.g. --entitiesToKeep="^(Cust|Vend).+" --entitiesareregularexpressions will keep all entities that start with Cust or Vend.
removeprimaryannotations: Removes annotations nodes directly under the first schema node.
removeactionimports: Removes action import nodes.

Examples

1) Keep a single entity

dotnet EDMXTrimmer.dll --edmxfile="C:\temp\custODataMetadata.edmx" --entitiestokeep=CustomersV3

This will keep only the CustomersV3 entity and its related objects. All other entities and their related objects will be removed.

2) Specify output file

dotnet EDMXTrimmer.dll --edmxfile="C:\temp\custODataMetadata.edmx" --entitiestokeep=CustomersV3 --outputfilename="C:\temp\custODataMetadataTrimmed.edmx"

This will keep only the CustomersV3 entity and its related objects. All other entities and their related objects will be removed. The output will be written to the file C:\temp\custODataMetadataTrimmed.edmx.

3) Keep multiple entities

dotnet EDMXTrimmer.dll --edmxfile="C:\temp\custODataMetadata.edmx" --entitiestokeep=CustomersV3,VendorsV2

This will keep the CustomersV3 and VendorsV2 entities and their related objects. All other entities and their related objects will be removed.

4) Keep entities using wild cards

dotnet EDMXTrimmer.dll --edmxfile="C:\temp\custODataMetadata.edmx" --entitiestokeep=Cust*

This will keep all entities that start with Cust. All other entities and their related objects will be removed.

5) Exclude entities

dotnet EDMXTrimmer.dll --edmxfile="C:\temp\custODataMetadata.edmx" --entitiestoexclude=Cust*,VendorsV2

This will remove all entities that start with Cust and the VendorsV2 entity.

6) Exclude entities from included entities

dotnet EDMXTrimmer.dll --edmxfile="C:\temp\custODataMetadata.edmx" --entitiestokeep=Cust* --entitiestoexclude=CustomersV2

This will first remove all entities that do not start with Cust. From the remaining entities, CustomersV2 will be removed.

Note that --entitiestoexclude is applied after --entitiestokeep. This means that if you specify both parameters, the entities that are excluded will be removed from the entities that are kept.

7) Use regular expressions

dotnet EDMXTrimmer.dll --edmxfile "C:\temp\custODataMetadata.edmx" --entitiesToKeep="^(Cust|Vend).+" --entitiesareregularexpressions

This will keep all entities that start with Cust or Vend. All other entities will be removed.

Note that this could also be achieved with wild cards: --entitiestokeep=Cust*,Vend*. The regular expression is only needed if you want to use more complex patterns.

8) Remove annotations and action imports

dotnet EDMXTrimmer.dll --edmxfile "C:\temp\custODataMetadata.edmx" --removeprimaryannotations --removeactionimports

This will remove all primary annotations and action import nodes from the edmx file.

edmxtrimmer's People

Contributors

Stargazers

Watchers

Forkers

siva497 jasonhyland ievgenm ppotturi mdrewanz papyr fh-inway dontte ablizorukov fartwhif andrewk2

edmxtrimmer's Issues

Add build action for pull requests

To have some basic automated validation of pull requests, a build action should be added.

This could look similar to https://github.com/FH-Inway/EDMXTrimmer/blob/fh-main/.github/workflows/build-release.yml
See https://github.com/FH-Inway/EDMXTrimmer/actions/runs/4096944914 for an example run of that action.

NavigationProperty

Hello,
Nice work with this trimmer. It is very useful. I noticed after running it and building a proxy class from the new EDMX file that SalesOrderHeader does not have a SalesOrderLine collection in it because the NavigationProperty is being removed from EntityType. What is the reason you excluded NavigationProperty from the EntityType? Just to keep the size down? I would like to add it because if it is in, then you could reference in some C# code like:

foreach (var salesLine in salesOrder.SalesOrderLine)
{
Console.WriteLine(string.Format("Sales order line with description {0} contains item id {1}", salesLine.LineDescription, salesLine.ItemNumber));
}

Keep bound actions for entity

I have a bound action on an entity, which I call like this:

_resources.MyEntity.FirstOrDefault().MyAction()

The action is generated like this in the EDMX

<Action Name="MyAction" IsBound="true">
  <Parameter Name="MyEntity" Type="Microsoft.Dynamics.DataEntities.MyEntity" />
</Action>

Is it possible to keep the entity in the EDMX along with all its bound actions? Currently if I run this, the entity is kept, but the actions are removed

dotnet EDMXTrimmer.dll --edmxfile="MyEDM.edmx" --entitiestokeep=MyEntity

Thanks

Allow for standard DataManagement Entity

Standard Data managemnet entity has the definitions for the Batch API. These should be an option (or always include this in the file) There are Actions associated with this. Currently the trimmer removes all actions.

Can't build the project

Hi,

I've noticed that after the last merge to master the project can't be builded anymore.
I think that some lines were lost during merge, the code below is just to fix the building issue but there is might be something else

Index: EDMXTrimmer/EDMXTrimmer/EdmxTrimmer.cs
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/EDMXTrimmer/EDMXTrimmer/EdmxTrimmer.cs b/EDMXTrimmer/EDMXTrimmer/EdmxTrimmer.cs
--- a/EDMXTrimmer/EDMXTrimmer/EdmxTrimmer.cs	(revision fdb45c8719810ff1f499021d9122a8724149d76a)
+++ b/EDMXTrimmer/EDMXTrimmer/EdmxTrimmer.cs	(date 1675779949872)
@@ -166,6 +166,7 @@
             this._xmlDocument.GetElementsByTagName(ACTION).Cast<XmlNode>()
                 .Where(action => !entityTypesFound.Any(entityType => action.ChildNodes.Cast<XmlNode>().
                     Any(childNode => EntityExists(childNode, entityType)))).ToList()
+                .ForEach(n => n.ParentNode.RemoveChild(n));
 
             // Determine enums to keep
             List<String> enumTypesFound = new List<string>();
Index: EDMXTrimmer/EDMXTrimmer/Program.cs
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/EDMXTrimmer/EDMXTrimmer/Program.cs b/EDMXTrimmer/EDMXTrimmer/Program.cs
--- a/EDMXTrimmer/EDMXTrimmer/Program.cs	(revision fdb45c8719810ff1f499021d9122a8724149d76a)
+++ b/EDMXTrimmer/EDMXTrimmer/Program.cs	(date 1675780140874)
@@ -1,3 +1,4 @@
+using System;
 using CommandLine;
 using System.Collections.Generic;
 using System.Linq;

Keep relations

@AndreasHassing mentioned in OData/ODataConnectedService#327 (comment) that relations are not kept in the trimmed file.

To avoid an off topic discussion in that issue, this issue was created to continue the discussion here.

The discussion so far:

@FH-Inway OData/ODataConnectedService#327 (comment)

For trimming down huge ERP OData endpoints (and specifically Dynamics 365 FO), take a look at https://github.com/shashisadasivan/EDMXTrimmer

@AndreasHassing OData/ODataConnectedService#327 (comment)

That's really cool, except that it doesn't keep relations which are very useful artifacts for certain complicated applications using the OData schema for code generation.

Edit: their code could be altered to include relations, while that increases the complexity quite a bit.

@FH-Inway OData/ODataConnectedService#327 (comment)

@AndreasHassing Do you mean the NavigationProperty tags under the EntityType?

@AndreasHassing OData/ODataConnectedService#327 (comment)

That's right.

:sparkles: Exclude unused enums

The tool currently does a good job of removing most of the stuff that is not needed after removing the entities that were not specified to be kept. However, the enums are not trimmed, which causes unnecessary content in the output file.

I propose to also remove the enums that are not required by the remaining entities or their actions.

Option to exclude entities

Very useful tool, thanks.

Instead of using the EntitiesToKeep parameter to specify the entities that should remain in the .edmx file, it would be useful to have a parameter EntitiesToExclude. This would allow to specify a list of entities that should be removed from the .edmx file (e.g. I want to remove all change management entities, but keep the rest).
It would also be nice if both parameters could support wild cards so that e.g. all entities starting with "Vend*" could be kept or excluded.

:sparkles: add support for dataverse edmx files

While looking at #20 (which basically adds support for Business Central edmx files), I also played around with a Dataverse edmx file. While the trimmer does its thing with it, the result does not seem usable by downstream tools.

E.g. with the ODataConnectedService, no C# client can be generated. The OOASUtil (from OpenAPI.NET.OData) cannot generate an OpenAPI specification from it.

After playing around with a trimmed Dataverse edmx file, I found that the Dataverse edmx file seems to make use of more advanced features of the OData specification. For example, it includes references to OData vocabulary extensions, defines base types for entities, functions and function imports (which seem to be similar to action imports).

To support Dataverse edmx files, the trimmer would need additional functionality that covers the points listed above and probably more.

namespace alias: 583155c
remove base types for entities/entities not in entity sets: e98e36c
remove function imports: d52f044
remove functions
functions to include
remove complex types: e3e2b69
actions to include: 4a73892
remove vocabulary extensions

Omits EntityType

Thanks for the effort, however we are interested in more than what it captures

EntitySets are extracted, but not EntityTypes. Only interested in EntityTypes

EXEC:

C:\Users\user\Desktop\EDMXTrimmer\EDMXTrimmer\EDMXTrimmer\bin\Debug\netcoreapp3.1>dotnet edmxtrimmer.dll --edmxfile "C:\Users\user\Downloads\metadata.xml" --outputfilename "C:\Users\user\Desktop\metadata_pruned.xml" --entitiestokeep "(Position)" --entitiesareregularexpressions --verbose
EDMX Saved to file: C:\Users\user\Desktop\metadata_pruned.xml
EDMX Saved to file: C:\Users\user\Desktop\metadata_pruned.xml

IN:

Search "position" (17 hits in 1 file of 1 searched)
  C:\Users\user\Downloads\metadata.xml (17 hits)
	Line 447: 				<NavigationProperty Name="ABCPosition"
	Line 448:                   					Type="Collection(NAV.ABCPosition)"
	Line 13236: 				<Property Name="Position_Code"
	Line 13240:           						String="Position Code" />
	Line 13245: 				<Property Name="Position_Department"
	Line 13249:           						String="Position Department" />
	Line 13254: 				<Property Name="Position_Category"
	Line 13258:           						String="Position Category" />
	Line 14234: 				<Property Name="Position_Num"
	Line 14238:           						String="Position Num" />
	Line 14437: 			<EntityType Name="ABCPosition">
	Line 14439: 					<PropertyRef Name="Position_Num" />
	Line 14441: 				<Property Name="Position_Num"
	Line 14446:           						String="Position Num" />
	Line 14656:           					String="ABCPosition" />
	Line 45074: 				<EntitySet Name="ABCPosition"
	Line 45075:          					EntityType="NAV.ABCPosition">

OUT:

Search "position" (2 hits in 1 file of 1 searched)
  C:\Users\user\Desktop\metadata_pruned.xml (2 hits)
	Line 160:         <EntitySet Name="ABCPosition" EntityType="NAV.ABCPosition">
	Line 160:         <EntitySet Name="ABCPosition" EntityType="NAV.ABCPosition">

:bug: NullReferenceException when using both --entitiestokeep and --entitiestoexlude

When using both parameters --entitiestokeep and --entitiestoexlude it can happen that a NullReferenceException is thrown.

This happens when --entitiestokeep does not retain the entities that are then specified to be excluded by --entitiestoexclude. Since the entities were already removed by --entitiestokeep, the logic for --entitiestoexclude then tries to remove entities that were already removed.

Here is a sample call for a D365FO .edmx file that causes the exception:

.\EDMXTrimmer.exe --edmxfile=C:\Temp\d365fo-odata.edmx --entitiestokeep=DataManagementDefinitionGroups --entitiestoexclude=RetailReturnPolicies --verbose
EDMX Saved to file: Output.edmx
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at EDMXTrimmer.EdmxTrimmer.<>c.<RemoveEntitySets>b__40_0(XmlNode n) in C:\Repositories\EDMXTrimmer\EDMXTrimmer\EDMXTrimmer\EdmxTrimmer.cs:line 134
   at System.Collections.Generic.List`1.ForEach(Action`1 action)
   at EDMXTrimmer.EdmxTrimmer.RemoveEntitySets(List`1 entitySets, List`1 entitiesKeep) in C:\Repositories\EDMXTrimmer\EDMXTrimmer\EDMXTrimmer\EdmxTrimmer.cs:line 134
   at EDMXTrimmer.EdmxTrimmer.RemoveExcludedEntities(List`1 entitiesToExclude, List`1 entitySets, List`1 entityTypes) in C:\Repositories\EDMXTrimmer\EDMXTrimmer\EDMXTrimmer\EdmxTrimmer.cs:line 107
   at EDMXTrimmer.EdmxTrimmer.AnalyzeFile() in C:\Repositories\EDMXTrimmer\EDMXTrimmer\EDMXTrimmer\EdmxTrimmer.cs:line 76
   at EDMXTrimmer.Program.Main(String[] args) in C:\Repositories\EDMXTrimmer\EDMXTrimmer\EDMXTrimmer\Program.cs:line 63

Automatically find the EntitySet or EntityType

Currently we need to mention both the value of the EntitySet and EntityType AS they are the Singular and Plural public names of the Entity. Need to find a way to reduce this down so that we only have to provide one of the values

Expand does not work after trimming

Given this code:
eg: var result = context.SalesOrderHeadersV2.Expand(x=>x.SalesOrderLines).Take(1).ToList();
If I trim my EDMX this will not work.
If I do not trim my EDMX this does work.

Both the SalesOrderHeadersV2, and SalesOrderLines are listed in my call to edmxTrimmer list of entities to keep.

I created a Stack overflow issue here: https://stackoverflow.com/questions/58263897/microsoft-odata-client-expand-does-not-populate-the-model