Comments (5)
great, It's not that im reluctant but was wondering if it was the right way to do it.
I will go with the MARS attribute.
thank you for your answer and for this pretty neat framework
from etl.net.
Can you give me a sample of the code you have for me to actually see what is the problem?
From what I understand, when you have several upsert or read in your process, it will be very likely they will occurs in a concurrent manner. Therefore, you connection should be setup with the MARS attribute set to true as it is the SQL driver that needs this setup to permit concurrent operations on the same connection. This attribute can only be setup at the connection string level.
I will update the documentation to make this clear.
Let me know if I got your problem correctly.
from etl.net.
basically, i created 2 dll that i scaffold the database from EF.
objects that start with Osusr1gh are the source and the others the destination.
i read from the source with EFCoreSelect and write to destination with SqlServerSave
i use autoMapper for the transformation.
The code sample will merge 2 tables (acheteur and titreContact)
if i only run part 1 alone it works fine, if i only run part 2 alone it's fine too.
but both at the same time, it throws exceptions.
if a setup the MARS attribute, i can run the this method with both parts without exception.
I want to have an atomic transaction for all tables that i merge.. either it's all sucess or it's all failed (no partial merge)
also i looked at this thread to achieve the same thing with a transaction.
#292
at the end of the thread, you said
This workaround is to make 2 ado.net transactions that you set to their respective context and that you manage manually. Am I right?
I'm not really sure how to implement this.. why 2 transactions ?
i tried this option with the following modification
but for this to work, i have to change the original source code in the processItem method (wich i would like to avoid if possible)
this seems to works fine, but i'm not sure if it's the best way to go with the transaction options.
So basically, it looks that i have 2 choice now to do what i'm trying to do.
Option 1
- have all ETL in one DefineProcess method and enable MARS attribute in the connectionString
Option 2
- keep MARS attribute off and implement a custom transaction mecanism and have one operation per DefineProcess method.
- if all ProcessRunner execute sucessfully, commit the transaction.
What do you think is the best way to do ETL between 2 database with ETL.net ?
thank you,
André
from etl.net.
You definitely must use the option 1. I actually don't understand why you are reluctant to enable MARS attribute in your connection string.
But... if you want to want to have a 2 phase commit mechanism (updating, in an atomic way, two databases in one transaction), you must use the TransactionScope; that's the only way to do with dotnet, but it will work only on windows platforms.
from etl.net.
great, It's not that im reluctant but was wondering if it was the right way to do it.
I will go with the MARS attribute.thank you for your answer and for this pretty neat framework
It is so much the right way to do that I will have to add this detail as an important information in the documentation because it actually is a must have for the whole thing to work properly.
from etl.net.
Related Issues (20)
- Question: calling stored procedure which returns values HOT 5
- Getting two database streams from the same database connection in parallel HOT 5
- Getting in-memory list from the database stream HOT 13
- EF Core Spatial Integration HOT 4
- StreamProcessRunner is holding file in memory when I need it not to do that HOT 1
- Reusing DB context issue with large collections HOT 4
- Problem when generating SQL for .SqlServerSave when using ODBC and OleDB connections HOT 1
- SFTP connections not working without trailing slashes in folder paths HOT 1
- Running a stored procedure only once after all records in a CSV file have been saved to the database HOT 2
- Buffer operator
- How to compare two streams?
- .SqlServerSave() support on MS SQL Server versions prior to 2005 HOT 2
- MySQL PostgreSQL SQLite support HOT 1
- Specify timeout for .CrossApplySqlServerQuery and .ToSqlCommand HOT 4
- How can i move data between two SQL server db's HOT 2
- Raising an error and short-circuiting the execution of remaining operators HOT 1
- How to select from raw sql using EF Core? HOT 1
- .ToTextFileValue fails when output is bigger than 2GB HOT 2
- SqlServerSave fails because column names are not enclosed in square brackets HOT 1
- Bug--When I save an entity with a primary key of guid to the database using the EfCoreSave method, an error occurs, prompting me to "Cannot insert value NULL into column 'Id', table 'SimpleTutorial. dbo. Author'; columns do not allow null values. UPDATE fails." When I change the insertion mode to SaveMode.EntityFrameworkCore Successfully save HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from etl.net.