Writing a query instead of using the table in an SQL source throws the following error and does not update the schema to show the columns from the query

"Detect Data Type failed

"

I have been to useingADF dataflow to transform file from one datalake to another datalake. But when I debug the pipeline, I get error as below:

{
"error": {
"code": 400,
"message": "java.lang.Exception: DataFlowManagerClient.getPayload: 400, resolvedContent: Some(HttpResponseProxy{HTTP/1.1 400 Bad Request [Cache-Control: no-cache, Pragma: no-cache, Content-Type: application/json; charset=utf-8, Expires: -1, X-Frame-Options: SAMEORIGIN, Strict-Transport-Security: max-age=15724800; includeSubDomains, X-Content-Type-Options: nosniff, Date: Tue, 20 Aug 2019 15:30:19 GMT, Content-Length: 2137] ResponseEntityProxy{[Content-Type: application/json; charset=utf-8,Content-Length: 2137,Chunked: false]}})"
},

And also I get error if i do data preview while creating dataflow pipelines as below:
"Input is malformed. Reason: Could not get entity details for Curated" (Curated is the name of datalake).
I have tested all the connection and everything is fine but still the pipeline fails.

Can you please have a look into it

Hi kromerm,
I am facing following error when performing a data transformation action using ADF, please if you could just give a hint on what needs to be corrected.

"code":"BadRequest","message":"ErrorCode=InvalidTemplate, ErrorMessage=Unable to parse expression '_c1'","target":"pipeline/pipeline1/runid/########","details":null,"error":null}

Thanks in Advance
Lav

Hello,

Am facing the following issue while executing the ADF V2 dataflow preview with Inner Join transformation "Sink 'sink1': Detected implicit cartesian product for INNER join between logical plans\nTypedFilter".Can you please throw some lights on this so that i can understand clearly and proceed.

In the data preview it's showing correct result but while i execute every time its giving the same issue even though join key names are having aliases.

I have a somewhat complex expression and it causes extreme slowness in cases like these:

• During editing of the expression, the user interface sometimes locks up for over a minute
• When exiting the expression editor
• Even when starting up the Azure Data Factory webapp (nothing opened yet). There is a phase when it is loading the resources in all the data flows that takes very long. This means time from clicking an url to open Azure Data Factory to it being ready to work with can take minutes just to load the resource (see image below).

This is the expression:

iif (instr(code_bekostigingstatus, 'pi') > 0 ,'', 
iif (instr(code_bekostigingstatus, 'jl') > 0, 'Geen 1e inschrijving', 
iif (instr(code_bekostigingstatus, 'nr') > 0, 'Nationaliteiteneis',
iif (instr(code_bekostigingstatus, 'nb') > 0, 'Eerdere graad behaald',
iif (instr(code_bekostigingstatus, 'nd') > 0,'Eerdere graad behaald',
iif (instr(code_bekostigingstatus, 'nh') > 0, 'Bekostigingsjaren verbruikt',
iif (instr(code_bekostigingstatus, 'ni') > 0, 'Bekostigingsjaren verbruikt',
iif (instr(code_bekostigingstatus, 'nk') > 0, 'Bekostigingsjaren verbruikt',
iif (instr(code_bekostigingstatus, 'ti') > 0, 'Geen tijdige aanlevering',
'Anders')))))))))

Fyi the logic behind this is that the field code_bekostigingsstatus can contain multiple codes seperated by a comma. The organisation only wants to know the meaning of most highest priority code that is contained in the field, so I scan the field from highest to lowest priority code.

Could we have an option for the sink dataset when using Azure SQL DB to enable bulk load? Currently, loading a table is done RBAR, and is time consuming and resource intensive for large data sets.

As per title, rather than just creating nvarchar(max) columns.

When reading all files from a folder, the default behaviour for ADF is to leave the file name blank. This causes an issue in a data flow, as you need to specify a * wildcard else it will not be able to read the files when debugging or running from a pipeline.

Hi! Perhaps this is not the best place to ask questions, but I am making a try. I am a beginner using ADF and azure Data Flows especially. I am very curious of your examples, and I really want to import all your examples files (json) to my newly created data factory. It must be an easier way than creating all activities, connections, datasets and others manually. The templates are of course good but I want to test out your example code in my azure portal and in my data factory.

And just a second question: I am a SSIS man used control flows with master packages executing other packages. When I am now building a data warehouse in ADF with many dimension tables and fact tables, is it best practice to have separate data flows or should I build general data flows that either have many parallel upserts to different dimensions? I think I need some guiding here

Thank you

Regards Geir