Guid generation question about pyapacheatlas HOT 4 CLOSED

Hi, @EdoQuasso - The guid you pass in should simply be a negative number. Azure Purview will automatically create a new guid if you're creating a brand new entity (it knows if its brand new based on Qualified Name) or it will automatically find the existing entity and update it (based on qualified name).

So your AtlasProcess will look like...

process = AtlasProcess(
        name="sample process",
        typeName="Process",
        qualified_name="pyapacheatlas://democustomprocess",
        inputs=[input01],
        outputs=[output01],
        guid=-102
    )

See the create entity and lineage sample for more examples.

from pyapacheatlas.

Hi @wjohnson, thank you very much for the quick reply.

May I ask you another question? Is it possible to attach an Atlas process to an Azure Data Factory Copy Activity?
Because when I try, it raises this error:
AtlasException: {"requestId":"96b9b506-7499-4455-859d-64b133c62aac","errorCode":"ATLAS-400-00-036","errorMessage":"invalid relationshipDef: dataset_process_inputs: end type 1: DataSet, end type 2: Process"}

The code is the following:

output01 = AtlasEntity(
    name="output01",
    typeName="azure_datalake_gen2_path",
    qualified_name="apacheatlas://fake_output",
    guid=-110
)

newLineage = AtlasProcess(
  name="Databricks Processing",
  typeName = "custom_databricks_notebook_process",
  qualified_name = process_qn,
  inputs = [entity1["entities"][0]],
  outputs = [output01],
  guid = gt.get_guid()
)

results = client.upload_entities(
  batch = [output01, newLineage]
)

The entity1 is an Azure Data Factory Copy Activity as you can see:
{'referredEntities': {'a56acc20-481e-43c7-b62f-192a1daffe8b': {'typeName': 'adf_copy_operation', 'attributes': {'owner': None, 'outputs': [{'guid': 'c44e46d4-eef5-4139-8eb8-1deb6cfec0a7', 'typeName': 'azure_datalake_gen2_path', 'uniqueAttributes': {'qualifiedName': 'https://pvdemo.dfs.core.windows.net/demo/raw/Customer.csv'}}], 'replicatedTo': None, 'replicatedFrom': None, 'qualifiedName': '/subscriptions/b8e41e61-5fb3-4590-aefb-714df2bdb6ec/resourceGroups/Purview_demo/providers/Microsoft.DataFactory/factories/pv-demo/pipelines/P_Customer/activities/Source to Raw#https://pvdemo.dfs.core.windows.net/demo/raw/Customer.csv#azure_datalake_gen2_path', 'inputs': [{'guid': '23459b6a-8d72-4b6e-9a31-3ff6f6f60000', 'typeName': 'azure_sql_table', 'uniqueAttributes': {'qualifiedName': 'mssql://pv-demo.database.windows.net/BI/src/Customer'}}], 'name': 'Source to Raw', 'errorMessage': None, 'description': None, 'status': None, 'columnMapping': '[{"DatasetMapping":{"Source":"*","Sink":"https://pvdemo.dfs.core.windows.net/demo/raw/Customer.csv"},"ColumnMapping":[{"Source":"CustomerID","Sink":"Prop_0"},{"Source":"NameStyle","Sink":"Prop_1"},{"Source":"Title","Sink":"Prop_2"},{"Source":"FirstName","Sink":"Prop_3"},{"Source":"MiddleName","Sink":"Prop_4"},{"Source":"LastName","Sink":"Prop_5"},{"Source":"Suffix","Sink":"Prop_6"},{"Source":"CompanyName","Sink":"Prop_7"},{"Source":"SalesPerson","Sink":"Prop_8"},{"Source":"EmailAddress","Sink":"Prop_9"},{"Source":"Phone","Sink":"Prop_10"},{"Source":"PasswordHash","Sink":"Prop_11"},{"Source":"PasswordSalt","Sink":"Prop_12"},{"Source":"rowguid","Sink":"Prop_13"},{"Source":"ModifiedDate","Sink":"Prop_14"}]}]'}, 'lastModifiedTS': '1', 'guid': 'a56acc20-481e-43c7-b62f-192a1daffe8b', 'status': 'ACTIVE', 'createdBy': 'ServiceAdmin', 'updatedBy': 'ServiceAdmin', 'source': 'DataFactory', 'createTime': 1614614786093, 'updateTime': 1614614786093, 'version': 0, 'relationshipAttributes': {'outputs': [{'guid': 'c44e46d4-eef5-4139-8eb8-1deb6cfec0a7', 'typeName': 'azure_datalake_gen2_path', 'entityStatus': 'ACTIVE', 'displayText': 'Customer.csv', 'relationshipType': 'process_dataset_outputs', 'relationshipGuid': 'b752a816-58a0-4a5b-9cae-c1fb6f812398', 'relationshipStatus': 'ACTIVE', 'relationshipAttributes': {'typeName': 'process_dataset_outputs'}}], 'parent': {'guid': 'c890633e-fd69-42e2-b5c5-64753d2f976a', 'typeName': 'adf_copy_activity', 'entityStatus': 'ACTIVE', 'displayText': 'Source to Raw', 'relationshipType': 'process_parent', 'relationshipGuid': '8d79ac6a-642c-4758-94ea-a15919457b52', 'relationshipStatus': 'ACTIVE', 'relationshipAttributes': {'typeName': 'process_parent'}}, 'inputs': [{'guid': '23459b6a-8d72-4b6e-9a31-3ff6f6f60000', 'typeName': 'azure_sql_table', 'entityStatus': 'ACTIVE', 'displayText': 'Customer', 'relationshipType': 'dataset_process_inputs', 'relationshipGuid': '36442682-9fc7-4b07-b8f0-72e08bd40a68', 'relationshipStatus': 'ACTIVE', 'relationshipAttributes': {'typeName': 'dataset_process_inputs'}}], 'subProcesses': [], 'meanings': []}}}, 'entities': [{'typeName': 'adf_copy_activity', 'attributes': {'owner': None, 'outputs': [], 'replicatedTo': None, 'replicatedFrom': None, 'qualifiedName': '/subscriptions/b8e41e61-5fb3-4590-aefb-714df2bdb6ec/resourceGroups/Purview_demo/providers/Microsoft.DataFactory/factories/pv-demo/pipelines/P_Customer/activities/Source to Raw', 'inputs': [], 'errorMessage': '', 'description': None, 'dataSize': 217829, 'lastRunTime': 1614614763078, 'name': 'Source to Raw', 'rowCount': 847, 'status': 'Completed', 'columnMapping': None}, 'lastModifiedTS': '2', 'guid': 'c890633e-fd69-42e2-b5c5-64753d2f976a', 'status': 'ACTIVE', 'createdBy': 'ServiceAdmin', 'updatedBy': 'ServiceAdmin', 'source': 'DataFactory', 'createTime': 1614614784352, 'updateTime': 1614614787018, 'version': 0, 'relationshipAttributes': {'outputs': [], 'parent': {'guid': '1d606f7b-ed4d-4387-8d34-b4fc506eded5', 'typeName': 'adf_pipeline', 'entityStatus': 'ACTIVE', 'displayText': 'P_Customer', 'relationshipType': 'process_parent', 'relationshipGuid': 'a3bc8e61-099a-45e2-9c76-a6a31aba9b3a', 'relationshipStatus': 'ACTIVE', 'relationshipAttributes': {'typeName': 'process_parent'}}, 'inputs': [], 'subProcesses': [{'guid': 'a56acc20-481e-43c7-b62f-192a1daffe8b', 'typeName': 'adf_copy_operation', 'entityStatus': 'ACTIVE', 'displayText': 'Source to Raw', 'relationshipType': 'process_parent', 'relationshipGuid': '8d79ac6a-642c-4758-94ea-a15919457b52', 'relationshipStatus': 'ACTIVE', 'relationshipAttributes': {'typeName': 'process_parent'}}], 'runInstances': [], 'meanings': []}}]}

Thank you very much in advance.

Edoardo

from pyapacheatlas.

@EdoQuasso - The adf_copy_operation type is a Process type. A Process entity cannot be an input or output to another Process entity. You have to have an intermediate dataset. Presumably, the copy activity creates a dataset, right? I'd take the output of the copy activity and then THAT entity's guid would be the input to your new lineage process with Databricks.

from pyapacheatlas.

Thank you very much again @wjohnson for the response. Yes, that's what I thought and how I managed the lineage, I asked you because I hoped there was another way.

Thank you very much for your time.

Edoardo

from pyapacheatlas.