Discovering diagram of dependencies in Synapse Analytics and ADF pipelines
Documenting objects dependencies of ETL processes is a tough task. Regardless it is SSIS, ADF, pipelines in Azure Synapse or other systems. The reasons for understanding the current solution can vary either: handover to other team/member of the team, troubleshooting, refactoring, debugging, investigating dependencies due to error, performance issue or others, as well as keen to remove selected/duplicated pipelines or logic.
But there is never a good time to make documentation, and even if that has been done – no one knows how much it’s up to date. The situation is not improved by the fact that quite often there is a lack of (free or built-in) tools for generating such documentation. Sounds familiar? I bet it does.
Wiki Markdown and Mermaid diagrams
If you ever work with Markdown, you know that it is more and more popular format to develop and share documentation, often as a part of the code repository. Most popular tools support the format, including Azure DevOps and GitHub. Markdown supports diagrams in a few different forms, hence one can enhance documentation with a handful sequence diagram, Gantt charts or flowcharts, simply by writing a few lines of plain text.
Generate diagram with PowerShell
It has been some time now since I released the first version of #adftools. The main reason behind the tool is to deploy Azure Data Factory directly from code. But as the name suggests – it is a toolkit. With the tool, you can very easily generate mermaid code and copy/paste it to your markdown file. What do we need to do?
- Install/import PowerShell module (if you haven’t done that yet)
- azure.datafactory.tools – for Azure Data Factory
- azure.synapse.tools – for Azure Synapse pipelines
- Download or clone the code onto a local machine
- Run one line of code in PowerShell
- Copy the result and paste it into the target .md (markdown) file
Let’s do all these steps together.
Install/import PowerShell module
Depending on which code of which service you want to discover – install and import one of the following PS modules:
$module = 'azure.synapse.tools' Install-Module $module -Scope CurrentUser Import-Module $module
Download or clone the code
If you don’t have Git Integration set up for the pipelines – do it. I was explaining how to do that in this post, and once you achieve that – you can clone the code into your local machine. The folder should contain the structure of subfolders similar to this one:
Run one line of code in PowerShell
It’s time to run “the magic”. Execute the following code (either in Visual Studio Code or PowerShell ISE / session):
$RootFolder = "x:\!WORK\GitAz\sqlplayer\DataServices\Synapse" $synapse = Import-SynapseFromFolder -RootFolder $RootFolder -SynapseWorkspaceName 'whatever' Get-SynapseDocDiagram -synapse $synapse
The first line is to set RootFolder variable with location of code. The second line (import) is going to read all files in subfolders. You’ll see similar result in output terminal:
The last line is to generate mermaid code for you:
Text representation of diagram
Copy the result and paste it into the target .md (markdown) file
Now, simply copy & paste the result into the existing or newly created MarkDown file (.md extension) and you should be able to see the diagram, whenever in Visual Studio Code or using Azure DevOps:
Preview ‘diagram.md’ file in Visual Studio Code
Alternatively, you can pass the output of the Get-SynapseDocDiagram command directly to a new file:
Get-SynapseDocDiagram -synapse $synapse | Set-Content -Path 'synapse-diagram.md'
Summary
Thanks for reading.
About author
You might also like
The recap of the passing year 2019
Good morning girls and boys, ladies and gentlemen, in the new year 2020! I recently thought that we had started a new decade. But that’s not true. The new decade
How to create DACPAC file?
In this post, I will explain what the DACPAC is and how to create it. In the previous post in DevOps series, I described how to create a new, empty
New blog, first post, big challenge!
Yeeaaahh! First post. First blog. I was waiting for it for at least 2 years. Yup, undoubtedly a “waiting” is the best definition for that. Waiting, because I was doing
4 Comments
AlbertoZGZ
January 25, 16:04Hello,
Im not sure if is working on Azure DevOps
The code generated is:
::: mermaid
graph LR
pipeline.pipeline1 –> dataset.DelimitedText1
dataset.DelimitedText1 –> linkedService.AzureBlobStorage1
IntegrationRuntime.AutoResolveIntegrationRuntime –> managedVirtualNetwork.default
:::
The error generated is
Parse error on line 4:
…nagedVirtualNetwork.default
———————–^
Expecting ‘SEMI’, ‘NEWLINE’, ‘SPACE’, ‘EOF’, ‘SQS’, ‘AMP’, ‘STYLE_SEPARATOR’, ‘PS’, ‘(-‘, ‘STADIUMSTART’, ‘SUBROUTINESTART’, ‘CYLINDERSTART’, ‘DIAMOND_START’, ‘TAGEND’, ‘TRAPSTART’, ‘INVTRAPSTART’, ‘START_LINK’, ‘LINK’, ‘DOWN’, ‘NUM’, ‘COMMA’, ‘ALPHA’, ‘COLON’, ‘MINUS’, ‘BRKT’, ‘DOT’, ‘PUNCTUATION’, ‘UNICODE_TEXT’, ‘PLUS’, ‘EQUALS’, ‘MULT’, ‘UNDERSCORE’, got ‘DEFAULT’
Synapse and ADF Pipeline Dependency Diagrams – Curated SQL
January 26, 13:10[…] Kamil Nowinski uses one of my favorite tools for diagram creation: […]
j500sut
July 19, 04:53Hi, I’m getting the following error:
Invoke-Expression : At line:1 char:30
+ $in = $node.’EndCondition:$[‘metadata’][‘nextPageToken’]’
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Unexpected token ‘metadata’][‘nextPageToken’]” in expression or statement.
At C:\Program Files\WindowsPowerShell\Modules\azure.synapse.tools\0.18.0\private\Get-ReferencedObjects.ps1:33 char:17
+ Invoke-Expression “`$in = `$node.`’$name`'”
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ParserError: (:) [Invoke-Expression], ParseException
+ FullyQualifiedErrorId : UnexpectedToken,Microsoft.PowerShell.Commands.InvokeExpressionCommand
Kamil Nowinski
August 09, 02:27Can you raise a new issue here: https://github.com/SQLPlayer/azure.synapse.tools/issues/new ?