Discovering diagram of dependencies in Synapse Analytics and ADF pipelines

Discovering diagram of dependencies in Synapse Analytics and ADF pipelines

Documenting objects dependencies of ETL processes is a tough task. Regardless it is SSIS, ADF, pipelines in Azure Synapse or other systems. The reasons for understanding the current solution can vary either: handover to other team/member of the team, troubleshooting, refactoring, debugging, investigating dependencies due to error, performance issue or others, as well as keen to remove selected/duplicated pipelines or logic.

But there is never a good time to make documentation, and even if that has been done – no one knows how much it’s up to date. The situation is not improved by the fact that quite often there is a lack of (free or built-in) tools for generating such documentation. Sounds familiar? I bet it does.

Wiki Markdown and Mermaid diagrams

If you ever work with Markdown, you know that it is more and more popular format to develop and share documentation, often as a part of the code repository. Most popular tools support the format, including Azure DevOps and GitHub. Markdown supports diagrams in a few different forms, hence one can enhance documentation with a handful sequence diagram, Gantt charts or flowcharts, simply by writing a few lines of plain text.

Generate diagram with PowerShell

It has been some time now since I released the first version of #adftools. The main reason behind the tool is to deploy Azure Data Factory directly from code. But as the name suggests – it is a toolkit. With the tool, you can very easily generate mermaid code and copy/paste it to your markdown file. What do we need to do?

  1. Install/import PowerShell module (if you haven’t done that yet)
    1. azure.datafactory.tools – for Azure Data Factory
    2. azure.synapse.tools – for Azure Synapse pipelines
  2. Download or clone the code onto a local machine
  3. Run one line of code in PowerShell
  4. Copy the result and paste it into the target .md (markdown) file

Let’s do all these steps together.

Install/import PowerShell module

Depending on which code of which service you want to discover – install and import one of the following PS modules:

$module = 'azure.synapse.tools'
Install-Module $module -Scope CurrentUser
Import-Module $module

Download or clone the code

If you don’t have Git Integration set up for the pipelines – do it. I was explaining how to do that in this post, and once you achieve that – you can clone the code into your local machine. The folder should contain the structure of subfolders similar to this one:

Run one line of code in PowerShell

It’s time to run “the magic”. Execute the following code (either in Visual Studio Code or PowerShell ISE / session):

$RootFolder = "x:\!WORK\GitAz\sqlplayer\DataServices\Synapse"
$synapse = Import-SynapseFromFolder -RootFolder $RootFolder -SynapseWorkspaceName 'whatever'
Get-SynapseDocDiagram -synapse $synapse

The first line is to set RootFolder variable with location of code. The second line (import) is going to read all files in subfolders. You’ll see similar result in output terminal:

The last line is to generate mermaid code for you:

Text representation of diagram

Copy the result and paste it into the target .md (markdown) file

Now, simply copy & paste the result into the existing or newly created MarkDown file (.md extension) and you should be able to see the diagram, whenever in Visual Studio Code or using Azure DevOps:

Preview ‘diagram.md’ file in Visual Studio Code

Alternatively, you can pass the output of the Get-SynapseDocDiagram command directly to a new file:

Get-SynapseDocDiagram -synapse $synapse | Set-Content -Path 'synapse-diagram.md'

Summary

I hope the command will be useful for you. As always, should you have any comments or suggestions for improvements, let me know: Raise an issue.

Thanks for reading.

Previous Database projects with SQL Server Data Tools (SSDT)
Next Cloud Formations - A New MVP Led Training Initiative

About author

Kamil Nowinski
Kamil Nowinski 200 posts

Blogger, speaker. Data Platform MVP, MCSE. Senior Data Engineer & data geek. Member of Data Community Poland, co-organizer of SQLDay, Happy husband & father.

View all posts by this author →

You might also like

Security 0 Comments

Store sensitive information with Azure Key Vault

Azure Key Vault is a service which allows you to keep and manage passwords, certificates and other sensitive information securely. Out of the box we have capabilities to manage permissions having

General 0 Comments

Last Week Reading (2021-11-28)

? Press SynapseML: A simple, multilingual, and massively parallel machine learning library Microsoft just announced the release of SynapseML (previously known as MMLSpark) SSH File Transfer Protocol (SFTP) support for

General 2 Comments

My last day of being MVP

Yesterday was my last day of being Microsoft Most Valuable Professional. There are several cases where you may leave the MVP Program: breaking MVP agreement by, for example, sharing confidential

4 Comments

  1. AlbertoZGZ
    January 25, 16:04 Reply

    Hello,

    Im not sure if is working on Azure DevOps

    The code generated is:
    ::: mermaid
    graph LR
    pipeline.pipeline1 –> dataset.DelimitedText1
    dataset.DelimitedText1 –> linkedService.AzureBlobStorage1
    IntegrationRuntime.AutoResolveIntegrationRuntime –> managedVirtualNetwork.default
    :::

    The error generated is
    Parse error on line 4:
    …nagedVirtualNetwork.default
    ———————–^
    Expecting ‘SEMI’, ‘NEWLINE’, ‘SPACE’, ‘EOF’, ‘SQS’, ‘AMP’, ‘STYLE_SEPARATOR’, ‘PS’, ‘(-‘, ‘STADIUMSTART’, ‘SUBROUTINESTART’, ‘CYLINDERSTART’, ‘DIAMOND_START’, ‘TAGEND’, ‘TRAPSTART’, ‘INVTRAPSTART’, ‘START_LINK’, ‘LINK’, ‘DOWN’, ‘NUM’, ‘COMMA’, ‘ALPHA’, ‘COLON’, ‘MINUS’, ‘BRKT’, ‘DOT’, ‘PUNCTUATION’, ‘UNICODE_TEXT’, ‘PLUS’, ‘EQUALS’, ‘MULT’, ‘UNDERSCORE’, got ‘DEFAULT’

  2. j500sut
    July 19, 04:53 Reply

    Hi, I’m getting the following error:

    Invoke-Expression : At line:1 char:30
    + $in = $node.’EndCondition:$[‘metadata’][‘nextPageToken’]’
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Unexpected token ‘metadata’][‘nextPageToken’]” in expression or statement.
    At C:\Program Files\WindowsPowerShell\Modules\azure.synapse.tools\0.18.0\private\Get-ReferencedObjects.ps1:33 char:17
    + Invoke-Expression “`$in = `$node.`’$name`'”
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : ParserError: (:) [Invoke-Expression], ParseException
    + FullyQualifiedErrorId : UnexpectedToken,Microsoft.PowerShell.Commands.InvokeExpressionCommand

Leave a Reply

Click here to cancel reply.