NotFound Status Code when working with ADFv2 via AzureRM.DataFactoryV2 PowerShell module

Azure Data Factory V2 is a great good tool for processing big data. It’s very scalable, flexible and modern cloud data integration service.
You can compose and manage reliable and secure data integration workflows at scale. Use native ADF data connectors to move and transform cloud and on-premises data that can be unstructured, semi-structured, and structured with Hadoop, Azure Data Lake, Spark, SQL Server, Cosmos DB and many other data platforms.
Version 2 of Azure Data Factory was released to general availability (GA) about three months ago. Comparing to previous ADF version – it’s completely new product, contains own UI improving developer’s experience, has many various connectors (including non-Microsoft systems) and capabilities to run Azure Databricks as one of the steps.
No wonder that completely refurbished product has a new module in PowerShell: AzureRM.DataFactoryV2.

Today, I would like to show you how to use one cmdlet to deploy DataFactory v2 using PowerShell script which can be used in order to automate the deployment process afterwards (next post will be about it).

Quite recently, I wanted to automate the deployment process using files being kept in a code repository.
This is important as the files are divided by type of object – as not as the same like ARM template files exported from Azure portal.
As I’m a big advocate of PowerShell for some time – it was obvious to use PowerShell in this case (not mentioning that it should be natural choice).
There are two separate modules: one for V1 and another one for V2.
Once checking documentation – I found that we must keep a specific order of deploying the pieces:

• Create a data factory (that was obvious)
• Create linked services
• Create datasets
• Create a pipeline

So, I have started preparing script to deploy ADFv2:

Set-AzureRmDataFactoryV2 -Name "KamilExp-adf2" `
   -ResourceGroupName "rg-somethingbad" `
   -Location "westeurope"

Then I have got the following error:

Set-AzureRmDataFactoryV2 : HTTP Status Code: NotFound
Error Code: NotFound
Error Message: Operation returned an invalid status code 'NotFound'
Request Id: 6b7eb2a5-b8dc-493c-b538-e3f14bcce730
Timestamp (Utc):09/12/2018 16:35:14
At line:1 char:1
+ Set-AzureRmDataFactoryV2 -Name "KamilExp-adf2" -ResourceGroupName "rg ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Set-AzureRmDataFactoryV2], ErrorResponseException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.DataFactoryV2.SetAzureDataFactoryCommand

Increasingly the description of errors says you… nothing. I got used to.
After an hour of investigation – I have found the issue. One fault was me (the original one), but the second one was caused by module (lack of information).

Hint 1

Check the version of your installed module.

So, firstly I have checked the version of the module I had locally. The name of the module is AzureRm.DataFactoryV2, so run:

Get-Module AzureRm.DataFactoryV2

ModuleType Version Name ExportedCommands
---------- ------- ---- ----------------
Script 0.5.3 AzureRm.DataFactoryV2 {Get-AzureRmDataFactoryV2, Get-AzureRmDataFactoryV2ActivityRun, Get-AzureRmDataFactoryV2Dataset, Get-AzureRmDataFactoryV2IntegrationRunt...

Then, let’s check the latest version fron the Internet:

Find-Module AzureRm.DataFactoryV2

Version Name Repository Description
------- ---- ---------- -----------
0.5.10 AzureRM.DataFactoryV2 PSGallery Microsoft Azure PowerShell - DataFactories service cmdlets for Azure Resource Manager

Seems like need to be updated (you must run a PowerShell session in administrator mode):

Update-Module AzureRm.DataFactoryV2

OK, let’s try run the creation of ADF again:

 
Set-AzureRmDataFactoryV2 -Name "KamilExp-adf2" `
   -ResourceGroupName "rg-somethingbad" `
   -Location "westeurope"

I’ve received the error again, BUT with different, more valuable cause:

Set-AzureRmDataFactoryV2 : HTTP Status Code: NotFound
Error Code: ResourceGroupNotFound
Error Message: Resource group 'rg-somethingbad' could not be found.
Request Id: d051c049-8fb5-4e40-be7e-461c97904018
Timestamp (Utc):09/12/2018 16:38:32
At line:1 char:1
+ Set-AzureRmDataFactoryV2 -Name "KamilExp-adf2" -ResourceGroupName "rg ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Set-AzureRmDataFactoryV2], CloudException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.DataFactoryV2.SetAzureDataFactoryCommand

As you can see – the problem was trivial – I wanted to use non-existing Resource Group.
This can happen when you make just a typo or switiching between many subscriptions and used resource group that belongs to other subscription.

Hint 2

Check your contexts and what RG are available:

Get-AzureRmContext

Name : Visual Studio Enterprise (******-5506-4104-8c84-0051fc36****) - Kamil.Nowinski@******
Account : Kamil.Nowinski@******
SubscriptionName : Visual Studio Enterprise
TenantId : ********-80ee-4819-a9ce-863d5afbea1c
Environment : AzureCloud

Get-AzureRmResourceGroup

ResourceGroupName : rg-datafactory
Location : westeurope
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/******-****/resourceGroups/rg-datafactory

ResourceGroupName : rg-dwh
Location : eastus
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/******-****/resourceGroups/rg-dwh

ResourceGroupName : rg-others
Location : northeurope
ProvisioningState : Succeeded
Tags :
ResourceId : /subscriptions/******-****/resourceGroups/rg-others

Switch to appropriate subscription or change the [ResourceGroupName] parameter and run it again:

Set-AzureRmDataFactoryV2 -Name "KamilExp-adf2" `
   -ResourceGroupName "rg-datafactory" `
   -Location "westeurope"

If that’s doesn’t work for you – restart your PowerShell session. It turning out that even though your module seems updated – still might not work correctly. For example, reloading the module to previous version I had (0.5.3) is not possible:

$m="AzureRM.DataFactoryV2"
Remove-Module $m
Get-Module
Import-Module $m -RequiredVersion 0.5.3

Exception calling "Add" with "2" argument(s): "The key 'Set-AzureRmDataFactoryV2:ResourceGroupName' has already been added to the dictionary."
At C:\Program Files\WindowsPowerShell\Modules\AzureRM.DataFactoryV2\0.5.3\AzureRM.DataFactoryV2.psm1:61 char:9
+ $global:PSDefaultParameterValues.Add($_,
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : PSArgumentException

Simply, restart the PowerShell session and the latest version of the module should works correctly.

Conclusion

Once you notice that error:

Set-AzureRmDataFactoryV2 : HTTP Status Code: NotFound
Error Code: NotFound
Error Message: Operation returned an invalid status code 'NotFound'

Do:
1) update AzureRm.DataFactoryV2 module first.
2) restart PowerShell session (if needed)
3) check the correctness of passing Resource Group Name

Please bear in mind that the module is still developing (0.something version) and thus might contain some errors.
I hope that helps and save you certain minutes/hours.

Cheers,
Kamil