ADF with Batch Compute Linked Service- Error with no description.

Azure Data Factory (ADF) is one of the promising services provided by Azure, bear in mind that this is still in public preview at the moment so the points mentioned here can get outdated pretty soon (and also that’s why you dont see many books on Azure as well). I had a requirement for processing a large amount of incoming data for analytics reasons which forced me to dig a bit deeper into this service along with other related services (SQL DW, Events Hub, HDInsight etc.) recently.

Problem I was facing was related to the Azure Batch recent changes when it went into GA. If you dont specify the region along with the batch account name in the compute Linked Service then it will fail without a much descriptive error message-


JSON script I was using was-

 "name": "HostLinkedService",
 "properties": {
 "type": "AzureBatch",
 "typeProperties": {
 "accountName": "bigdata",
 "accessKey": "**********",
 "poolName": "xmltoavropool",
 "linkedServiceName": "BatchStorageLinkedService"

Pay attention to the bold account name above which was causing this error. Change it to bigdata.northeurope and it will start to work. This was pointed to me by one of the MS program managers Harish Agarwal on the Disqus forum as a recent breaking change in Azure Batch.

As a side note using -Debug flag via powershell is very useful when you are deploying new artefacts in Azure, it shows you each step of the command and a lot of chatty AD interactions, you’ve been warned.