Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Microsoft is giving away 50,000 FREE Microsoft Certification exam vouchers. Get Fabric certified for FREE! Learn more

Reply
madendever
New Member

ADF Pipeline: Snowflake in AWS Performing Direct PutBlob Operations Instead of Using SHIR

Hello,

 

I'm hoping someone may be able to provide some assistance with a question I have regarding the scenario below.

 

I've created an ADF Pipeline for copying data from Snowflake (in AWS) to SQL Server (on an Azure VM) using Azure Blob as the Staging Location.  For this, I'm using a Self-Hosted Integration Runtime which is hosted on an Azure VM on the same network as the SQL Server.  However, this results in AWS performing PutBlob operations directly into Azure Blob staging instead of running it through the Self-Hosted Integration Runtime (hosted by the Azure VM).  Is this to be expected?

If so, this creates a security concern as the Blob would need to be left open to the public internet in order for AWS to access it (or left open to all AWS ip's in the region).  

The aws ip addresses hitting the blob do not even correspond to the dns of the aws snowflake instance.

 

If anyone can answer this question or provide assistance I'd greatly appreciate it. 

 

1 ACCEPTED SOLUTION
nilendraFabric
Community Champion
Community Champion

Hello @madendever 

 

the behavior you are observing is expected when using Azure Data Factory (ADF) to copy data from Snowflake (hosted on AWS) to SQL Server (hosted on Azure) with Azure Blob Storage as the staging location. This is because ADF optimizes data movement by allowing Snowflake to directly interact with Azure Blob Storage over the public internet, bypassing the Self-Hosted Integration Runtime (SHIR).

 

When using ADF’s Copy Activity, if both the source (Snowflake) and the staging location (Azure Blob Storage) are cloud-based and accessible via public endpoints, ADF uses direct cloud-to-cloud data transfer instead of routing through SHIR.

 

• “If your data store is a managed cloud data service, you can use the Azure Integration Runtime. If the access is restricted to IPs that are approved in the firewall rules, you can add Azure Integration Runtime IPs to the allowed list.”

referenced from this link :https://learn.microsoft.com/en-us/azure/data-factory/connector-snowflake?tabs=data-factory

 

According to Microsoft’s Copy Activity documentation, when copying data between two publicly accessible cloud data stores, ADF uses the Azure Integration Runtime (Azure IR) by default. This integration runtime facilitates direct communication between the source and sink over public endpoints without involving SHIR

The Copy Activity documentation outlines that ADF determines which IR to use based on the connectivity of the source and sink:
• “An integration runtime needs to be associated with each source and sink data store.”

 

If both are accessible via public endpoints, ADF uses Azure IR for direct interaction between the services.

 

To address security concerns:

 

Enable Private Connectivity:
• Use Azure Private Link for Azure Blob Storage or AWS PrivateLink for Snowflake to ensure traffic flows through private networks rather than public endpoints.
2. Restrict Access via Firewall Rules:
• Configure your storage account firewall to allow only specific IP ranges used by Snowflake in your AWS region.

 

please see if this is helpful and accept the solution 

 

thanks

 

 

View solution in original post

3 REPLIES 3
madendever
New Member

Thanks for the reply. That was very informative. Unfortunately, the Snowflake instance is hosted by a 3rd party so there is no way to secure it.

@madendever Could you please accept the solution if this is helpful 

nilendraFabric
Community Champion
Community Champion

Hello @madendever 

 

the behavior you are observing is expected when using Azure Data Factory (ADF) to copy data from Snowflake (hosted on AWS) to SQL Server (hosted on Azure) with Azure Blob Storage as the staging location. This is because ADF optimizes data movement by allowing Snowflake to directly interact with Azure Blob Storage over the public internet, bypassing the Self-Hosted Integration Runtime (SHIR).

 

When using ADF’s Copy Activity, if both the source (Snowflake) and the staging location (Azure Blob Storage) are cloud-based and accessible via public endpoints, ADF uses direct cloud-to-cloud data transfer instead of routing through SHIR.

 

• “If your data store is a managed cloud data service, you can use the Azure Integration Runtime. If the access is restricted to IPs that are approved in the firewall rules, you can add Azure Integration Runtime IPs to the allowed list.”

referenced from this link :https://learn.microsoft.com/en-us/azure/data-factory/connector-snowflake?tabs=data-factory

 

According to Microsoft’s Copy Activity documentation, when copying data between two publicly accessible cloud data stores, ADF uses the Azure Integration Runtime (Azure IR) by default. This integration runtime facilitates direct communication between the source and sink over public endpoints without involving SHIR

The Copy Activity documentation outlines that ADF determines which IR to use based on the connectivity of the source and sink:
• “An integration runtime needs to be associated with each source and sink data store.”

 

If both are accessible via public endpoints, ADF uses Azure IR for direct interaction between the services.

 

To address security concerns:

 

Enable Private Connectivity:
• Use Azure Private Link for Azure Blob Storage or AWS PrivateLink for Snowflake to ensure traffic flows through private networks rather than public endpoints.
2. Restrict Access via Firewall Rules:
• Configure your storage account firewall to allow only specific IP ranges used by Snowflake in your AWS region.

 

please see if this is helpful and accept the solution 

 

thanks

 

 

Helpful resources

Announcements
MarchFBCvideo - carousel

Fabric Monthly Update - March 2025

Check out the March 2025 Fabric update to learn about new features.

March2025 Carousel

Fabric Community Update - March 2025

Find out what's new and trending in the Fabric community.

"); $(".slidesjs-pagination" ).prependTo(".pagination_sec"); $(".slidesjs-pagination" ).append("
"); $(".slidesjs-play.slidesjs-navigation").appendTo(".playpause_sec"); $(".slidesjs-stop.slidesjs-navigation").appendTo(".playpause_sec"); $(".slidesjs-pagination" ).append(""); $(".slidesjs-pagination" ).append(""); } catch(e){ } /* End: This code is added by iTalent as part of iTrack COMPL-455 */ $(".slidesjs-previous.slidesjs-navigation").attr('tabindex', '0'); $(".slidesjs-next.slidesjs-navigation").attr('tabindex', '0'); /* start: This code is added by iTalent as part of iTrack 1859082 */ $('.slidesjs-play.slidesjs-navigation').attr('id','playtitle'); $('.slidesjs-stop.slidesjs-navigation').attr('id','stoptitle'); $('.slidesjs-play.slidesjs-navigation').attr('role','tab'); $('.slidesjs-stop.slidesjs-navigation').attr('role','tab'); $('.slidesjs-play.slidesjs-navigation').attr('aria-describedby','tip1'); $('.slidesjs-stop.slidesjs-navigation').attr('aria-describedby','tip2'); /* End: This code is added by iTalent as part of iTrack 1859082 */ }); $(document).ready(function() { if($("#slides .item").length < 2 ) { /* Fixing Single Slide click issue (commented following code)*/ // $(".item").css("left","0px"); $(".item.slidesjs-slide").attr('style', 'left:0px !important'); $(".slidesjs-stop.slidesjs-navigation").trigger('click'); $(".slidesjs-previous").css("display", "none"); $(".slidesjs-next").css("display", "none"); } var items_length = $(".item.slidesjs-slide").length; $(".slidesjs-pagination-item > button").attr("aria-setsize",items_length); $(".slidesjs-next, .slidesjs-pagination-item button").attr("tabindex","-1"); $(".slidesjs-pagination-item button").attr("role", "tab"); $(".slidesjs-previous").attr("tabindex","-1"); $(".slidesjs-next").attr("aria-hidden","true"); $(".slidesjs-previous").attr("aria-hidden","true"); $(".slidesjs-next").attr("aria-label","Next"); $(".slidesjs-previous").attr("aria-label","Previous"); //$(".slidesjs-stop.slidesjs-navigation").attr("role","button"); //$(".slidesjs-play.slidesjs-navigation").attr("role","button"); $(".slidesjs-pagination").attr("role","tablist").attr("aria-busy","true"); $("li.slidesjs-pagination-item").attr("role","list"); $(".item.slidesjs-slide").attr("tabindex","-1"); $(".item.slidesjs-slide").attr("aria-label","item"); /*$(".slidesjs-stop.slidesjs-navigation").on('click', function() { var itemNumber = parseInt($('.slidesjs-pagination-item > a.active').attr('data-slidesjs-item')); $($('.item.slidesjs-slide')[itemNumber]).find('.c-call-to-action').attr('tabindex', '0'); });*/ $(".slidesjs-stop.slidesjs-navigation, .slidesjs-pagination-item > button").on('click keydown', function() { $.each($('.item.slidesjs-slide'),function(i,el){ $(el).find('.c-call-to-action').attr('tabindex', '-1'); }); var itemNumber = parseInt($('.slidesjs-pagination-item > button.active').attr('data-slidesjs-item')); $($('.item.slidesjs-slide')[itemNumber]).find('.c-call-to-action').attr('tabindex', '0'); }); $(".slidesjs-play.slidesjs-navigation").on('click', function() { $.each($('.item.slidesjs-slide'),function(i,el){ $(el).find('.c-call-to-action').attr('tabindex', '-1'); }); }); $(".slidesjs-pagination-item button").keyup(function(e){ var keyCode = e.keyCode || e.which; if (keyCode == 9) { e.preventDefault(); $(".slidesjs-stop.slidesjs-navigation").trigger('click').blur(); $("button.active").focus(); } }); $(".slidesjs-play").on("click",function (event) { if (event.handleObj.type === "click") { $(".slidesjs-stop").focus(); } else if(event.handleObj.type === "keydown"){ if (event.which === 13 && $(event.target).hasClass("slidesjs-play")) { $(".slidesjs-stop").focus(); } } }); $(".slidesjs-stop").on("click",function (event) { if (event.handleObj.type === "click") { $(".slidesjs-play").focus(); } else if(event.handleObj.type === "keydown"){ if (event.which === 13 && $(event.target).hasClass("slidesjs-stop")) { $(".slidesjs-play").focus(); } } }); $(".slidesjs-pagination-item").keydown(function(e){ switch (e.which){ case 37: //left arrow key $(".slidesjs-previous.slidesjs-navigation").trigger('click'); e.preventDefault(); break; case 39: //right arrow key $(".slidesjs-next.slidesjs-navigation").trigger('click'); e.preventDefault(); break; default: return; } $(".slidesjs-pagination-item button.active").focus(); }); }); // Start This code is added by iTalent as part of iTrack 1859082 $(document).ready(function(){ $("#tip1").attr("aria-hidden","true").addClass("hidden"); $("#tip2").attr("aria-hidden","true").addClass("hidden"); $(".slidesjs-stop.slidesjs-navigation, .slidesjs-play.slidesjs-navigation").attr('title', ''); $("a#playtitle").focus(function(){ $("#tip1").attr("aria-hidden","false").removeClass("hidden"); }); $("a#playtitle").mouseover(function(){ $("#tip1").attr("aria-hidden","false").removeClass("hidden"); }); $("a#playtitle").blur(function(){ $("#tip1").attr("aria-hidden","true").addClass("hidden"); }); $("a#playtitle").mouseleave(function(){ $("#tip1").attr("aria-hidden","true").addClass("hidden"); }); $("a#play").keydown(function(ev){ if (ev.which ==27) { $("#tip1").attr("aria-hidden","true").addClass("hidden"); ev.preventDefault(); return false; } }); $("a#stoptitle").focus(function(){ $("#tip2").attr("aria-hidden","false").removeClass("hidden"); }); $("a#stoptitle").mouseover(function(){ $("#tip2").attr("aria-hidden","false").removeClass("hidden"); }); $("a#stoptitle").blur(function(){ $("#tip2").attr("aria-hidden","true").addClass("hidden"); }); $("a#stoptitle").mouseleave(function(){ $("#tip2").attr("aria-hidden","true").addClass("hidden"); }); $("a#stoptitle").keydown(function(ev){ if (ev.which ==27) { $("#tip2").attr("aria-hidden","true").addClass("hidden"); ev.preventDefault(); return false; } }); }); // End This code is added by iTalent as part of iTrack 1859082