Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Microsoft is giving away 50,000 FREE Microsoft Certification exam vouchers. Get Fabric certified for FREE! Learn more

Reply
jisaenz
Helper I
Helper I

Performance issues running pipelines and notebooks

I am working with the trial capacity  and I am experiencing performance issues excuting pipelines and notebooks that use spark. They are running too slow or never finish. I would like to know what is happening because in the past was runiing normal but since about 15 or 20 days ago the performance is too bad

12 REPLIES 12
tmjones2
Helper I
Helper I

I have the same issue. 40 seconds when run in the notebook, over 3 minutes in a pipeline. And really that 40 seconds should be 1 second, basically the entire time is starting the spark session.

MarcoBB
Regular Visitor

I have the same problem, too. Running the notebook directly to read and transform some data takes, including the startup of the session, around 30 seconds.

 

Executing the same notebook from a data pipeline takes between 22 and 26 minutes. I have no explanation so far. We only have a small fabric capacity, but running the notebook stand alone or from within the pipeline should take about the same amount of time, imo.

 

EDIT: This may not be for every use case that has this problem. In my pipeline that took so long, I executed the Notebooks from a ForEach loop. When I reduced the Batch Count parameter to 2, it went much faster. Still not as fast as running the Notebook directly, but each execution of the notebook took between 30 seconds and 7 minutes. Now it is getting to the usable range. Before I had no value for the batch count, thus 3 Notebooks ran in parallel.

Marco117
Frequent Visitor

I have the same problem, when I test from my spark notebook that transforms and stores a considerable amount of records in the lakehouse, the execution time is 5 min, with this time I am fine!

 

When I do the same execution calling the notebook from a pipeline the execution time increases to 30 min, the execution time increases by 600% 😓 , why does this happen? does the execution of the notebook from a fabric pipeline has less resources?

 

Marusyk
Advocate II
Advocate II

I have the same problem, it took 27 minutes to simply insert 3500+ records in a warehouse.

Marusyk_0-1715774303289.png

For now I'm also using Free Trial but the docs says it is 64 CU!!! It is more than 10 000$/month after trial!!! 

What is the reason of low performance? 

Also some Direct Lake queries runs ~5-10sec.

 

arjaano
Advocate II
Advocate II

I'm experiencing the same issue.

 

Running a small notebook by hand takes about 1 minute. The runtime for running the exact same notebook from a pipeline is over 5 minutes. The notebook is ingesting 10 small CSV-files into lakehouse tables, nothing fancy.

 

v-cboorla-msft
Microsoft Employee
Microsoft Employee

Hi @jisaenz 

 

Thanks for using Microsoft Fabric Community.

There might be a chance of an intermittent issue could you please follow the below steps that might help you.

Temporary Glitch: Clearing cookies and caches can sometimes resolve temporary glitches within the application that might be causing the issue.

Corrupted Data: In rare cases, corrupted data stored in the browser's cache related to Microsoft Fabric might be causing the issue. Clearing the cache removes this potentially problematic data.

Hard Refresh: A hard refresh bypasses the cached version of the webpage and forces the browser to download the latest version from the server. Press Ctrl+Shift+R (Windows) or Cmd+Shift+R (Mac).

Try Microsoft Edge: If you are currently using Chrome or Firefox, try switching to Microsoft Edge to see if the issue persists. Edge is the native browser for Microsoft products and might have better compatibility with Microsoft Fabric.

 

I hope this information helps. Please do let us know if you have any further queries.

 

Thank you.

Microsoft Edge? Really? I'd want to know how Edge has better compatibility with Microsoft Fabric? In what way?

Maybe we should also use Microsoft Surface PC for better Fabric performance since it is also native to Microsoft.

Hi @jisaenz 

 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

 

Thanks.

I am still having issues with performance I have opened a case with Microsoft Fabric Support and until now I don't have received any soltion to my issue. I stop working with Fabric and now I am working with IBM watson and Jupyter labs and the perfromance is much better

Hi @jisaenz 

 

Apologies for the inconvenience.

A reference to the ticket number would be greatly appreciated. This will allow us to track the progress of your request and ensure you receive the most efficient support possible.

 

Thanks.

One of the ticket numbers is 2404260040011639 I have more tickets open for other issues with the same results not a solution. My perception of support is really bad you need to improve

Hi @jisaenz 

 

Thank you for sharing the details.

Helpful resources

Announcements
MarchFBCvideo - carousel

Fabric Monthly Update - March 2025

Check out the March 2025 Fabric update to learn about new features.

March2025 Carousel

Fabric Community Update - March 2025

Find out what's new and trending in the Fabric community.

"); $(".slidesjs-pagination" ).prependTo(".pagination_sec"); $(".slidesjs-pagination" ).append("
"); $(".slidesjs-play.slidesjs-navigation").appendTo(".playpause_sec"); $(".slidesjs-stop.slidesjs-navigation").appendTo(".playpause_sec"); $(".slidesjs-pagination" ).append(""); $(".slidesjs-pagination" ).append(""); } catch(e){ } /* End: This code is added by iTalent as part of iTrack COMPL-455 */ $(".slidesjs-previous.slidesjs-navigation").attr('tabindex', '0'); $(".slidesjs-next.slidesjs-navigation").attr('tabindex', '0'); /* start: This code is added by iTalent as part of iTrack 1859082 */ $('.slidesjs-play.slidesjs-navigation').attr('id','playtitle'); $('.slidesjs-stop.slidesjs-navigation').attr('id','stoptitle'); $('.slidesjs-play.slidesjs-navigation').attr('role','tab'); $('.slidesjs-stop.slidesjs-navigation').attr('role','tab'); $('.slidesjs-play.slidesjs-navigation').attr('aria-describedby','tip1'); $('.slidesjs-stop.slidesjs-navigation').attr('aria-describedby','tip2'); /* End: This code is added by iTalent as part of iTrack 1859082 */ }); $(document).ready(function() { if($("#slides .item").length < 2 ) { /* Fixing Single Slide click issue (commented following code)*/ // $(".item").css("left","0px"); $(".item.slidesjs-slide").attr('style', 'left:0px !important'); $(".slidesjs-stop.slidesjs-navigation").trigger('click'); $(".slidesjs-previous").css("display", "none"); $(".slidesjs-next").css("display", "none"); } var items_length = $(".item.slidesjs-slide").length; $(".slidesjs-pagination-item > button").attr("aria-setsize",items_length); $(".slidesjs-next, .slidesjs-pagination-item button").attr("tabindex","-1"); $(".slidesjs-pagination-item button").attr("role", "tab"); $(".slidesjs-previous").attr("tabindex","-1"); $(".slidesjs-next").attr("aria-hidden","true"); $(".slidesjs-previous").attr("aria-hidden","true"); $(".slidesjs-next").attr("aria-label","Next"); $(".slidesjs-previous").attr("aria-label","Previous"); //$(".slidesjs-stop.slidesjs-navigation").attr("role","button"); //$(".slidesjs-play.slidesjs-navigation").attr("role","button"); $(".slidesjs-pagination").attr("role","tablist").attr("aria-busy","true"); $("li.slidesjs-pagination-item").attr("role","list"); $(".item.slidesjs-slide").attr("tabindex","-1"); $(".item.slidesjs-slide").attr("aria-label","item"); /*$(".slidesjs-stop.slidesjs-navigation").on('click', function() { var itemNumber = parseInt($('.slidesjs-pagination-item > a.active').attr('data-slidesjs-item')); $($('.item.slidesjs-slide')[itemNumber]).find('.c-call-to-action').attr('tabindex', '0'); });*/ $(".slidesjs-stop.slidesjs-navigation, .slidesjs-pagination-item > button").on('click keydown', function() { $.each($('.item.slidesjs-slide'),function(i,el){ $(el).find('.c-call-to-action').attr('tabindex', '-1'); }); var itemNumber = parseInt($('.slidesjs-pagination-item > button.active').attr('data-slidesjs-item')); $($('.item.slidesjs-slide')[itemNumber]).find('.c-call-to-action').attr('tabindex', '0'); }); $(".slidesjs-play.slidesjs-navigation").on('click', function() { $.each($('.item.slidesjs-slide'),function(i,el){ $(el).find('.c-call-to-action').attr('tabindex', '-1'); }); }); $(".slidesjs-pagination-item button").keyup(function(e){ var keyCode = e.keyCode || e.which; if (keyCode == 9) { e.preventDefault(); $(".slidesjs-stop.slidesjs-navigation").trigger('click').blur(); $("button.active").focus(); } }); $(".slidesjs-play").on("click",function (event) { if (event.handleObj.type === "click") { $(".slidesjs-stop").focus(); } else if(event.handleObj.type === "keydown"){ if (event.which === 13 && $(event.target).hasClass("slidesjs-play")) { $(".slidesjs-stop").focus(); } } }); $(".slidesjs-stop").on("click",function (event) { if (event.handleObj.type === "click") { $(".slidesjs-play").focus(); } else if(event.handleObj.type === "keydown"){ if (event.which === 13 && $(event.target).hasClass("slidesjs-stop")) { $(".slidesjs-play").focus(); } } }); $(".slidesjs-pagination-item").keydown(function(e){ switch (e.which){ case 37: //left arrow key $(".slidesjs-previous.slidesjs-navigation").trigger('click'); e.preventDefault(); break; case 39: //right arrow key $(".slidesjs-next.slidesjs-navigation").trigger('click'); e.preventDefault(); break; default: return; } $(".slidesjs-pagination-item button.active").focus(); }); }); // Start This code is added by iTalent as part of iTrack 1859082 $(document).ready(function(){ $("#tip1").attr("aria-hidden","true").addClass("hidden"); $("#tip2").attr("aria-hidden","true").addClass("hidden"); $(".slidesjs-stop.slidesjs-navigation, .slidesjs-play.slidesjs-navigation").attr('title', ''); $("a#playtitle").focus(function(){ $("#tip1").attr("aria-hidden","false").removeClass("hidden"); }); $("a#playtitle").mouseover(function(){ $("#tip1").attr("aria-hidden","false").removeClass("hidden"); }); $("a#playtitle").blur(function(){ $("#tip1").attr("aria-hidden","true").addClass("hidden"); }); $("a#playtitle").mouseleave(function(){ $("#tip1").attr("aria-hidden","true").addClass("hidden"); }); $("a#play").keydown(function(ev){ if (ev.which ==27) { $("#tip1").attr("aria-hidden","true").addClass("hidden"); ev.preventDefault(); return false; } }); $("a#stoptitle").focus(function(){ $("#tip2").attr("aria-hidden","false").removeClass("hidden"); }); $("a#stoptitle").mouseover(function(){ $("#tip2").attr("aria-hidden","false").removeClass("hidden"); }); $("a#stoptitle").blur(function(){ $("#tip2").attr("aria-hidden","true").addClass("hidden"); }); $("a#stoptitle").mouseleave(function(){ $("#tip2").attr("aria-hidden","true").addClass("hidden"); }); $("a#stoptitle").keydown(function(ev){ if (ev.which ==27) { $("#tip2").attr("aria-hidden","true").addClass("hidden"); ev.preventDefault(); return false; } }); }); // End This code is added by iTalent as part of iTrack 1859082