Skip to main content
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Microsoft is giving away 50,000 FREE Microsoft Certification exam vouchers. Get Fabric certified for FREE! Learn more

Reply
gthill
Frequent Visitor

Data workflows - Run a Microsoft Fabric Item Job

‎07-30-2024 01:58 AM

Has anyone been able to get a Fabric Data workflow to run a Fabric Item Job?

Run a Microsoft Fabric item job in Apache Airflow DAGs. - Microsoft Fabric | Microsoft Learn

I have the authentication working, but hitting this error and not sure where to find this module? I cannot find it on PyPI

ModuleNotFoundError: No module named 'airflow.providers.microsoft.powerbi'
Labels:
Message 1 of 6
1,508 Views
0
5 REPLIES 5
nsprudhvi
Advocate I
Advocate I

‎10-23-2024 05:26 AM

Hi @gthill ,

It sounds like you're trying to use the Power BI integration in Airflow but are running into issues with the module. You can resolve this by adding the airflow-powerbi-plugin to your requirements.txt file. Alternatively, you can add it directly to your Airflow environment settings.

 

nsprudhvi_0-1729686164964.png

 

Once you've done that, make sure to restart your Airflow scheduler and web server for the changes to take effect.

Let me know if you need any further assistance!

 

Best regards,
Sai Prudhvi Neelakantam
Data Engineer | 3x Microsoft Certified

💼 LinkedIn: in/saiprudhvineelakantam | 🌐 geekindata.com

If this helps, kindly mark this as the accepted solution! 👍

 

Message 6 of 6
923 Views
0
Anonymous
Not applicable

‎08-08-2024 02:50 AM

Hey @gthill , Can you share Airflow logs for your issue? 

Message 4 of 6
1,396 Views
0
gthill
Frequent Visitor
In response to Anonymous

‎08-09-2024 02:43 AM

Hey @Anonymous - thank you for your reply, sure here is the log contents...

 

af-8a562d6a58184d509fe8af27d6267e64-worker-0.af-8a562d6a58184d509fe8af27d6267e64-worker.adf.svc.cluster.local
*** Found logs served from host http://af-8a562d6a58184d509fe8af27d6267e64-worker-0.af-8a562d6a58184d509fe8af27d6267e64-worker.adf.svc.cluster.local:8793/log/dag_id=Run_Fabric_Item/run_id=manual__2024-07-31T14:04:38.329715+00:00/task_id=run_fabric_pipeline/attempt=1.log
[2024-07-31T14:04:40.081+0000] {logging_mixin.py:150} INFO - Changing /opt/airflow/logs/dag_id=Run_Fabric_Item/run_id=manual__2024-07-31T14:04:38.329715+00:00/task_id=run_fabric_pipeline permission to 509
[2024-07-31T14:04:40.262+0000] {logging_mixin.py:150} INFO - Changing /opt/airflow/logs/dag_id=Run_Fabric_Item/run_id=manual__2024-07-31T14:04:38.329715+00:00/task_id=run_fabric_pipeline permission to 509
[2024-07-31T14:04:40.364+0000] {taskinstance.py:1103} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: Run_Fabric_Item.run_fabric_pipeline manual__2024-07-31T14:04:38.329715+00:00 [queued]>
[2024-07-31T14:04:40.377+0000] {taskinstance.py:1103} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: Run_Fabric_Item.run_fabric_pipeline manual__2024-07-31T14:04:38.329715+00:00 [queued]>
[2024-07-31T14:04:40.377+0000] {taskinstance.py:1308} INFO - Starting attempt 1 of 1
[2024-07-31T14:04:40.395+0000] {taskinstance.py:1327} INFO - Executing <Task(FabricRunItemOperator): run_fabric_pipeline> on 2024-07-31 14:04:38.329715+00:00
[2024-07-31T14:04:40.400+0000] {standard_task_runner.py:57} INFO - Started process 51 to run task
[2024-07-31T14:04:40.406+0000] {standard_task_runner.py:84} INFO - Running: ['airflow', 'tasks', 'run', 'Run_Fabric_Item', 'run_fabric_pipeline', 'manual__2024-07-31T14:04:38.329715+00:00', '--job-id', '48', '--raw', '--subdir', 'DAGS_FOLDER/fabric_test.py', '--cfg-path', '/tmp/tmpv9e7md6v']
[2024-07-31T14:04:40.407+0000] {standard_task_runner.py:85} INFO - Job 48: Subtask run_fabric_pipeline
[2024-07-31T14:04:40.493+0000] {warnings.py:109} WARNING - /home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:195: DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
SQL_ALCHEMY_CONN = conf.get("database", "SQL_ALCHEMY_CONN")
[2024-07-31T14:04:40.731+0000] {logging_mixin.py:150} INFO - Changing /opt/airflow/logs/dag_id=Run_Fabric_Item/run_id=manual__2024-07-31T14:04:38.329715+00:00/task_id=run_fabric_pipeline permission to 509
[2024-07-31T14:04:40.743+0000] {task_command.py:410} INFO - Running <TaskInstance: Run_Fabric_Item.run_fabric_pipeline manual__2024-07-31T14:04:38.329715+00:00 [running]> on host af-8a562d6a58184d509fe8af27d6267e64-worker-0.af-8a562d6a58184d509fe8af27d6267e64-worker.adf.svc.cluster.local
[2024-07-31T14:04:41.172+0000] {taskinstance.py:1545} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='Run_Fabric_Item' AIRFLOW_CTX_TASK_ID='run_fabric_pipeline' AIRFLOW_CTX_EXECUTION_DATE='2024-07-31T14:04:38.329715+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-07-31T14:04:38.329715+00:00'
[2024-07-31T14:04:41.285+0000] {base.py:73} INFO - Using connection ID 'fabric_default' for task execution.
[2024-07-31T14:04:42.522+0000] {fabric.py:158} INFO - Deferring the task to wait for item run to complete.
[2024-07-31T14:04:42.597+0000] {taskinstance.py:1415} INFO - Pausing task as DEFERRED. dag_id=Run_Fabric_Item, task_id=run_fabric_pipeline, execution_date=20240731T140438, start_date=20240731T140440
[2024-07-31T14:04:42.665+0000] {local_task_job_runner.py:222} INFO - Task exited with return code 100 (task deferral)
[2024-07-31T14:04:43.829+0000] {logging_mixin.py:150} INFO - Changing /opt/airflow/logs/dag_id=Run_Fabric_Item/run_id=manual__2024-07-31T14:04:38.329715+00:00/task_id=run_fabric_pipeline permission to 509
[2024-07-31T14:04:44.004+0000] {logging_mixin.py:150} INFO - Changing /opt/airflow/logs/dag_id=Run_Fabric_Item/run_id=manual__2024-07-31T14:04:38.329715+00:00/task_id=run_fabric_pipeline permission to 509
[2024-07-31T14:04:44.109+0000] {taskinstance.py:1103} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: Run_Fabric_Item.run_fabric_pipeline manual__2024-07-31T14:04:38.329715+00:00 [queued]>
[2024-07-31T14:04:44.125+0000] {taskinstance.py:1103} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: Run_Fabric_Item.run_fabric_pipeline manual__2024-07-31T14:04:38.329715+00:00 [queued]>
[2024-07-31T14:04:44.130+0000] {taskinstance.py:1306} INFO - Resuming after deferral
[2024-07-31T14:04:44.148+0000] {taskinstance.py:1327} INFO - Executing <Task(FabricRunItemOperator): run_fabric_pipeline> on 2024-07-31 14:04:38.329715+00:00
[2024-07-31T14:04:44.153+0000] {standard_task_runner.py:57} INFO - Started process 53 to run task
[2024-07-31T14:04:44.158+0000] {standard_task_runner.py:84} INFO - Running: ['airflow', 'tasks', 'run', 'Run_Fabric_Item', 'run_fabric_pipeline', 'manual__2024-07-31T14:04:38.329715+00:00', '--job-id', '49', '--raw', '--subdir', 'DAGS_FOLDER/fabric_test.py', '--cfg-path', '/tmp/tmpzr7ui6kt']
[2024-07-31T14:04:44.159+0000] {standard_task_runner.py:85} INFO - Job 49: Subtask run_fabric_pipeline
[2024-07-31T14:04:44.241+0000] {warnings.py:109} WARNING - /home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py:195: DeprecationWarning: The sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option in [database] - the old setting has been used, but please update your config.
SQL_ALCHEMY_CONN = conf.get("database", "SQL_ALCHEMY_CONN")
[2024-07-31T14:04:44.412+0000] {logging_mixin.py:150} INFO - Changing /opt/airflow/logs/dag_id=Run_Fabric_Item/run_id=manual__2024-07-31T14:04:38.329715+00:00/task_id=run_fabric_pipeline permission to 509
[2024-07-31T14:04:44.434+0000] {task_command.py:410} INFO - Running <TaskInstance: Run_Fabric_Item.run_fabric_pipeline manual__2024-07-31T14:04:38.329715+00:00 [running]> on host af-8a562d6a58184d509fe8af27d6267e64-worker-0.af-8a562d6a58184d509fe8af27d6267e64-worker.adf.svc.cluster.local
[2024-07-31T14:04:44.848+0000] {taskinstance.py:1598} ERROR - Trigger failed:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/triggerer_job_runner.py", line 686, in update_triggers
trigger_class = self.get_trigger_by_classpath(new_trigger_orm.classpath)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/triggerer_job_runner.py", line 727, in get_trigger_by_classpath
self.trigger_cache[classpath] = import_string(classpath)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/module_loading.py", line 36, in import_string
module = import_module(module_path)
File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'airflow.providers.microsoft.powerbi'
[2024-07-31T14:04:44.921+0000] {taskinstance.py:1824} ERROR - Task failed with exception
airflow.exceptions.TaskDeferralError: Trigger failure
[2024-07-31T14:04:44.926+0000] {taskinstance.py:1345} INFO - Marking task as FAILED. dag_id=Run_Fabric_Item, task_id=run_fabric_pipeline, execution_date=20240731T140438, start_date=20240731T140440, end_date=20240731T140444
[2024-07-31T14:04:44.944+0000] {standard_task_runner.py:104} ERROR - Failed to execute job 49 for task run_fabric_pipeline (Trigger failure; 53)
[2024-07-31T14:04:44.972+0000] {local_task_job_runner.py:225} INFO - Task exited with return code 1
[2024-07-31T14:04:45.066+0000] {taskinstance.py:2653} INFO - 0 downstream tasks scheduled from follow-on schedule check

Message 5 of 6
1,383 Views
0
v-kongfanf-msft
Community Support
Community Support

‎07-30-2024 07:25 PM

Hi @gthill ,

 

This kind of reported problem is generally related to whether the module is installed correctly, and whether any other dependencies are required.
See if the following similar solution helps you.

python - No module named 'airflow.providers' - Stack Overflow

python - ModuleNotFoundError: No module named 'airflow.providers.microsoft' in Airflow v2.9.0 - Stac...

 

Best Regards,
Adamk Kong

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Message 2 of 6
1,465 Views
0
gthill
Frequent Visitor
In response to v-kongfanf-msft

‎07-31-2024 05:38 AM

Hi Adamk - thank you for your response. I understand that there is a missing module, however my problem is I cannot find the module to install from anywhere - I tried apache-airflow-providers-microsoft-azure but it is not contained in that

Message 3 of 6
1,457 Views
0

Helpful resources

Announcements
MarchFBCvideo - carousel

Fabric Monthly Update - March 2025

Check out the March 2025 Fabric update to learn about new features.

March2025 Carousel

Fabric Community Update - March 2025

Find out what's new and trending in the Fabric community.

View All
"); $(".slidesjs-pagination" ).prependTo(".pagination_sec"); $(".slidesjs-pagination" ).append("
"); $(".slidesjs-play.slidesjs-navigation").appendTo(".playpause_sec"); $(".slidesjs-stop.slidesjs-navigation").appendTo(".playpause_sec"); $(".slidesjs-pagination" ).append(""); $(".slidesjs-pagination" ).append(""); } catch(e){ } /* End: This code is added by iTalent as part of iTrack COMPL-455 */ $(".slidesjs-previous.slidesjs-navigation").attr('tabindex', '0'); $(".slidesjs-next.slidesjs-navigation").attr('tabindex', '0'); /* start: This code is added by iTalent as part of iTrack 1859082 */ $('.slidesjs-play.slidesjs-navigation').attr('id','playtitle'); $('.slidesjs-stop.slidesjs-navigation').attr('id','stoptitle'); $('.slidesjs-play.slidesjs-navigation').attr('role','tab'); $('.slidesjs-stop.slidesjs-navigation').attr('role','tab'); $('.slidesjs-play.slidesjs-navigation').attr('aria-describedby','tip1'); $('.slidesjs-stop.slidesjs-navigation').attr('aria-describedby','tip2'); /* End: This code is added by iTalent as part of iTrack 1859082 */ }); $(document).ready(function() { if($("#slides .item").length < 2 ) { /* Fixing Single Slide click issue (commented following code)*/ // $(".item").css("left","0px"); $(".item.slidesjs-slide").attr('style', 'left:0px !important'); $(".slidesjs-stop.slidesjs-navigation").trigger('click'); $(".slidesjs-previous").css("display", "none"); $(".slidesjs-next").css("display", "none"); } var items_length = $(".item.slidesjs-slide").length; $(".slidesjs-pagination-item > button").attr("aria-setsize",items_length); $(".slidesjs-next, .slidesjs-pagination-item button").attr("tabindex","-1"); $(".slidesjs-pagination-item button").attr("role", "tab"); $(".slidesjs-previous").attr("tabindex","-1"); $(".slidesjs-next").attr("aria-hidden","true"); $(".slidesjs-previous").attr("aria-hidden","true"); $(".slidesjs-next").attr("aria-label","Next"); $(".slidesjs-previous").attr("aria-label","Previous"); //$(".slidesjs-stop.slidesjs-navigation").attr("role","button"); //$(".slidesjs-play.slidesjs-navigation").attr("role","button"); $(".slidesjs-pagination").attr("role","tablist").attr("aria-busy","true"); $("li.slidesjs-pagination-item").attr("role","list"); $(".item.slidesjs-slide").attr("tabindex","-1"); $(".item.slidesjs-slide").attr("aria-label","item"); /*$(".slidesjs-stop.slidesjs-navigation").on('click', function() { var itemNumber = parseInt($('.slidesjs-pagination-item > a.active').attr('data-slidesjs-item')); $($('.item.slidesjs-slide')[itemNumber]).find('.c-call-to-action').attr('tabindex', '0'); });*/ $(".slidesjs-stop.slidesjs-navigation, .slidesjs-pagination-item > button").on('click keydown', function() { $.each($('.item.slidesjs-slide'),function(i,el){ $(el).find('.c-call-to-action').attr('tabindex', '-1'); }); var itemNumber = parseInt($('.slidesjs-pagination-item > button.active').attr('data-slidesjs-item')); $($('.item.slidesjs-slide')[itemNumber]).find('.c-call-to-action').attr('tabindex', '0'); }); $(".slidesjs-play.slidesjs-navigation").on('click', function() { $.each($('.item.slidesjs-slide'),function(i,el){ $(el).find('.c-call-to-action').attr('tabindex', '-1'); }); }); $(".slidesjs-pagination-item button").keyup(function(e){ var keyCode = e.keyCode || e.which; if (keyCode == 9) { e.preventDefault(); $(".slidesjs-stop.slidesjs-navigation").trigger('click').blur(); $("button.active").focus(); } }); $(".slidesjs-play").on("click",function (event) { if (event.handleObj.type === "click") { $(".slidesjs-stop").focus(); } else if(event.handleObj.type === "keydown"){ if (event.which === 13 && $(event.target).hasClass("slidesjs-play")) { $(".slidesjs-stop").focus(); } } }); $(".slidesjs-stop").on("click",function (event) { if (event.handleObj.type === "click") { $(".slidesjs-play").focus(); } else if(event.handleObj.type === "keydown"){ if (event.which === 13 && $(event.target).hasClass("slidesjs-stop")) { $(".slidesjs-play").focus(); } } }); $(".slidesjs-pagination-item").keydown(function(e){ switch (e.which){ case 37: //left arrow key $(".slidesjs-previous.slidesjs-navigation").trigger('click'); e.preventDefault(); break; case 39: //right arrow key $(".slidesjs-next.slidesjs-navigation").trigger('click'); e.preventDefault(); break; default: return; } $(".slidesjs-pagination-item button.active").focus(); }); }); // Start This code is added by iTalent as part of iTrack 1859082 $(document).ready(function(){ $("#tip1").attr("aria-hidden","true").addClass("hidden"); $("#tip2").attr("aria-hidden","true").addClass("hidden"); $(".slidesjs-stop.slidesjs-navigation, .slidesjs-play.slidesjs-navigation").attr('title', ''); $("a#playtitle").focus(function(){ $("#tip1").attr("aria-hidden","false").removeClass("hidden"); }); $("a#playtitle").mouseover(function(){ $("#tip1").attr("aria-hidden","false").removeClass("hidden"); }); $("a#playtitle").blur(function(){ $("#tip1").attr("aria-hidden","true").addClass("hidden"); }); $("a#playtitle").mouseleave(function(){ $("#tip1").attr("aria-hidden","true").addClass("hidden"); }); $("a#play").keydown(function(ev){ if (ev.which ==27) { $("#tip1").attr("aria-hidden","true").addClass("hidden"); ev.preventDefault(); return false; } }); $("a#stoptitle").focus(function(){ $("#tip2").attr("aria-hidden","false").removeClass("hidden"); }); $("a#stoptitle").mouseover(function(){ $("#tip2").attr("aria-hidden","false").removeClass("hidden"); }); $("a#stoptitle").blur(function(){ $("#tip2").attr("aria-hidden","true").addClass("hidden"); }); $("a#stoptitle").mouseleave(function(){ $("#tip2").attr("aria-hidden","true").addClass("hidden"); }); $("a#stoptitle").keydown(function(ev){ if (ev.which ==27) { $("#tip2").attr("aria-hidden","true").addClass("hidden"); ev.preventDefault(); return false; } }); }); // End This code is added by iTalent as part of iTrack 1859082
Top Kudoed Authors
View All