Would I use Dataflow for this use case? Or is it easier to use notebook for transformations

[SamsonYoung20]

Hello,

I am very new to Azure and MS services so I hope this is the right place to ask.

I have a list of csv files for 5 different tables that are generated daily (5 new csvs daily) with new data and placed in our Azure Blob storage account.

I want to create a pipeline that automates the ETL process. Where the new csv files are processed each day then appended onto the tables that I have created in my lake house. 

If I create a dataflow, do I need to select Azure blobs or Azure data lake storage? If I connect to the storage account via either method, I am given just a single query with a table that lists all the contents of the container (all csvs, all subfolders each csv is located in etc).

If I open up one of the csv files and do transformations on it and then set the destination to append onto the table I have already created. How does Dataflow know to get all the files from that folder to process it using the same transformations that I had applied already. Also how does it know which datasets it would need to apply the transfomations to based on which set of data I am transforming?

The csv files always have the same column headers etc, just different rows of data.

There are no Youtube videos I can find that show situations where you want to automate an ETL process using dataflows and pipelines where you would want to be processing a different csv file each day for multiple tables.

Is this possible with dataflows or do I just need to use a notebook instead? 

Thank You,

Paul S

[v-tsaipranay]

Hi @SamsonYoung20 ,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[miguel]

What logic defines what a new file is? Both Notebooks and Dataflows can work, but it all relies on how you define what a "new file" is and how that logic can be translated inside of a Notebook or a Dataflow.

[v-tsaipranay · Accepted solution]

Hi @SamsonYoung20 ,

Thank you for posting in the Microsoft Fabric Community.

For your use case automating the daily ingestion and transformation of CSV files into your Fabric Lakehouse both Dataflows and Notebooks can work, but the choice depends on your requirements.

If your transformations are simple (such as renaming columns, filtering, or basic data cleaning), Dataflows is the easier, no-code solution. You can connect directly to Azure Blob Storage or ADLS, select a folder (so new files are automatically included), apply transformations using Power Query, and append data to your Lakehouse table. Then, use Data Pipelines to schedule and automate the process daily.

However, if you need more control, such as dynamically selecting only the latest files, handling schema changes, or performing complex transformations, a Notebook (PySpark/Pandas) is the better choice. A Notebook allows you to programmatically read new files, apply advanced transformations, and efficiently write the data to the Lakehouse in Delta format for better performance. While Notebooks require coding, they provide greater flexibility, scalability, and efficiency for large datasets. If your needs are straightforward, Dataflows should work well; otherwise, Notebooks give you more power and control over the process.

To decide between these approaches, you can refer to official Microsoft documentation. The Microsoft Fabric Decision Guide provides insights into when to use Dataflows, Pipelines, or Notebooks. If you're specifically looking for guidance on transforming data with Dataflows and Pipelines, you may find the Move and Transform Data with Dataflows guide helpful. 

I hope this will resolve your issue, if you need any further assistance, feel free to reach out.

If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

Thankyou.

[v-tsaipranay]

Hi @SamsonYoung20 ,

I wanted to check if you had the opportunity to review the information provided and also thank you @miguel  for your information. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

[v-tsaipranay]

Hi @SamsonYoung20 ,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.