How to Integrate Log Analytics Workspace to Eventhouse

[nandasatria]

I need to analyze data from a Log Analytics workspace in Fabric.

Is it possible to integrate it with Eventhouse so I can run Kusto Queries from Eventhouse while the data remains in the Log Analytics workspace?

Or do we need to move the data from the Log Analytics workspace to Eventhouse? If yes, how can we do it?

Currently, I need to export the data to Blob Storage and use it in a notebook, but in that case, I cannot use KQL

Thanks

[nandasatria]

Hi @v-tangjie-msft 

That was great! I successfully stored the data from Log Analytics using Data Exporter and Event Hub to the KQL database in the Eventhouse. However, the problem is with the data structure; it is in the project fields, not parsed data. When I try to add a transformation event to expand the data, it shows an error: "Please connect this node to the upstream node with a schema before configuring it."

Is this a common issue? Do we need to manually use KQL, like using mv-expand to expand the data in the array and project for every field in the records?

Or is there another best practice to make the data more ready to use?

Thanks

[v-tangjie-msft]

Hi @nandasatria ,

The error message "Please connect this node to the upstream node with a schema before configuring it" typically indicates that the data schema is not properly defined or recognized in the transformation pipeline. This can happen when the data is not parsed correctly or when the schema is not propagated through the nodes.

It is recommended to use the Kusto Query Language (KQL) to parse and expand data. the “mv-expand” operator can be used to expand arrays and nested data structures. The following is an example:

   ```kusto
   let rawData = datatable (RawData: dynamic)
   [
       dynamic({"field1": "value1", "field2": ["value2a", "value2b"]}),
       dynamic({"field1": "value3", "field2": ["value2c", "value2d"]})
   ];
   rawData
   | mv-expand field2
   | project field1, field2
   ```

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly. 

[nandasatria]

Hi @v-tangjie-msft 

Yes, currently I use KQL to parse the data manually, but the problem is that I need to project every field, which requires a lot of effort if the table is large. Moreover, when we need to aggregate incoming data for various analyses, we have to use mv-expand and project for each field repeatedly

This is my current Event Stream Flow:

Is it typical for data from Event Hub to come without a schema, or is the schema from the data in Event Hub not recognized. How to make this more seamless for many aggregation ?

Thanks 

[v-tangjie-msft]

Hi @nandasatria ,

When data is sent to Event Hub, it is typically in a raw format (e.g., JSON, Avro, or plain text). Event Hub itself does not enforce or recognize a schema; it simply acts as a message broker. As a result:

The data may arrive in a nested or semi-structured format (e.g., JSON arrays or objects).

Eventhouse (KQL database) does not automatically infer the schema unless explicitly defined during ingestion or transformation.

This is why you often need to manually parse and flatten the data using KQL functions like parse_json, mv-expand, and extend.

You can try to use Azure Data Factory to transform the data and write it to Eventhouse in a flattened format or use a notebook to preprocess the data and write it back to Eventhouse.

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly. 

[nandasatria]

Thanks for your help @v-tangjie-msft 

[v-tangjie-msft · Accepted solution]

Hi @nandasatria ,

You can export the data from the Log Analytics workspace to an Azure Storage Account or Azure Event Hubs. This can be done using the Log Analytics workspace data export feature. You can then create an Eventhouse and then fetch the data from the event hub into the KQL database in the Eventhouse.

For more details, please refer:

Log Analytics workspace data export in Azure Monitor - Azure Monitor | Microsoft Learn

Create an eventhouse - Microsoft Fabric | Microsoft Learn

Get data from Azure Event Hubs - Microsoft Fabric | Microsoft Learn

If you prefer to continue using Blob Storage, you can set up a scheduled export from a log query using Azure Logic Apps. This method allows you to query data from a Log Analytics workspace and send it to Azure Storage.

Export data from a Log Analytics workspace to a storage account by using Logic Apps - Azure Monitor ...

Best Regards,

Neeko Tang

If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly.