Update Built-in libraries in new Environment

[MikkelFeddersen]

Hi everyone,

I am working in notebooks in Fabric, and I want to update a built-in library to a newer version, by creating a new environment and installing the newer version through PyPI.

However, when I start a session with the new notebook, it still uses the same version as in the public library (transformers 4.37.2 instead of 4.49.0).

I have also tried installing it through %pip and !pip install. This works in the current session but does not install when I create a new version. I have been given admin rights, but that does not help either. Has anyone experienced this?

[Pradeep_O_S]

Unfortunately, there is currently no way to override the runtime packages in Fabric. Only session-level libraries can be modified using %pip and !pip install.

[MikkelFeddersen]

So the Library Management only work for packages that are not in built-in?

[Pradeep_O_S]

It seems so. I encountered the same issue you did. Installing a newer version at the environment level doesn’t appear to have any effect. My hypothesis is that Microsoft is intentionally restricting this to avoid potential dependency conflicts with the standard packages bundled in the environment.

[v-pbandela-msft]

Hi @MikkelFeddersen,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please "Accept  as  Solution" and give a 'Kudos' so other members can easily find it.

Thank you,
Pavan.

[v-pbandela-msft]

Hi @MikkelFeddersen,

Thank you for reaching out in Microsoft Community Forum.

Thank you @nilendraFabric  for the helpful response.

The reason you’re seeing the older library version after restarting the session is because Fabric notebooks use session-based environments, which do not persist custom library installations across sessions.

Please follow below steps to resolve the error;

1. please make sure the updated version is available, run the following at the start of your notebook:
%pip install transformers==4.49.0

2.Create a new environment with the updated library version by specifying the following in requirements.txt:
transformers==4.49.0

3.If you’re still seeing the older version, clear the session cache by restarting the kernel or using:
!pip cache purge

Please continue using Microsoft community forum.

If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.

Regards,
Pavan.

[MikkelFeddersen]

Hi,

I have followed step number 2 the entire time, but whenever I choose the new environment, I still get the old package versions. The command !pip cache purge does not resolve the issue. Best regards.

[v-pbandela-msft]

Hi @MikkelFeddersen,

Thank you for reaching out in Microsoft Community Forum.

Please follow below steps to resolve the error;

1. Run the following command to confirm the notebook is using the correct environment, If it still points to the base environment, reattach the new environment.
!which python

2. Close the notebook completely > Reopen it, go to Kernel > Change Environment, select the new environment, and restart the notebook.

3.Check if the correct version is loaded by running:
import transformers
print(transformers.__version__)

If the version is still incorrect, force reinstall it:
!pip install --force-reinstall transformers==4.49.0

Please continue using Microsoft community forum.

If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.

Regards,
Pavan.

[MikkelFeddersen]

Hi,

When I run the command `!which python` in my own environment, I consistently get the following output:
`/home/trusted-service-user/cluster-env/trident_env/bin/python`
This happens regardless of whether I use my own environment or the default one. I have reattached my own environment multiple times, closed the notebook, restarted the session, etc., but with no changes. It seems I am unable to attach my own environment to the notebook. I have also tried to recreate the environment, but I encountered the same result.

[v-pbandela-msft]

Hi @MikkelFeddersen,

Thank you for the detailed update. Please follow below steps to resolve the error;

1.Even if you created the environment, it won’t be applied unless you publish it.
Go to Fabric > Environments → Verify your environment is "Published" (not just "Saved").

2.Manually Attach the Environment, Please run the below command
!source /home/trusted-service-user/cluster-env/<your_env_name>/bin/activate

3.Force Reinstall the Library and Verify:
For installation; %pip install --force-reinstall transformers==4.49.0

For Verification;
import transformers
print(transformers.__version__)

Please continue using Microsoft community forum.

If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.

Regards,
Pavan.

[MikkelFeddersen]

Hi,

The packages are all published:

But I still cant connect to the environment:

[v-pbandela-msft]

Hi @MikkelFeddersen,

Thank you for the update and sharing the screenshots.

1. Please check the fabric permissions
Go to Admin Portal > Tenant Settings > Environments → Ensure you have "Create and Manage Environments" permissions.

2.Manually Set Environment in Notebook Metadata
Go to Notebook Settings > Advanced > Metadata → Add

{
"kernel": {
"name": "python3",
"environment": "<your_env_name>"
}
}

3.Create a new notebook, select your environment, and verify with below code
!which python

Please continue using Microsoft community forum.

If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.

Regards,
Pavan.

[MikkelFeddersen]

Hi,

We do not see Environments in Tenant Settings, or Advanced in Notebook Settings:

[v-pbandela-msft]

Hi @MikkelFeddersen,

Thank you for reaching out in Microsoft Community Forum.

please follow below steps to resolve the issue;

1. Since you can't see Environments in Tenant Settings, it indicates you lack admin privileges. Ask your Fabric Admin to grant you access to "Create and Manage Environments" in Tenant Settings.

2. Make sure you are using a Premium workspace with Fabric capacity enabled.

3. Manually activate the environment in the current session using:
!source /home/trusted-service-user/cluster-env/<your_env_name>/bin/activate
!pip install --force-reinstall transformers==4.49.0

Please continue using Microsoft community forum.

If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.

Regards,
Pavan.

[MikkelFeddersen]

Hi,

1. The pictures I sent in my previous reply are from our admin user, so the admins cannot see the settings you mentioned.
2. I do have a Fabric capacity.

Best regards

[v-pbandela-msft]

Hi @MikkelFeddersen,

Thank you for reaching out in Microsoft Community Forum.

Please go through the above mentioned steps and check step by step again ,if still faces the same issue then please consider raising a support ticket for further assistance.

To raise a support ticket for Fabric and Power BI, kindly follow the steps outlined in the following guide:
https://learn.microsoft.com/en-us/power-bi/support/create-support-ticket

Please continue using Microsoft community forum.

If you found this post helpful, please consider marking it as "Accept as Solution" and give it a 'Kudos'. if it was helpful. help other members find it more easily.

Regards,
Pavan.

[MikkelFeddersen]

Hi,

I have done both, but unfortunately, that it is not the issue.

[nilendraFabric]

Hello @MikkelFeddersen 

Few things to check :

1) Please make sure in new env , publish of library is successful (Not just saved)

2) selectthe environment from the Environment dropdown in your notebook