setclrpath fails with "Runtime required" when debugging live process

[loop-evgeny]

Description

I use LLDB to attach to a live .NET process using self-contained deployment, and load SOS:

lldb -o plugin load "/opt/dotnet-sdk/sos/libsosplugin.so" -o sethostruntime "/path/to/my/self-contained/app" -p 1234
...
Executable module set to "/path/to/my/self-contained/app/My.Exe".
Architecture set to: x86_64-pc-linux-gnu.
(lldb) plugin load "/opt/dotnet-sdk/sos/libsosplugin.so"
(lldb) sethostruntime "/path/to/my/self-contained/app"
Using .NET Core runtime (version 0.0) to host the managed SOS code
Host runtime path: /path/to/my/self-contained/app

It appears to work, although the "version 0.0" looks suspicious! Some SOS commands then fail, however:

(lldb) logging on

(lldb) clrstack
 Information: 0 : HostServices.UpdateTarget 1489343 #0
Failed to find runtime module (libcoreclr.so), 0x80004002
Extension commands need it in order to have something to do.
For more information see https://go.microsoft.com/fwlink/?linkid=2135652
ClrStack  failed

(lldb) setclrpath /path/to/my/self-contained/app
 Information: 0 : HostServices.UpdateTarget 1489343 #0
 Error: 0 : System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> Microsoft.Diagnostics.DebugServices.DiagnosticsException: Runtime required
   at Microsoft.Diagnostics.ExtensionCommands.SetClrPath.Invoke()
   at InvokeStub_SetClrPath.Invoke(Object, Object, IntPtr*)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
   --- End of inner exception stack trace ---
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
   at Microsoft.Diagnostics.DebugServices.Implementation.Utilities.Invoke(MethodBase method, Object instance, IServiceProvider provider)
 Error: 0 : Microsoft.Diagnostics.DebugServices.DiagnosticsException: Runtime required
   at Microsoft.Diagnostics.DebugServices.Implementation.Utilities.Invoke(MethodBase method, Object instance, IServiceProvider provider)
   at Microsoft.Diagnostics.DebugServices.Implementation.CommandService.CommandHandler.Invoke(InvocationContext context, IServiceProvider services)
   at Microsoft.Diagnostics.DebugServices.Implementation.CommandService.CommandGroup.Execute(IReadOnlyList`1 commandLine, IServiceProvider services)
   at Microsoft.Diagnostics.DebugServices.Implementation.CommandService.Execute(String commandName, String[] commandLineArray, IServiceProvider services)
   at Microsoft.Diagnostics.DebugServices.Implementation.CommandService.Execute(String commandName, String commandArguments, IServiceProvider services)
   at SOS.Extensions.HostServices.DispatchCommand(IntPtr self, String commandName, String commandArguments, Boolean displayCommandNotFound)
Runtime required
SetClrPath /path/to/my/self-contained/app  failed

(lldb) dumpheap -stat
No CLR runtime found.
This means that a .NET runtime module or the DAC for the runtime can not be found, loaded or downloaded.
For more information see https://go.microsoft.com/fwlink/?linkid=2135652

Configuration

Running on Ubuntu 24.04.2 x64, physical machine, no containerization, lldb version 18.1.3. The .NET app is self-contained, built on an Ubuntu 22.04 machine using .NET SDK 8.0.404.

.NET runtime is not installed globally on the machine running , but .NET SDK 9.0.2 was downloaded to /opt/dotnet-sdk and SOS was installed there using dotnet tool. The same also happens with .NET SDK 8.0.406.

$ /opt/dotnet-sdk/dotnet --info
.NET SDK:
 Version:           9.0.200
 Commit:            90e8b202f2
 Workload version:  9.0.200-manifests.b4a8049f
 MSBuild version:   17.13.8+cbc39bea8

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  24.04
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /opt/dotnet-sdk/sdk/9.0.200/

.NET workloads installed:
There are no installed workloads to display.
Configured to use loose manifests when installing new manifests.

Host:
  Version:      9.0.2
  Architecture: x64
  Commit:       80aa709f5d

.NET SDKs installed:
  9.0.200 [/opt/dotnet-sdk/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 9.0.2 [/opt/dotnet-sdk/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 9.0.2 [/opt/dotnet-sdk/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

(lldb) sosstatus
Target OS: LINUX Architecture: X64 ProcessId: 1489343 (0x16B9BF)
SpecialDiagInfoHeader   : 00007FFFFFF10000 <NONE>
Current symbol store settings:
-> Cache: /home/engine/.dotnet/symbolcache
-> Server: https://msdl.microsoft.com/download/symbols/ Timeout: 4 RetryCount: 0
Extensions loaded:
-> 9.0.607501+a651406e39038aef1dbc7c8097b52953284dba27 /opt/dotnet-sdk/sos/Microsoft.Diagnostics.ExtensionCommands.dll
GC memory usage for managed SOS components: 1,425,648 bytes

Regression?

Worked on Ubuntu 22.04 with .NET 8 SDK, at least.

Other information

The only relevant issue I could find was #3222, but that was about a core dump, not a live process. I have checked that libcoreclr.so is loaded in my case:

(lldb) target modules list
...
[ 11] 0B59EBA9-A939-C7A4-3478-E9281698F809-55E58D4C 0x000077b711c00000 /path/to/my/self-contained/app/libcoreclr.so 
...

The app directory also contains libmscordaccore.so.

[loop-evgeny]

Interestingly, if I run setclrpath immediately upon attaching LLDB (rather than after other SOS commands) I get a different error:

(lldb) sethostruntime "/path/to/my/self-contained/app"
Using .NET Core runtime (version 0.0) to host the managed SOS code
Host runtime path: /path/to/my/self-contained/app
(lldb) logging on
Logging is enabled
/home/engine/on
(lldb) setclrpath /path/to/my/self-contained/app
Could not load file or assembly 'Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8'. The system cannot find the file specified.

SetClrPath /path/to/my/self-contained/app  failed

/path/to/my/self-contained/app/Azure.Core.dll exists. /opt/dotnet-sdk/sos/Azure.Core.dll also exists. Perhaps SOS is somehow getting confused between those two.

[hoyosjs]

@loop-evgeny sethostruntime can't be the directory of a self-contained app. It's the runtime that hosts SOS commands - not the applications host runtime. setclrpath is the correct command here, without setting hostruntime.

[mikem8361]

(lldb) clrstack
 Information: 0 : HostServices.UpdateTarget 1489343 #0
Failed to find runtime module (libcoreclr.so), 0x80004002
Extension commands need it in order to have something to do.
For more information see https://go.microsoft.com/fwlink/?linkid=2135652
ClrStack  failed

Has libcoreclr.so loaded yet? Did you execute process launch yet? target modules should show what modules are loaded.

[loop-evgeny]

@mikem8361 Yes, libcoreclr.so has loaded. The process was already running before LLDB was attached to it.

@hoyosjs Thanks! OK, so with setclrpath /path/to/my/self-contained/app before sethostruntime running clrstack works, but sethostruntime is still needed for other commands like dumpheap:

(lldb) setclrpath /path/to/my/self-contained/app
SOS_HOSTING: Failed to find runtime directory
Runtime module directory: /path/to/my/self-contained/app

(lldb) dumpheap -stat
Unrecognized command 'dumpheap' because managed hosting failed or was disabled. See sethostruntime command for details.

(lldb) sethostruntime /opt/dotnet-sdk/sos
Using .NET Core runtime (version 0.0) to host the managed SOS code
Host runtime path: /opt/dotnet-sdk/sos
(lldb) dumpheap -stat
SOS_HOSTING: Failed to load runtime module /opt/dotnet-sdk/sos/libcoreclr.so
Unrecognized command 'dumpheap' because managed hosting failed or was disabled. See sethostruntime command for details.

(lldb) sethostruntime /opt/dotnet-sdk/shared/Microsoft.AspNetCore.App/8.0.13
Using .NET Core runtime (version 8.0) to host the managed SOS code
Host runtime path: /opt/dotnet-sdk/shared/Microsoft.AspNetCore.App/8.0.13
(lldb) dumpheap -stat
SOS_HOSTING: Failed to load runtime module /opt/dotnet-sdk/shared/Microsoft.AspNetCore.App/8.0.13/libcoreclr.so
Unrecognized command 'dumpheap' because managed hosting failed or was disabled. See sethostruntime command for details.

(lldb) sethostruntime /opt/dotnet-sdk/shared/Microsoft.NETCore.App/8.0.13
Using .NET Core runtime (version 8.0) to host the managed SOS code
Host runtime path: /opt/dotnet-sdk/shared/Microsoft.NETCore.App/8.0.13
(lldb) dumpheap -stat
... (dumpheap finally works!) ...

So it seems like I need setclrpath first, then sethostruntime, but... why?

It's the runtime that hosts SOS commands - not the applications host runtime.

It's unclear to me which runtime that should be. In this case it happens to be the same as the self-contained app's runtime (8.0.13) and the only one installed with the SDK. But what if I installed SDK 8.0 with runtime 9.0 and the self-contained app was running on 6.0? If what SOS needs is the "default" runtime installed with the SDK (same major and minor version) why can't it just find that itself based on its own path?

[loop-evgeny]

Also, this is a regression - what I tried to do (sethostruntime to the app's dir, no setclrpath) works on Ubuntu 22.04.5 with lldb 14.0.0, .NET SDK 7.0.410:

lldb -o "plugin load /opt/dotnet-sdk/sos/libsosplugin.so" -o "sethostruntime /path/to/my/self-contained/app" -p 12345

Executable module set to "/path/to/my/self-contained/app/My.Exe".
Architecture set to: x86_64-pc-linux-gnu.
(lldb) plugin load "/opt/dotnet-sdk/sos/libsosplugin.so"
(lldb) sethostruntime "/path/to/my/self-contained/app"
Using .NET Core runtime to host the managed SOS code
Host runtime path: /path/to/my/self-contained/app

clrstack and dumpheap -stat then work

I then tried it on the same machine, with the same app, but with .NET SDK 8.0.406 and got the problem in the original issue. So this appears to have been broken by .NET SDK 8, rather than the Ubuntu 22->24 upgrade, which makes sense.

[hoyosjs]

Setclrpath tells sos which app runtime you're debugging. This should point to your app only and should usually be not needed unless you're doing custom runtime kind of scenario. Be default SOS will try to find the runtime of your app and look for debugging DLLs for that given runtime. I am not sure why it didn't automatically find it in your session, I believe 7.0 runtime was a Microsoft provided one and 8.0 comes from Canonical - but I'd have to verify to say for sure.

sethostruntime is... A command that is pointing to an odd place: the directory of the runtime/folder containing coreclr that's meant to host the managed commands of SOS. It's meant to point to an SDK style runtime dir/standalone global runtime dir. In your original post it should have pointed to /opt/dotnet-sdk/shared/Microsoft.NETCore.App/9.0.2. pointing it to a self-contained app runtime dif might work, but it might not. We don't host out of self contained as we can't easily reason of what DLLs have been superseded and what runtime APIs will be available.

[loop-evgeny]

OK, so the fact that it worked with .NET SDK 7.0 with sethostruntime pointing to a .NET 8.0 self-contained app was a happy coincidence, I guess.

But why is SOS not able to find its runtime (at ../shared/Microsoft.NETCore.App/X.Y.Z) automatically? Having to script finding the runtime folder is annoying, and it worked without that before.

Is there any documentation explaining all this stuff? I could not find it. https://learn.microsoft.com/en-us/dotnet/core/diagnostics/sos-debugging-extension has a command reference, including those two, but I don't know what "dac/dbi" is and nowhere does it say: "to debug a self-contained application using SOS here is what you need to do: ..."

[hoyosjs]

But why is SOS not able to find its runtime (at ../shared/Microsoft.NETCore.App/X.Y.Z) automatically? Having to script finding the runtime folder is annoying, and it worked without that before.

If the runtime is installed globally, SOS picks it up. Otherwise, it's the users job to tell it what's "trustworthy". It follows classic .NET rules:

diagnostics/src/SOS/extensions/hostcoreclr.cpp

Line 484 in 8a8e7ee

std::vector<ProbingStrategy> strategyList = {

. DOTNET_ROOT and well know locations. Otherwise you have to tell it where you put a runtime.

[loop-evgeny]

Alright, so I have to script finding the runtime somehow.

I hope you can see how all this is totally not intuitive from the point of view of a user of SOS:

1. I had no idea that SOS even required a runtime (separate to that of the app).
2. Knowing that it required a runtime, I would still expect it to use the one installed by the SDK into which I installed SOS (using dotnet-tool). From my point of view, that is all one "dotnet installation" under /opt/dotnet-sdk: an SDK with its runtime and tools.
3. Even knowing that I need to run sethostruntime, it's not at all obvious what directory I need to pass to it, nor that it must be run after setclrpath, nor what setclrpath needs (and why, when it already has the EXE dir?)

It's really a usability minefield. It certainly needs better documentation, but SOS itself really needs to give more helpful error messages to the user.

Unhelpful messages include:

• "Using .NET Core runtime (version 0.0) to host the managed SOS code" actually meaning: no runtime loaded!
• "Runtime required" - OK, what do I do about that? A bit more detail, please!
• (lldb) setclrpath /path/to/my/app
SOS_HOSTING: Failed to find runtime directory
Runtime module directory: /path/to/my/app
  when in fact everything is working fine, as far as I can see
• "Failed to find runtime module (libcoreclr.so), 0x80004002" - but it's loaded into the target process, how is SOS still failing to find it?
• No CLR runtime found.
This means that a .NET runtime module or the DAC for the runtime can not be found, loaded or downloaded.
For more information see https://go.microsoft.com/fwlink/?linkid=2135652
  The page at that URL does not contain the "No CLR runtime found" error
• "Could not load file or assembly 'Azure.Core, Version=1.40.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8'. The system cannot find the file specified." - huh? Why is SOS trying to do something with Azure?!

... and all of that is just from this one "experience" with SOS.