Spring Boot with native image container image build fails on podman due to directory permissions

[dmikusa]

Steps to Repro

1. https://start.spring.io, Spring Boot 3.4.4, Java 21, Gradle. Add deps for native image + spring web. Download. Extract.
2. Run ./gradlew bootBuildImage.

It will fail with the following error:

    [creator]     Finished generating 'com.example.demo.DemoApplication' in 1m 3s.
    [creator]       Removing bytecode
    [creator]     unable to invoke layer creator
    [creator]     unable to remove /workspace/BOOT-INF
    [creator]     unlinkat /workspace/BOOT-INF: permission denied
    [creator]     ERROR: failed to build: exit status 1

Normally, this is the point where the buildpack would clear out the Java class files and things you no longer need given this is a native image. It should be able to write to /workspace, but obviously cannot given this error.

Usually file permissions are inherited from the contents that are pushed into the build container, which I believe are the Boot JAR contents. I don't know the exact mechanics of how the Boot build tools plugin does this though.

Full build log -> build-log-gradle.txt

This was run on Mac OS Sequoia 15.4, Podman Desktop 1.17.2 (running Podman 5.4.1).

If I run the same build with the latest pack cli on the same system using the same JAR file, it does not have this issue. See build log from pack -> pack-log.txt

Paketo had a similar report where the user had this problem in conjunction with the <applicationDirectory>/myapp</applicationDirectory> option (--workspace with pack cli). I'm able to reproduce that as well, but that does not work with pack cli either. I'm mentioning this in case it does have the same underlying cause.

Thanks

[philwebb]

I'm unable to replicate the problem locally but I have Docker Desktop. Flagging to see if anyone else on the team has Podman.

[dmikusa]

I asked around a bit on this, because it's something that has come up with the pack CLI. In a nutshell, what happened with pack in the past had to do with mounting volumes. What I'm told is that when mounting a volume to a non-existing mount point within a container, it results in the mount point being owned by root/root. Furthermore, if at any point the volume is mounted as root/root, you can never mount it as a less privileged user. The root permissions are sticky.

Usually, /workspace is part of the builder image and it has non-root permissions in that image so things work. In pack, the problem ended up being that the tool was loading a different image for the lifecycle first and that image didn't have /workspace defined, so it was triggering this behavior. That was fixed as part of this PR. The fix in a nutshell was to force those directories to exist with the right permissions before volumes are mounted into the container. I don't think that's exactly what's happening here, because I don't see Spring Boot build tools pulling in the lifecycle image. It looks like it's running the lifecycle from the builder. It could be something similar though.

This is why setting the application directory to a custom location seems to trigger this issue more consistently, because that directory is not in the default builder with the required permissions, so it's triggering this behavior and causing the permissions to be root, which means the buildpacks cannot delete the files and you end up with an error.

Anyway, thought I'd share. Maybe this helps track things down.

[philwebb]

Thanks for all the pointers @dmikusa. I installed Podman but unfortunately I still couldn't replicate the problem. I did, however, manager to reproduce the issue in paketo-buildpacks/native-image#366 and I think I have a fix that I hope will also deal with your issue.

Could you give the latest SNAPSHOT a go if you get a minute and see if the problem is resolved?