SELinux prevents ./svc.sh install executing.

[arcotek-ltd]

Agent Version and Platform

Version of your agent?
vsts-agent-rhel.6-x64-2.163.1

OS of the machine running the agent?
Fedora Server 31

Azure DevOps Type and Version

dev.azure.com

If dev.azure.com, what is your organization name? https://dev.azure.com/public_forum

What's not working?

./svc.sh install is not working.

[vsts@mycomputer vsts_agent]$ sudo ./svc.sh install
Creating launch agent in /etc/systemd/system/vsts.agent.foo\x2dltd.bar.mycomputer.service
Run as user: vsts
Run as uid: 1001
gid: 1001
Failed to enable unit: Unit file vsts.agent.foo\x2dltd.bar.mycomputer.service does not exist.
Failed: failed to enable vsts.agent.foo\x2dltd.bar.mycomputer.service
[vsts@mycomputer vsts_agent]$

foo = VSTS organisation name
bar = VSTS agent pool.

I note that the script does create a file in /etc/systemd/system, however, the name is very odd. I susspect Linux doesn't like the ' and \ in the name:

'vsts.agent.Foo\x2dltd.bar.wsvr22.service'

According to your documentation, it should be in the format:

/etc/systemd/system/vsts.agent.{tfs-name}.{agent-name}.service

Agent and Worker's Diagnostic Logs

Haven't got that far.

[arcotek-ltd]

What may be causing the incorrect formatting is the fact that my organisation name contains a hyphen (-).

I have modified your script:

#SVC_NAME=`systemd-escape --path "vsts.agent.Foo.bar.mycomputer.service"`
SVC_NAME="vsts.agent.Foo.bar.mycomputer.service"

Which gives:

[vsts@mycomputer vsts_agent]$ sudo ./svc.sh install
[sudo] password for vsts:
Creating launch agent in /etc/systemd/system/vsts.agent.Foo.bar.mycomputer.service
Run as user: vsts
Run as uid: 1001
gid: 1001
Failed to enable unit: Unit file vsts.agent.Foo.bar.mycomputer.service does not exist.
Failed: failed to enable vsts.agent.Foo.bar.mycomputer.service
[vsts@wsvr22 vsts_agent]$

The file /etc/systemd/system/vsts.agent.Foo.bar.mycomputer.service is created and contains:

[Unit]
Description=Azure Pipelines Agent (Foo.bar.mycomputer)
After=network.target

[Service]
ExecStart=/home/sa-vsts/vsts_agent/runsvc.sh
User=sa-vsts
WorkingDirectory=/home/sa-vsts/vsts_agent
KillMode=process
KillSignal=SIGTERM
TimeoutStopSec=5min

[Install]
WantedBy=multi-user.target

Which looks correct to me.

[arcotek-ltd]

I've narrowed the error Failed to enable unit: Unit file vsts.agent.Foo.bar.mycomputer.service does not exist. down to the line in the install function:

systemctl enable ${SVC_NAME} || failed "failed to enable ${SVC_NAME}"

Listing the "vsts" service, it is reported as "bad":

[vsts@mycomputer vsts_agent]$ systemctl list-unit-files | grep vsts
vsts.agent.Foo.bar.mycomputer.service bad
[vsts@mycomputer vsts_agent]$

I then analyse the file:

[root@mycomputer system]# systemd-analyze verify vsts.agent.Foo.bar.mycomputer.service
/usr/lib/systemd/system/sssd-kcm.socket:7: ListenStream= references a path below legacy directory /var/run/, updating /var/run/.heim_org.h5l.kcm-socket → /run/.heim_org.h5l.kcm-socket; please update the unit file accordingly.
/usr/lib/systemd/system/pcscd.socket:5: ListenStream= references a path below legacy directory /var/run/, updating /var/run/pcscd/pcscd.comm → /run/pcscd/pcscd.comm; please update the unit file accordingly.
/usr/lib/systemd/system/iscsiuio.service:13: PIDFile= references a path below legacy directory /var/run/, updating /var/run/iscsiuio.pid → /run/iscsiuio.pid; please update the unit file accordingly.
/usr/lib/systemd/system/iscsid.service:11: PIDFile= references a path below legacy directory /var/run/, updating /var/run/iscsid.pid → /run/iscsid.pid; please update the unit file accordingly.
/usr/lib/systemd/system/sssd.service:12: PIDFile= references a path below legacy directory /var/run/, updating /var/run/sssd.pid → /run/sssd.pid; please update the unit file accordingly.

Googling around suggests these are just warnings, but I'm a bit stumped at this point. I wonder if the agent needs to be installed in a directory other than /home/vsts/vsts_agent which is what your documentation suggests.

[arcotek-ltd]

I have solved it. It's SELinux causing the problem, which you do not mention in your documentation.
I totally disable SELinux, rebooted and then ran ./svc.sh install. I reenabled SELinux, rebooted and tried to start the service, which, as expected failed.

I have created a policy to allow SELinux to run, even surviving a reboot, however, I don't know if it will allow the service to be installed. These are the steps I took:

1. Check "current mode" with: sestatus
2. If enforcing, set to permissive mode: setenforce permissive
3. Check auditd is running with: ps -ef | grep auditd. If so:
4. Get AVC denials with: audit2allow -a
5. Create a policy: audit2allow -l -a -M vsts (This creates two files: vsts.pp and vsts.te).
6. View the policy: cat vsts.te
7. Activate the policy: semodule -i vsts.pp
8. Make sure thevsts agent service is stopped: systemctl stop vsts.agent.Foo.bar.mycomputer.service`
9. View audit log with: tail -f /var/log/audi/audit.log (in another SSH session).
10. Start the service: systemctl start vsts.agent.Foo.bar.mycomputer.service
11. Confirm there are no denials in the audit.log. To double-check it will work:
12. Stop service: systemctl stop vsts.agent.Foo.bar.mycomputer.service
13. Put SELinux into enforcing mode: setenforce enforcing
14. Confirm SELinux is enforcing: sestatus | grep "Current mode"
15. Start VSTS agent service: systemctl start vsts.agent.Foo.bar.mycomputer.service
16. Check the service is running: systemctl status vsts.agent.Foo.bar.mycomputer.service
17. Additionally, check in the Azure DevOps portal.
18. Finally, reboot and run the checks again.

I'd be grateful if any Linux boffins out there care to chip in on whether I have done this correctly.

I will leave this open for @microsoft to update their scripts to support "real-world" scenarios.

[arcotek-ltd]

I have asked the document authors to add a piece on SELinux in #7010.

[arcotek-ltd]

SELinux also causes access denied errors when trying to run a pipeline. I put SELinux into permissive mode and ran audit2allow -l -a -M vsts_pipeline as per the above steps.

For Google's bots here is the error I was getting when trying to execute the pipeline:

##[error]System.UnauthorizedAccessException: Access to the path '/home/vsts/vsts_agent/_diag/pages' is denied. ---> System.IO.IOException: Permission denied

[kasper-rutten]

Here ends my 2 day debugging odyssey. Thanks for documenting your struggle! Helped me out tremendously.

Same story, hyphen in org name, SELinux marking unit file as bad. RHEL 8.
Some stuff I googled for search engine purposes:

"Failed to enable unit: Unit file vsts.agent.ORGNAME.AGENTPOOLNAMEAGENTNAME.service does not exist."

"Failed to stop/start/enable vsts.agent.ORGNAME.AGENTPOOLNAME.AGENTNAME.service: Unit vsts.agent.ORGNAME.AGENTPOOLNAME.AGENTNAME.service not loaded."

[vtbassmatt]

It was noted here that users will have to edit /etc/selinux/config to persist changes across reboots. Capturing here for posterity 😁

[FilBot3]

This should probably be updated for all versions of Red Hat Enterprise Linux, RHEL, not just RHEL 6.

[machner]

I had also to add the systemd_unit_file_t label for making the service accessible by systemd (on CentOS 8) - see https://unix.stackexchange.com/questions/573760/service-file-exists-but-is-not-found-by-systemd

[EzzhevNikita]

@ghost Is still an actual issue for you?

[EzzhevNikita]

@ghost Is it still an issue for you?

[EzzhevNikita]

@ghost I'm closing this one since there no activity on this ticket for long time. Feel free to reopen it in case issue is still actual for you.

[prattak86]

I know this doesn't address the full issue, but I realized that the \x2d actually needs to be escaped with an additional \ to replace the hyphens. So a service that has this kind of name: vsts.agent.ORG-NAME.AGENTPOOL-NAMEAGENTNAME.service would look like this on when you try to check the status of the service: vsts.agent.ORG\x2dNAME.AGENTPOOL\x2dNAMEAGENTNAME.service.

This means you need to add an extra backslash (\) for additional escape in order for it to reference correctly: vsts.agent.ORG\\x2dNAME.AGENTPOOL\\x2dNAMEAGENTNAME.service

Therefore, awk and sed come in handy to replace that hyphen sequence value with an additional backslash (\) and then pipe that value back to systemctl with help from xargs. Here is the command pattern I use:

sudo systemctl list-units | grep vsts.agent | awk '{print $1}' | sed 's/\\x2d/\\\\x2d/g' | xargs -I {} sudo systemctl status "{}"

It's ugly, but this works every time for me. Just make sure that if you have a single VM with multiple agents installed (I wouldn't recommend), then be cautious what you are grepping for using the string pattern vsts.agent name that you aren't accidentally getting the wrong agent.