Finding aws-cfn-bootstrap hidden SELinux violations

Using aws-cfn-bootstrap helper scripts on Amazon Linux is really simple, straightforward and it comes pre-installed. However, using the helper scripts on a Red Hat variant has it’s challenges.

One such challenge is installation of the helper scripts on RHEL/CentOS 7. Trying to install the RPM does not work out of the box. Running one of the scripts yields:

# ./cfn-init 
Traceback (most recent call last):
  File "./cfn-init", line 19, in 
    import cfnbootstrap
ImportError: No module named cfnbootstrap

I have not found a way to install the missing Python module directly from pip but installing the helper scripts from the provided tarball works much better:

/usr/bin/easy_install --script-dir /opt/aws/bin https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz

Now, I believe I have a working installation of the helper scripts… Until I try attempting the next challenge… Configuring a hook that executes a command other than the standard /opt/aws/bin/cfn-init --stack ${AWS::StackId} --resource EC2Instance --region ${AWS::Region} command which is intended to re-execute the user-data on changes to the stack resource’s metadata:

/etc/cfn/hooks.d/cfn-user-data.conf:
  content:
    Fn::Sub: |
      [cfn-user-data-hook]
      triggers=post.update
      path=Resources.EC2Instance.Metadata.AWS::CloudFormation::Init
      action=/bin/curl -sf http://169.254.169.254/latest/user-data | bash -x
      runas=root
  mode: '000400'
  owner: root
  group: root

So… I run some more tests and determine the user-data was not re-executed as I intended.

I start troubleshooting and there’s nothing obvious in logs. Which is confusing because running cfn-init as a one-shot command creates files described in metadata… WTH is going on.

Finally, I realize SELinux is in enforcing mode. I run setenforce 0 and magically, all is well.

The simplest fix would be to run a command to permanently disable SELinux system wide and put the whole problem behind me, but this way of thinking has the potential of causing countless other problems…

So, I decided to dig in and figure out why SELinux is preventing the execution of commands other than cfn-init.

First thought — setroubleshoot — Yes, this should be simple…

I re-enable SELinux, install setroubleshoot and run tests once again and the failure returns.

But when I try to find the violation in the audit logs, I get this:

# sealert -a /var/log/audit/audit.log 
100% done
found 0 alerts in /var/log/audit/audit.log

Whaaaa??

I actually have never run into this before… Typically, SELinux is good about logging reasons why commands and processes are denied execution. So now I’m in un-charted territory. Until a lengthy Google-ing session uncovered this process to disable dontaudit rules.

I execute:

semodule -DB

And re-execute some test changes through a stack update and I start seeing some violations in the audit log:

--------------------------------------------------------------------------------

SELinux is preventing /usr/bin/bash from using the rlimitinh access on a process.

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that bash should be allowed rlimitinh access on processes labeled initrc_t by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c '00-netreport' --raw | audit2allow -M my-00netreport
# semodule -i my-00netreport.pp

--------------------------------------------------------------------------------

SELinux is preventing /usr/bin/python2.7 from using the rlimitinh access on a process.

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that python2.7 should be allowed rlimitinh access on processes labeled setroubleshootd_t by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'setroubleshootd' --raw | audit2allow -M my-setroubleshootd
# semodule -i my-setroubleshootd.pp

Ok, this is something but still not very useful… The suggestions to allow the access do not directly reference any of the commands I’m running so I’m skeptical that creating the policies to allow “rlimitinh access on processes” will work.

I found a bug report of a similar error which gave me the idea that I may have a filesystem label problem. Ok, easy enough, I run this:

# restorecon -R -v /usr
restorecon reset /usr/lib/python2.7/site-packages/aws_cfn_bootstrap-1.4-py2.7.egg/EGG-INFO/scripts context system_u:object_r:lib_t:s0->system_u:object_r:bin_t:s0
restorecon reset /usr/lib/python2.7/site-packages/aws_cfn_bootstrap-1.4-py2.7.egg/EGG-INFO/scripts/cfn-init context system_u:object_r:lib_t:s0->system_u:object_r:bin_t:s0
restorecon reset /usr/lib/python2.7/site-packages/aws_cfn_bootstrap-1.4-py2.7.egg/EGG-INFO/scripts/cfn-signal context system_u:object_r:lib_t:s0->system_u:object_r:bin_t:s0
restorecon reset /usr/lib/python2.7/site-packages/aws_cfn_bootstrap-1.4-py2.7.egg/EGG-INFO/scripts/cfn-get-metadata context system_u:object_r:lib_t:s0->system_u:object_r:bin_t:s0
restorecon reset /usr/lib/python2.7/site-packages/aws_cfn_bootstrap-1.4-py2.7.egg/EGG-INFO/scripts/cfn-hup context system_u:object_r:lib_t:s0->system_u:object_r:bin_t:s0
restorecon reset /usr/lib/python2.7/site-packages/aws_cfn_bootstrap-1.4-py2.7.egg/EGG-INFO/scripts/cfn-elect-cmd-leader context system_u:object_r:lib_t:s0->system_u:object_r:bin_t:s0
restorecon reset /usr/lib/python2.7/site-packages/aws_cfn_bootstrap-1.4-py2.7.egg/EGG-INFO/scripts/cfn-send-cmd-result context system_u:object_r:lib_t:s0->system_u:object_r:bin_t:s0
restorecon reset /usr/lib/python2.7/site-packages/aws_cfn_bootstrap-1.4-py2.7.egg/EGG-INFO/scripts/cfn-send-cmd-event context system_u:object_r:lib_t:s0->system_u:object_r:bin_t:s0

Well, ain’t that something, I relabeled all the files installed by the aws-cfn-bootstrap tarball.

The commands now execute properly so I add this relabel command as part of the install process for aws-cfn-bootstrap.

PS, if you’ve followed along so far, don’t forget to re-enable the dontaudit rules:

semodule -B

So the audit logs don’t fill up with garbage.