AppArmor integration

Rationale

Here #2840 (comment 72526) are some arguments of choosing AppArmor over SELinux, as mandatory access control framework for ISPConfig.

Instead of redundant general considerations, let’s target them section by section, with implementation proposals.

AppArmor current status in ISPConfig

There are no specific problems with ispconfig, just general problems that apparmor caused scripts and applications to fail on hosting systems due to false positives or non optimal rules that were delivered with apparmor. We had seen problems in the past that apparmor blocked services or the execution of scripts like cgi scripts etc. If you dont see any such problems on your server, then you can keep apparmor acticated. Maybe the apparmor rules had improved in the meantime so that these false positives dont occur anymore. https://www.howtoforge.com/community/threads/why-would-i-disable-apparmor.57564/

It is not planned to write an apparmor ruleset for ISPConfig at the moment. https://www.howtoforge.com/community/threads/how-to-set-up-apparmor-for-ispconfig-3-052.62673/

Disable and remove AppArmor as it might cause conflicts during the installation of ISPConfig: https://www.scaleway.com/en/docs/installing-and-configuring-ispconfig-on-ubuntu/

Try installing with AppArmor disabled and enforce it later on.

https://www.howtoforge.com/community/search/1009263/?q=apparmor&o=relevance - Majority of forum posts recommends disabling AppArmor. There are also several false positives (issues non related to AppArmor).

How is competition doing

Proprietary control panels are kind of lazy, giving up feature request of AppArmor integration https://plesk.uservoice.com/forums/184549-feature-suggestions/suggestions/6091562-implement-apparmor-support-for-apache, in favor of CloudLinux integrations https://www.cloudlinux.com/all-products/product-overview/cloudlinuxos. A similar approach has http://wiki.centos-webpanel.com/cwp-secure-centos-kernel, requiring MANAGED CWP, from http://centos-webpanel.com/support-services

Cloudlinux approach, of proprietary technologies and patches, means vendor lock and doesn’t match ISPConfig true open source policy.

Other open source hosting panels fail to implement, recommending to disable it ( also like ISPConfig does at this moment).

Compared to cPanel + CloudLinux pricing licenses, a full open source implementation of VPS with ISPConfig + AppArmor would be a great marketing tool, with excellent end-user pricing and outstanding ROI (Return of Investment).

This could become a prominent feature of ISPConfig, together with multiserver.

This only missing feature comparing to CloudLinux would be resource limiting (for cpu,ram,disk I/O), https://centos-webpanel.com/cwppro does it also with cgroups, but let's take it one step at the time ( see https://www.howtoforge.com/community/threads/how-to-limit-cpu-performance-on-per-user-level.81184/#post-384545)

Anyway, this limitation is mitigated by the fact that we could group the end-user clients on more virtual machines, without the constraints of license pricing. A hosting reseller could have not only a shared cpanel reseller account, but a fully managed VPS with ISPConfig and AppArmor.

Securing Shell Users

Restricted ssh shell users is a must have feature for hosting providers to offer to their clients ( for git, ssh tunnel for Xdebug, php composer, specialized cli tools- like drupal drush).

From my point of view, this is the highest priority feature for AppArmor integration with ISPConfig. Also, this was the trigger for me to focus on this issue. I could live with open_basedir and chrooted php-fpm #3220 (closed) ( also we shall see later that AppArmor could do it better, without performance penalty). But, privilege escalation of chrooted shell users is a serious problem.

Chroot Jailkit implementation

Also chrooted ssh shell user has an excellent implementation with Jailkit in ISPconfig, there are some major drawbacks:

• chroot isn't a real security feature

The chroot(2) system call is used by jailkit to put the user inside the jail. Setting up a jail is much easier using the jailkit utilities that doing so 'by hand'. https://olivier.sessink.nl/jailkit/jailkit.8.html

It is not hard to consider the chroot() system call a security feature. In theory, it sounds great, but if you really take the time to understand what is going on, it is not really a security feature, it is closer to what we would call a hardening feature. It might slow down an attacker, but in most situations it is not going to stop them. We are dealing with a situation where calling this a security feature is likely more damaging than not because it creates a false sense of security. It is human nature to let our guard down if we believe we are safe. Using chroot is no safer than not using a chroot. You would be far better off investing your resources into a custom SELinux policy and ensuring your system is properly hardened. Good security has no shortcuts. https://access.redhat.com/blogs/766093/posts/1975883

sed 's/SELinux/AppArmor/g' :-)

• Chroot is breakable

https://web.archive.org/web/20160127150916/http://www.bpfh.net/simes/computing/chroot-break.html

http://pentestmonkey.net/blog/chroot-breakout-perl

• There’s a burden of maintenance of updating all jails, also being space consuming.

Why do I need this?

jk_update will compare the files in a jail with the corresponding files on the real system. If the corresponding file on the real system is newer, and the file on the real system is different, the file from the real system will be copied to the jail including any required libraries just like jk_cp would do. Files that do not exist on the real system will be deleted in the jail.. https://olivier.sessink.nl/jailkit/jk_update.8.html

There’s a feature request, reported as working.

Option to update Jails - A feature allowing admins to update jails from the admin interface using jk_update. #2140 (closed)

chroot for users sucks! It is work to maintain them and all in all you have to do a lot of nasty hacks to get it going! In the meantime AppArmor is in the mainline kernel. It is pretty simple to write rules for it and you can easily update it with tools like ‘logprof’. https://blog.cryptomilk.org/2011/09/02/jailbash/

Implementing AppArmor on top of Jailkit

It means further securing chrooted shell users with AppArmor.

There are some limitations of using AppArmor on top of chroot.

AppArmor can be used to confine chroots but there are currently some limitations, because chroots will share the system profile set unless the profile namespace is changed for the chroot. AppArmor evaluates all paths within a chroot relative to the fs namespace not the chroot. This means that all file rules and profile names need to begin with the chroot base if they are to work with the chroot. See more https://wiki.ubuntu.com/DebuggingApparmor#AppArmor_and_chroot_environments

Find some early architectural discussions here https://lists.ubuntu.com/archives/apparmor/2011-December/001776.html

Some dedicated chroot flags show that AppArmor is able to handle chroot. https://gitlab.com/apparmor/apparmor/-/wikis/AppArmor_Core_Policy_Reference#profile-flags

<relative flag> ::= 'chroot_relative' | 'namespace_relative' 
<attach flag> ::= 'attach_disconnected' | 'no_attach_disconnected' | 'chroot_attach' | 'chroot_no_attach' 

relative flags - The relative flags are mutually exclusive and control how name resolution is handled for the profile. If chroot_relative is used pathnames are looked up relative to the base of the chroot and names outside of the chroot are determined by the attach flags. The namespace_relative flag is similar except that names within a chroot are resolved absolute to the namespace with path names out side of the namespace being controlled by the attach flags.

chroot_attach, and chroot_no_attach are mutually exclusive and control pathname generation when in a chroot and a file is accessed that is exernal to the chroot but within the namespace. With chroot_attach disconnected path names are attached to the root of the filesystem (ie. / prepended), other wise they are left disconnect.

But, due to security limitations of chroot, might prefer a pure AppArmor implementation.

Drop Jailkit chroot for a pure AppArmor implementation

This is my choice.

At first glance, this could be a real challenge, because

AppArmor's security model is to bind access control attributes to programs rather than to users. https://wiki.ubuntu.com/AppArmor

Fortunately, a solution is provided here https://stackoverflow.com/questions/8149791/ssh-user-lock-to-their-home-dir-one-service/8150560#8150560, also confirmed by https://askubuntu.com/questions/144211/restricting-ssh-users-to-certain-access-permissions/144217#144217

The author is Seth Arnold, member of Ubuntu Security Team. https://stackoverflow.com/users/377270/sarnold and https://wiki.kubuntu.org/sarnold

Let’s highlight the relevant parts from the quote:

I'll explain this using the AppArmor system; I've worked on AppArmor for over a decade, and it is the system I know best ...

The easiest way to get started is to copy /bin/bash to /bin/jail_bash (or some other name not in /etc/shells), set the shell for the user in /etc/passwd (chsh(1) can make this easy), and create an AppArmor profile for /bin/jail_bash that allows only the actions you want to allow. If we confine the process correctly, then the user cannot escape the profile we make for them. ...

In one terminal, run aa-genprof jail_bash. In another terminal, log in as the user (or otherwise run /bin/jail_bash) and begin doing tasks that you want to allow the person to do. We'll use what you do as training material to build a profile iteratively. You might be interested to watch /var/log/syslog or /var/log/audit/audit.log (if you have the auditd package installed) to see what operations AppArmor notices your program doing. Don't do too much at once -- just a few new things per iteration. ...

In the aa-genprof terminal, answer the questions as they come up. Allow what needs to be allowed. Deny what ought to be denied. When you are asked about execution privileges, prefer inherit or child over profile. AppArmor is designed to make it easy to grow the profiles on a system over time. AppArmor isn't applicable to all security situations, but we deployed scenarios very similar to this at the DEF CON Capture-the-flag http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.14.4792 hacking contest with excellent results. We had to allow fellow attackers root (and ephemeral user accounts) access to the machine via telnet, as well as POP3, SMTP, HTTP+CGI, and FTP. ...

Be sure to hand-inspect the profiles in /etc/apparmor.d/ before allowing your user to log in. You can fix anything you want with a plain text editor; run /etc/init.d/apparmor restart to reload all profiles (and unload the profiles you might remove). It's handy to keep an unconfined root sash(1) shell open when you're first learning how to configure AppArmor. If you ignore the warning about programs that shouldn't have their own profile, it might be difficult to get back into your own system. (Don't forget about booting with init=/bin/sh in the worst of situations.)

My startup code

I’ll be using ln, instead of copy.

Initial:

cat etc/passwd
defaultaashell:x:5006:5005::/var/www/clients/client0/web3/home/defaultaashell:/bin/bash

root@bob:~# ln /bin/bash /usr/sbin/defaultaashell

Modified shell: defaultaashell:x:5006:5005::/var/www/clients/client0/web3/home/defaultaashell:/usr/sbin/defaultaashell

Login as shell user:

web3@bob:~$ echo "$SHELL"
/usr/sbin/defaultaashell

Generate AA profile, refusing general inclusions (1), going with more granular permssions (4):

root@bob:~# aa-genprof /usr/sbin/defaultaashell

  1 - #include <abstractions/bash>
  2 - #include <abstractions/lxc/container-base>
  3 - #include <abstractions/lxc/start-container>
 [4 - /etc/profile.d/ r,]

Use aa-logprof to further generate the profile, doing operations as web3 in another shell window.

root@bob:~# aa-enforce /etc/apparmor.d/usr.sbin.defaultaashell
Setting /etc/apparmor.d/usr.sbin.defaultaashell to enforce mode.
root@bob:~# systemctl reload apparmor

root@bob:~# cat /etc/apparmor.d/usr.sbin.defaultaashell
# Last Modified: Sat May  9 02:48:54 2020
#include <tunables/global>

/usr/sbin/defaultaashell {
  #include <abstractions/base>

  deny / r,
  deny /bin/cat x,
  deny /usr/bin/mc r,
  deny /usr/bin/mc x,
  deny /usr/bin/top r,
  deny /usr/bin/top x,

  /bin/ls mrix,
  /dev/tty rw,
  /etc/bash.bashrc r,
  /etc/bash_completion.d/ r,
  /etc/bash_completion.d/apport_completion r,
  /etc/bash_completion.d/fail2ban r,
  /etc/bash_completion.d/git-prompt r,
  /etc/bash_completion.d/grub r,
  /etc/group r,
  /etc/init.d/ r,
  /etc/inputrc r,
  /etc/nsswitch.conf r,
  /etc/passwd r,
  /etc/profile r,
  /etc/profile.d/ r,
  /etc/profile.d/*.sh r,
  /lib/x86_64-linux-gnu/ld-*.so mr,
  /usr/bin/groups mrix,
  /usr/bin/locale mrix,
  /usr/bin/locale-check mrix,
  /usr/sbin/defaultaashell mr,
  /usr/share/bash-completion/bash_completion r,
  owner /var/www/clients/client0/web3/home/defaultaashell/ r,
  owner /var/www/clients/client0/web3/home/defaultaashell/.bash_history rw,
  owner /var/www/clients/client0/web3/home/defaultaashell/.profile r,

}

This is dummy code, still needs to figure out how to restrict it to his home directory. Web3 user can change directories, but cannot execute commands in those directories.

UPDATE 21 May 2020: IMPORTANT Quote from Seth Arnold's video https://youtu.be/PRZ59lxLlOY?t=767: !

They can execute anything, that's in bin or usr/bin, they're allowed to use libraries, and they're allowed to do anything in their own home directories to data they own. AppArmor doesn't have yet a way to say " a user can operate only in his home directory ", because the kernel doesn't know where the home directory is. But you can say :" The user has to own the data that they're going to operate on", so they can't read each other's files apparmor_confined_user_home_directory

A good implementation of AppArmor for shell users, would make ISPCONFIG jailkit feature obsolete.

Apparmor profiles for ISPConfig Services

Till says: I guess that Apparmor's default settings are not that strict in Debian 10, at least I have not encountered issues with having it activated yet, that's why it's on. On the ISPConfig side there are no changes implemented. https://www.howtoforge.com/community/threads/ispconfig-3-1-14rc1-available.82307/#post-390435

But, this doesn’t mean that AppArmor, out of the box, is efficient for ISPConfig as a whole. As long as a process doesn’t have his own AppArmor profile, there’s no control access for that process. We would need to define custom security profiles for ISPConfig services (web server, php fpm, mail, shell users etc), as follows:

Apparmor profiles for php fpm

Best online resources are:

• https://nordisch.org/posts/php-fpm-apparmor/
• http://knowledgebasement.com/apparmor-php-fpm-configuration-with-changehat/

My php fpm profile for a single website. How to work with aa-genprof and aa-logprof

• Let’s start generating a profile for php fpm 7.2

root@bob:~# aa-genprof /usr/sbin/php-fpm7.2
Writing updated profile for /usr/sbin/php-fpm7.2.
Setting /usr/sbin/php-fpm7.2 to complain mode.

Before you begin, you may wish to check if a
profile already exists for the application you
wish to confine. See the following wiki page for
more information:
http://wiki.apparmor.net/index.php/Profiles

Profiling: /usr/sbin/php-fpm7.2

Please start the application to be profiled in
another window and exercise its functionality now.

Once completed, select the "Scan" option below in
order to scan the system logs for AppArmor events.

For each AppArmor event, you will be given the
opportunity to choose whether the access should be
allowed or denied.

[(S)can system log for AppArmor events] / (F)inish
Reading log entries from /var/log/syslog.
Updating AppArmor profiles in /etc/apparmor.d.
Complain-mode changes:

Profile:    /usr/sbin/php-fpm7.2
Capability: chown
Severity:   9

 [1 - #include <abstractions/lxc/container-base>]
  2 - #include <abstractions/lxc/start-container>
  3 - capability chown,
(A)llow / [(D)eny] / (I)gnore / Audi(t) / Abo(r)t / (F)inish

Profile:    /usr/sbin/php-fpm7.2
Capability: chown
Severity:   9

  1 - #include <abstractions/lxc/container-base>
  2 - #include <abstractions/lxc/start-container>
 [3 - capability chown,]
(A)llow / [(D)eny] / (I)gnore / Audi(t) / Abo(r)t / (F)inish
Adding capability chown, to profile.

Profile:    /usr/sbin/php-fpm7.2
Capability: dac_override
Severity:   9

 [1 - #include <abstractions/lxc/container-base>]
  2 - #include <abstractions/lxc/start-container>
  3 - capability dac_override,
(A)llow / [(D)eny] / (I)gnore / Audi(t) / Abo(r)t / (F)inish

Profile:    /usr/sbin/php-fpm7.2
Capability: dac_override
Severity:   9

  1 - #include <abstractions/lxc/container-base>
  2 - #include <abstractions/lxc/start-container>
 [3 - capability dac_override,]
(A)llow / [(D)eny] / (I)gnore / Audi(t) / Abo(r)t / (F)inish
Adding capability dac_override, to profile.

etc...

• Don’t forget, in another ssh session : Please start the application to be profiled in another window and exercise its functionality now.
  systemctl restart php7.2-fpm.service

• IMPORTANT: If multiple options are presented to you, usually hit the biggest number, in order to grant the most granular permissions ( in our case capability chown. If you hit 1 #include <abstractions/lxc/container-base>, the wizard will end, because it will abruptly grant too much permissions, more than needed. Please checkout root@bob:~# cat /etc/apparmor.d/abstractions/lxc/container-base

1 - #include <abstractions/lxc/container-base>
2 - #include <abstractions/lxc/start-container>
[3 - capability chown,]

• You could do manual further optimizations: Replace

owner /etc/php/7.2/mods-available/apcu.ini r,
owner /etc/php/7.2/mods-available/apcu_bc.ini r,
owner /etc/php/7.2/mods-available/bz2.ini r,

with

owner /etc/php/7.2/mods-available/** r,

• Don’t take for granted everything aa-genprof of aa-logprof provides to you: For instance, you could eliminate the unsecure net_admin capability

• AppArmor has a nice “learning” mode. After you fool around with your website/ssh session, run aa-logprof.

Running aa-logprof will scan the log file and if there are new AppArmor events that are not covered by the existing profile set, the user will be prompted with suggested modifications to augment the profile. http://manpages.ubuntu.com/manpages/xenial/man8/aa-logprof.8.html

• A nice tutorial, explaining how to do it, step by step, is https://www.inguardians.com/2017/06/08/protecting-the-mr-robot-vuln-hub-machine-part-2-confining-wordpress-with-apparmor/

• A generated profile would look like, but this will work only for web2

root@bob:~# cat /etc/apparmor.d/usr.sbin.php-fpm7.2
# Last Modified: Sun May 10 16:36:36 2020
#include <tunables/global>

/usr/sbin/php-fpm7.2 {
  #include <abstractions/base>
  #include <abstractions/nameservice>
  #include <abstractions/openssl>
  #include <abstractions/php>
  #include <abstractions/ssl_certs>

  capability chown,
  capability kill,
  capability net_admin,
  capability setgid,
  capability setuid,
  capability dac_override,

  /run/php/php7.2-fpm.sock w,
  /var/lib/php7.2-fpm/web2.sock w,
  /var/www/clients/client0/web2/web/** r,
  owner /etc/php/7.2/fpm/php-fpm.conf r,
  owner /etc/php/7.2/fpm/pool.d/web2.conf r,
  owner /etc/php/7.2/fpm/pool.d/www.conf r,
  owner /run/php/php7.2-fpm.pid w,
  owner /run/systemd/notify w,
  owner /var/lib/php7.2-fpm/web2.sock rw,
  owner /var/log/php7.2-fpm.log w,
  owner /var/www/clients/client0/web2/web/** rw,  
}

My php fpm profile for multiple websites. Sample web1 + web2

But, the previous setup isn’t good enough, from a security point of view. Changing /var/www/clients/client0/web2/web/** r, to /var/www/clients/** r, would be suicidal.

In order to confine each web user, we shall be using 2 concepts:

• AppArmor subprofiles

AppArmor supports self directed profile transitions via the change_profile api. Change_profile rules control which permissions for which profiles a confined task can transition to. http://manpages.ubuntu.com/manpages/xenial/man5/apparmor.d.5.html

We can create a subprofile for each php fpm pool.

//web1 and //web2 are subprofiles

root@bob:~# aa-status | grep php
   /usr/sbin/php-fpm7.2
   /usr/sbin/php-fpm7.2//web1
   /usr/sbin/php-fpm7.2//web2
   /usr/sbin/php-fpm7.2 (871)
   /usr/sbin/php-fpm7.2 (1161)
   /usr/sbin/php-fpm7.2 (1162)
   /usr/sbin/php-fpm7.2//web1 (1157)
   /usr/sbin/php-fpm7.2//web1 (1158)
   /usr/sbin/php-fpm7.2//web2 (1159)
   /usr/sbin/php-fpm7.2//web2 (1160)

signal send peer=/usr/sbin/php-fpm7.2//*,
change_profile /usr/sbin/php-fpm7.2//*,

The know-how is from https://nordisch.org/posts/php-fpm-apparmor/, but is confirmed by core AppArmor apache profile

cat /etc/apparmor.d/abstractions/apache2-common
  # Allow unconfined processes to send us signals by default
  signal (receive) peer=unconfined,
  # Allow apache to send us signals by default
  signal (receive) peer=/usr/sbin/apache2,
  # Allow other hats to signal by default
  signal peer=/usr/sbin/apache2//*,
  # Allow us to signal ourselves
  signal peer=@{profile_name},

• Apparmor change_hat functionality for php-fpm

Apparmor also provides functionality for applications to change to a different hat. With this mechanism it is possible to change to a different set of policies in the application. Add functionality to php-fpm to support the additional parameter apparmor_hat in the pool config. Upon spawning a new worker, the worker tries to change to this specific hat. With this feature it is easier for shared hosters to isolate and/or restrict different users. This approach has the advantage over unix access rights, that the apparmor policies allow for a more fine grained control. https://wiki.php.net/rfc/fpm_change_hat

root@bob:/# cat /etc/php/7.2/fpm/pool.d/web2.conf | grep apparmor
apparmor_hat = web2

• Working setup for web1 and web2

root@bob:/# cat /etc/apparmor.d/usr.sbin.php-fpm7.2
# Last Modified: Sun May 10 16:36:36 2020
#include <tunables/global>

/usr/sbin/php-fpm7.2 flags=(attach_disconnected) {
  #include <abstractions/base>
  #include <abstractions/nameservice>
  #include <abstractions/openssl>
  #include <abstractions/php>
  #include <abstractions/ssl_certs>

  capability chown,
  capability kill,
  capability net_admin,
  capability setgid,
  capability setuid,
  capability dac_override,

  /tmp/** rw,  
  /{usr/,}lib{,32,64}/** mr,
  /usr/share/** r,
  /run/php/php7.2-fpm.sock w,
  /var/lib/php7.2-fpm/web*.sock w,
  owner /etc/php/7.2/fpm/php-fpm.conf r,
  owner /etc/php/7.2/fpm/pool.d/web*.conf r,
  owner /etc/php/7.2/fpm/pool.d/www.conf r,
  owner /run/php/php7.2-fpm.pid w,
  owner /run/systemd/notify w,
  owner /var/lib/php7.2-fpm/web*.sock rw,
  owner /var/log/php7.2-fpm.log w,


  # we need write access here to move it into a different apparmor sub profile
  /proc/@{pid}/attr/current rw,

  # no idea why php tries to open / read/write
  deny / rw,

  # allow sending signals to our subprocesses
  signal (send) peer=/usr/sbin/php-fpm7.2//*,

  # allow switching processes to those subprofiles
  change_profile /usr/sbin/php-fpm7.2//*,

  # load all files from this directory
  # store your configurations per pool in this dir
  #include <php-fpm.d>
}

root@bob:/# cat /etc/apparmor.d/php-fpm.d/web1
profile web1 flags=(attach_disconnected) {
  /var/www/clients/client0/web1/web/** r,
  owner /var/www/clients/client0/web1/web/** rw,
}

root@bob:/# cat /etc/apparmor.d/php-fpm.d/web2
profile web2 flags=(attach_disconnected) {
  /run/mysqld/mysqld.sock rw,
  /tmp/** rwk,
  /usr/share/zoneinfo/** r,
  /var/lib/php7.2-fpm/web2.sock r,
  /var/www/clients/client0/web2/web/** r,
  owner /var/www/clients/client0/web2/web/** rw,
}

root@bob:/# cat /etc/php/7.2/fpm/pool.d/web1.conf | grep apparmor
apparmor_hat = web1

root@bob:/# cat /etc/php/7.2/fpm/pool.d/web2.conf | grep apparmor
apparmor_hat = web2

PHP fpm security tests

• File_get_contents test

Running php in web2 site, we shall try to get content from web1 site, and vice versa. Normally, open_basedir would protect against such security flaws.

In order for the test to be relevant, we need to disable the open_basedir directive.

Edit the website record in ISPConfig 3, there go to the options tab and in line "PHP open_basedir" set the value "none". https://ma.juii.net/blog/disable-open-basedir-in-ispconfig

root@bob:~# cat /var/www/clients/client0/web1/web/open_basedir.php
<?php
$output = file_get_contents('/var/www/clients/client0/web2/web/index.html');
print "<pre>$output</pre>";

root@bob:~# cat /var/www/clients/client0/web2/web/open_basedir.php
<?php
$output = file_get_contents('/var/www/clients/client0/web1/web/index.html');
print "<pre>$output</pre>";

The output is Warning: file_get_contents(/var/www/clients/client0/web1/web/index.html): failed to open stream: Permission denied in /var/www/clients/client0/web2/web/open_basedir.php on line 2

• Shell_exec test

Test with the php shell_exec function, where open_basedir is helpless. In a safe environment, this kind of dangerous php function should be disable for ISPConfig websites.

open_basedir is a PHP setting, and as such only has an effect on PHP's functions. When you run an external program that program is not subject to the same limitations as PHP is. https://forums.phpfreaks.com/topic/273091-open_basedir-does-not-apply-to-shell_exec-backticks/

From these security tests, it results that the function “cat” is blocked by apparmor, when running from web2/web1 user. What if the attacker will be using other functions? I guess they will be blocked by default also, since they aren’t whitelisted in apparmor php fpm profile.

root@bob:~# cat /var/www/clients/client0/web1/web/jail.php
<?php
$output = shell_exec('cat /var/www/clients/client0/web2/web/index.html 2>&1');
print "<pre>$output</pre>";

root@bob:~# cat /var/www/clients/client0/web2/web/jail.php
<?php
$output = shell_exec('cat /var/www/clients/client0/web1/web/index.html 2>&1');
print "<pre>$output</pre>";

The output is a blank page.

ToDo AppArmor profiles for php fpm ISPConfig core

It worth exploring this kind of mitigation for ISPConfig core itself, since it prevents disabling any kind of php functions, in order to keep full functionality

cat /usr/local/ispconfig/server/server.sh
cd /usr/local/ispconfig/server
/usr/bin/php -q \
    -d disable_classes= \
    -d disable_functions= \
    -d open_basedir= \
    /usr/local/ispconfig/server/server.php

Drop slow and relatively insecure open_basedir directive, for native kernel performance with AppArmor

AppArmor provides much stronger security for web applications than what is currently available, especially since PHP open_basedir is both slow and known to be insecure. https://plesk.uservoice.com/forums/184549-feature-suggestions/suggestions/6091562-implement-apparmor-support-for-apache

Drop chrooted php-fpm

Also #3220 (closed) Add support for chrooted php-fpm is a nice addition to ISPConfig, AppArmor profiles for php-fpm would make it obsolete.

Php fpm ToDo List

The above php code is raw, without further deep analysis and optimizations of aa-genprof and aa-logprof

• Dac_override Using Dac_override is a lazy workaround, I should find a more granular permission.

A dac_override denial means that the offending process is attempting to access a file with the incorrect unix user/group/world permissions. The proper solution is almost never to grant the dac_override permission. Instead change the unix permissions on the file or process. A few domains such as init, vold, and installd genuinely need the ability to override unix file permissions to access other processes’ files. See Dan Walsh’s blog for a more in-depth explanation. https://source.android.com/security/selinux/device-policy

• Issues restarting php-fpm services

Here, I’m missing some kill permissions in my apparmor profile.

root@bob:~# systemctl status php7.2-fpm.service
● php7.2-fpm.service - The PHP 7.2 FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php7.2-fpm.service; enabled; vendor preset: enabled)
   Active: deactivating (stop-sigterm) since Mon 2020-05-11 03:27:08 EEST; 11s ago
     Docs: man:php-fpm7.2(8)
 Main PID: 813 (php-fpm7.2)
   Status: "Processes active: 0, idle: 4, Requests: 0, slow: 0, Traffic: 0req/sec"
    Tasks: 3 (limit: 4703)
   CGroup: /system.slice/php7.2-fpm.service
           ├─ 813 php-fpm: master process (/etc/php/7.2/fpm/php-fpm.conf)
           ├─1089 php-fpm: pool web2
           └─1091 php-fpm: pool web2

May 11 03:23:38 bob systemd[1]: Starting The PHP 7.2 FastCGI Process Manager...
May 11 03:23:39 bob systemd[1]: Started The PHP 7.2 FastCGI Process Manager.
May 11 03:27:08 bob systemd[1]: php7.2-fpm.service: Failed to kill control group /system.slice/php7.2-fpm.service, ignoring: Permission
May 11 03:27:08 bob systemd[1]: Stopping The PHP 7.2 FastCGI Process Manager...

Apparmor profiles for apache

There’s a dedicated module mod_apparmor - fine-grained AppArmor confinement for Apache http://manpages.ubuntu.com/manpages/bionic/man8/mod_apparmor.8.html

• Setup workflow

  1. https://gitlab.com/apparmor/apparmor/-/wikis/mod_apparmor_example
  2. https://www.pks.mpg.de/~mueller/docs/suse10.1/suselinux-manual_en/manual/bx5dh07.html

• ToDo List

Adding other features such as SSL would require additional access rules https://gitlab.com/apparmor/apparmor/-/wikis/mod_apparmor_example

• Limitation

mod_apparmor() currently only supports apache2, and has only been tested with the prefork MPM configuration -- threaded configurations of Apache may not work correctly. For Apache 2.4 users, you should enable the mpm_prefork module. http://manpages.ubuntu.com/manpages/bionic/man8/mod_apparmor.8.html

Although apache prefork is working fine with php-fpm, personally I’ve encountered some memory issues with php-fpm on certain VPS, so it’s kind of frustrating not being able to use other Apache mpm modes ( worker/event).

The question is if it’s really useful mod_apparmor() or php fpm apparmor profiles would be enough as security mitigation.

• nginx

For nginx, so far I couldn't find an equivalent https://serverfault.com/questions/456448/strict-security-and-virtual-host-isolation-with-nginx

Apparmor profiles for other services

Not tested yet. Out of the box, Ubuntu provides https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/AppArmorProfiles To install other than main profile, run

apt install apparmor-profiles
apt install apparmor-profiles-extra

root@bob:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.4 LTS
Release:        18.04
Codename:       bionic

root@bob:~# dpkg -l | grep apparmor
ii  apparmor                          2.12-4ubuntu5.1                             amd64        user-space parser utility for AppArmor
ii  apparmor-profiles                 2.12-4ubuntu5.1                             all          experimental profiles for AppArmor security policies
ii  apparmor-profiles-extra           1.19                                        all          Extra profiles for AppArmor Security policies
ii  apparmor-utils                    2.12-4ubuntu5.1                             amd64        utilities for controlling AppArmor
ii  libapparmor1:amd64                2.12-4ubuntu5.1                             amd64        changehat AppArmor library
ii  python3-apparmor                  2.12-4ubuntu5.1                             amd64        AppArmor Python3 utility library
ii  python3-libapparmor               2.12-4ubuntu5.1                             amd64        AppArmor library Python3 bindings

There are apparmor profiles for Dovecod, Bind, ClamAV. There’s a general consensus do not confine mysql.

root@bob:~# aa-status | grep dovecot
   /usr/lib/dovecot/anvil
   /usr/lib/dovecot/auth
   /usr/lib/dovecot/config
   /usr/lib/dovecot/deliver
   /usr/lib/dovecot/dict
   /usr/lib/dovecot/dovecot-auth
   /usr/lib/dovecot/dovecot-lda
   /usr/lib/dovecot/dovecot-lda///usr/sbin/sendmail
   /usr/lib/dovecot/imap
   /usr/lib/dovecot/imap-login
   /usr/lib/dovecot/lmtp
   /usr/lib/dovecot/log
   /usr/lib/dovecot/managesieve
   /usr/lib/dovecot/managesieve-login
   /usr/lib/dovecot/pop3
   /usr/lib/dovecot/pop3-login
   /usr/lib/dovecot/ssl-params
   /usr/sbin/dovecot
   /usr/lib/dovecot/anvil (965)
   /usr/lib/dovecot/auth (27232)
   /usr/lib/dovecot/config (975)
   /usr/lib/dovecot/log (966)
   /usr/sbin/dovecot (899)
root@bob:~# aa-status | grep named
   /usr/sbin/named
   /usr/sbin/named (866)
root@bob:~# aa-status | grep clamd
   /usr/sbin/clamd
   /usr/sbin/clamd (930)

Checkout the full list of apparmor processes aa-status_apparmor_extra_profiles-ubuntu_bionic.txt

ISPConfig code implementation

Before diving into ISPConfig code, any tips on how to implement the apparmor plugins, based on the proof of concept codes above, would be highly appreciated.

Following the naming pattern of shelluser_jailkit_plugin.inc.php and php_fpm_jailkit_plugin.inc.php, there should be:

• shelluser_apparmor_plugin.inc.php

app->plugins->registerEvent('shell_user_insert', $this->plugin_name, 'insert');
$app->plugins->registerEvent('shell_user_update', $this->plugin_name, 'update');
$app->plugins->registerEvent('shell_user_delete', $this->plugin_name, 'delete');

• php_fpm_apparmor_plugin.inc.php

$app->plugins->registerEvent('web_domain_insert', $this->plugin_name, 'insert');
$app->plugins->registerEvent('web_domain_update', $this->plugin_name, 'update');
$app->plugins->registerEvent('web_domain_delete', $this->plugin_name, 'delete');

Following these events, the apparmor profiles files should be generated and run apparmor_parser - loads AppArmor profiles into the kernel.

apparmor_hat setting (https://wiki.php.net/rfc/fpm_change_hat) would have a different workflow, requiring core changes to reflect a new site options entry, like Add support for chrooted php-fpm, implements #3220 (closed) !870 (diffs)

root@bob:/# cat /etc/php/7.2/fpm/pool.d/web1.conf | grep apparmor
apparmor_hat = web1  

Apparmor tips

Apparmor installation

apt install apparmor apparmor-utils apparmor-profiles apparmor-profiles-extra auditd snapd

In Ubuntu, the kernel module is enabled by default.

Apparmor debug

Recommended resources:

• https://wiki.ubuntu.com/DebuggingApparmor#Debugging_procedure
• https://doc.opensuse.org/documentation/leap/archive/42.3/security/html/book.security/cha.apparmor.commandline.html#sec.apparmor.commandline.profiling

After profile modification, run systemctl reload apparmor and also restart the affected service ( eg: systemctl restart php7.2-fpm.service)

In the Apparmor profiles for php fpm section, we saw at work aa-genprof and aa-logprof.

Further tips includes analyzing Apparmor Logs

tail -50 /var/log/kern.log | grep "DENIED"

Not only kernel logs, but applications logs could give you important clues

tail -50 /var/log/php7.2-fpm.log

For performance reasons, it might be usefull disable logging, when the rules will be reached a certain level of maturity.

Final thoughts

Acting at Linux Kernel level, AppArmor integration with ISPConfig would bring enhanced security, without any performance penalty. It will replace the slow and relatively insecure open_basedir and exploitable chroot Jailkit shell users, with profiles offering enterprise grade protection.

ISPConfig with AppArmor would play in the commercial league with cPanel and CloudLinux. Even better, it will become a full featured hosting control panel, without license fees-cost effective, and most of all, truly open source, being for competition like Metallica’s Kill 'Em All.

At first glance, the AppArmor implementation looks intimidating, but, after hands on, it looks pretty doable!

A number of default policies are included with AppArmor, and using a combination of advanced static analysis and learning-based tools, AppArmor policies for even very complex applications can be deployed successfully in a matter of hours. https://www.drupal.org/node/2377985

For the moment, I’ve scratched the surface of the code, hoping to be a good start point for other developers to jump in. Personally, I won't be available to pick this up in the next period.