Shell-User fails to connect

Did you ran an ispconfig update with reconfigure services = yes after you installed jailkit 2.21 from backports repo? If not, you overwrote the jailkit config that is set up by ispconfig with an incompatible config file and the creation of jailkit users will fail.

During my various tests, I have certainly also executed an "ispconfig update" and "reconfigure services". However, I can hardly say at which step of the testing.

However, I will verify the proposed procedure once again with a backup machine and will then report.

Debug mode might help you:

https://www.faqforge.com/linux/debugging-ispconfig-3-server-actions-in-case-of-a-failure/

The command jk_addjailuser is not used by ISPConfig.

I have now executed a jailkit upgrade on my test machine.

apt list jailkit -a
Listing... Done
jailkit/unstable,now 2.21-2 amd64 [installed]
jailkit/buster-backports 2.21-2~bpo10+1 amd64

Without performing an "ispconfig update" existing shell users will work as before - a new one created now does not work, if the jail is active.

Then I have done an ISPConfig update (incl. reconfigure services) by executing the following commands:

cd /tmp
wget http://www.ispconfig.org/downloads/ISPConfig-3.1.15p3.tar.gz
tar xvfz ISPConfig-3.1.15p3.tar.gz
cd ispconfig3_install/install
php -q update.php

Neither the last "new" user nor another newly created shell user can log in. In auth.log, the following messages can be found:

...server useradd[25609]: new user: name=xxxx, UID=5008, GID=5005, home=/var/www/clients/client1/web5, shell=/bin/bash
...server usermod[25622]: change user 'xxxx' shell from '/bin/bash' to '/bin/false'
...server usermod[25622]: lock user 'xxxx' password
...server usermod[25690]: unlock user 'xxxx' password

...server sshd[25700]: Accepted publickey for xxxx from 192.168.1.15 port 55047 ssh2: RSA SHA256:0VTDvkBpl2+Y1876bFm8jTmoUJtbDCVARgMW97GxSc0
...server jk_chrootsh[25703]: now entering jail /var/www/clients/client1/web5 for user xxxx (5008) with arguments 
...server jk_chrootsh[25703]: ERROR: failed to execute shell /bin/bash for user xxxx (5008), check the permissions and libraries of /var/www/clients/client1/web5//bin/bash

I noticed the two slashes near the end of the last line (... /var/www/clients/client1/web5//bin/bash)

Does that have any significance at all?

Now again I did a downgrade to version 2.20 of Jailkit by executing the following commands:

cd /tmp
wget http://olivier.sessink.nl/jailkit/jailkit-2.20.tar.gz
tar xvfz jailkit-2.20.tar.gz
cd jailkit-2.20
echo 5 > debian/compat
./debian/rules binary

cd ..
dpkg -i jailkit_2.20-1_*.deb
rm -rf jailkit-2.20*

Without a new ispconfig update I can access the server again with all shell users (one only by deactivating and then reactivating the user first).

I had already activated the debug mode a few days ago – but, like now, I didn't get any message:

# /usr/local/ispconfig/server/server.sh

finished.

I had already activated the debug mode a few days ago – but, like now, I didn't get any message:

According to the shell output you did not enable debug mode (debug log level), check again in ISPConfig UI.

The double slash in the path should not matter.

According to the shell output you did not enable debug mode (debug log level), check again in ISPConfig UI.

Oh, right! I was too fast. Here are the correct messages:

# /usr/local/ispconfig/server/server.sh


24.08.2020-17:22 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
24.08.2020-17:22 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished.

The lock file ".ispconfig_lock" can be found neither in the specified directory nor anywhere else in the system.

The lock file ".ispconfig_lock" can be found neither in the specified directory nor anywhere else in the system. Of course, it's a lock file and lock files get removed when the process finished and that's also what the messages tell you.

Beside that, you did not create a new jailed shell user, that#s why you did not got any additional details for the process of the creation of a shell user.

Please follow these steps:

Create a new website, can be a temporary 'fake' domain name like test.local and then create a jailed shell user. The reason why you have to create a website first is that the jail gets created only once when the first jailed user for a new website gets created.
run server.sh, you should then see the full debug output of the steps to create that jailed shell user.

I should probably do this with version 2.21 of Jailkit, right?

And all steps with active debug mode!

Yes, with the new version please. And beside debug.log, please check the line for this new user in /etc/passwd and in the /var/www/yourdomain.tld/etc/passwd file.

The following steps were made:

activated debug mode in ISPConfig UI
commented out the server.sh cron job
created new website
created shell user
had run command: /usr/local/ispconfig/server/server.sh

There are now two new users in the passwd file:

web15:x:5014:5005::/var/www/clients/client1/web15/./home/web15:/usr/sbin/jk_chrootsh
dzdebug:x:5014:5005::/var/www/clients/client1/web15/./home/dzdebug:/usr/sbin/jk_chrootsh

The attached file ispconfig.log.txt comes from /var/log/ispconfig/

The attached file debug.txt is taken from terminal output.

The new shell user can't connect:

In auth.log, the following messages are found:

Aug 24 20:47:45 aserver sshd[11152]: Accepted publickey for dzdebug from 192.168.1.15 port 57546 ssh2: RSA SHA256:0CNJngDWNvTQeLrOFMxbzCx2LJ12u7lYqyTfeQm3fSo
Aug 24 20:47:46 aserver jk_chrootsh[11155]: now entering jail /var/www/clients/client1/web15 for user dzdebug (5014) with arguments 
Aug 24 20:47:46 aserver jk_chrootsh[11155]: ERROR: failed to execute shell /bin/bash for user dzdebug (5014), check the permissions and libraries of /var/www/clients/client1/web15//bin/bash

Thanks, the user in /etc/password are looking ok. and what's in /var/www/yourdomain.tld/etc/passwd file? And please post the output of:

ls -la /var/www/clients/client1/web15/bin/bash ls -la /var/www/clients/client1/web15/

The reason for the error is probably this:

configparser.DuplicateOptionError: While reading from '/etc/jailkit/jk_init.ini' [line 115]: option 'includesections' in section 'openvpn' already exists

please check the ini file to ensure that includesections is used only once in openvpn section. You can combine both includesections lines.

I just checked the current stable dev version and it exists there just once, so the reported issue is probably solved already in 3.1dev version.

Yeah, that's probably it, both 3.1.15 and the debian jk_init.ini have this problem.

There was also a fix in 3.1dev for newly created jailkit users to do with an empty homedir in the jailkit passwd file I believe, but not 100% sure of the details, nor if it's related here.

A quick grep shows jk_addjailuser is not used, so its absence should not be a problem.

One thing that stands out as a significant error are the debug lines like:

24.08.2020-19:32 - DEBUG - safe_exec cmd: jk_cp -k '/var/www/clients/client1/web15' '/usr/bin/lesspipe' - return code: 0

Note that jk_cp requires a -j in front of the jail name, and should not at all use -k, that is a significant security problem. (Using hard links allows an easy escape from the chroot jail if/once root access within has been achieved.) @dziller, can you check if you have hardlinks = 1 set in your jk_update.ini? Perhaps that is something left behind by the backports package causing this behavior, and not the ISPConfig code.

Looks like the jk -k issue is indeed ISPConfig code, in server/lib/classes/system.inc.php and server/scripts/create_jailkit_programs.sh. This would only affect programs installed with jk_cp, not entire sections from jk_init.ini, so it failing might go unnoticed for a while. This is a different issue than what causes @dziller's problem though.

Doing a bit of testing, this is not failing, the jail is determined correctly and the binary attempted to be hardlinked, and failing that will be copied. (-j is optional if the jail is the first argument (after options, apparently)).

So on my system a test jk_cp resulted in a copy - but with a different filesystem layout it would have been a hardlink, which is bad.

root@host:~# ls -l /var/www/clients/client*/web*/bin/chown
ls: cannot access '/var/www/clients/client*/web*/bin/chown': No such file or directory
root@host:~# jk_cp -k /var/www/clients/client##/web## /bin/chown
Trying to link /bin/chown to /var/www/clients/client##/web##/bin/chown
Linking /bin/chown to /var/www/clients/client##/web##/bin/chown failed, will revert to copying
root@host:~# ls -l /var/www/clients/client*/web*/bin/chown
-rwxr-xr-x 1 root root 72512 Feb 28  2019 /var/www/clients/client##/web##/bin/chown
root@host:~# ls -li /var/www/clients/client##/web##/bin/chown /bin/chown
 273618 -rwxr-xr-x 1 root root 72512 Feb 28  2019 /bin/chown
5116699 -rwxr-xr-x 1 root root 72512 Feb 28  2019 /var/www/clients/client##/web##/bin/chown

Maybe a quick cleanup routine in the installer should be looked at? find(1) can find them (see eg. this), or stat files and compare inode numbers.

mentioned in issue #5710 (closed)

!1118 (merged) is a start, with no cleanup for existing jails. Also note that when digging further, jk_init also had -k specified, so forget my earlier claim that only jk_cp programs would be affected.

Perhaps relevant to your problem, @dziller, is the line:

24.08.2020-19:32 - DEBUG - chmod failed: /var/www/clients/client1/web15/bin : 493

I just created a new site + jailkit user on a similar vintage system (it was 3.1dev from Dec 9, 2019, which basically 3.1.15p3, sans an unrelated a bugfix), and did not have a "chmod failed" line. What owner and permissions do you have for that directory? I see:

root@host-web-1:~# ls -ld /var/www/clients/client##/web##/bin
drwxr-xr-x 2 root root 4096 Aug 24 15:46 /var/www/clients/client##/web##/bin

Nevermind, @till caught the real issue (bad jk_init.ini) earlier, this is just a symptom.

@tbrehm

Thanks, the user in /etc/password are looking ok. and what's in /var/www/yourdomain.tld/etc/passwd file? And please post the output of:

That passwd file (/client1/web15/...) is empty!

ls -la /var/www/clients/client1/web15/bin/bash

ls: cannot access '/var/www/clients/client1/web15/bin/bash': No such file or directory

ls -la /var/www/clients/client1/web15/

total 56
drwxr-xr-x 14 root  root    4096 Aug 24 19:32 .
drwxr-xr-x 14 root  root    4096 Aug 24 19:32 ..
-rwxr-x---  1 web15 client1    0 Aug 24 19:32 .bash_history
-rw-r--r--  1 web15 client1    0 Aug 24 19:32 .profile
drwx------  2 web15 client1 4096 Aug 24 19:32 .ssh
lrwxrwxrwx  1 root  root       7 Aug 24 19:32 bin -> usr/bin
drwxr-xr-x  2 web15 client1 4096 Aug 24 19:32 cgi-bin
drwxr-xr-x  3 root  root    4096 Aug 24 19:32 etc
drwxr-xr-x  4 root  root    4096 Aug 24 19:32 home
lrwxrwxrwx  1 root  root       7 Aug 24 19:32 lib -> usr/lib
lrwxrwxrwx  1 root  root       9 Aug 24 19:32 lib64 -> usr/lib64
drwxr-xr-x  2 root  root    4096 Aug 25 00:02 log
drwx--x---  2 web15 client1 4096 Aug 24 19:32 private
drwxr-xr-x  2 root  root    4096 Aug 24 19:32 ssl
drwxrwxrwx  2 web15 client1 4096 Aug 24 19:32 tmp
drwxr-xr-x  5 root  root    4096 Aug 24 19:32 usr
drwxr-xr-x  3 root  root    4096 Aug 24 19:32 var
drwx--x--x  4 web15 client1 4096 Aug 24 19:32 web
drwx--x---  2 web15 client1 4096 Aug 24 19:32 webdav

jk_init.ini file

Just for information I have uploaded my jk_init.ini file. However, caused by the multiple up- and downgrades there are 2 more files which were created automatically.

jk_init.ini.220

jk_init.ini.dpkg-dist

I noticed that the lines 78+79 are not present in the other two (backup?) files.

Especially line 79 (directories = /usr/lib/locale/en_US.utf8) refers to a non-existent file:

ls -la /usr/lib/locale
total 2904
drwxr-xr-x  3 root root    4096 May  6  2019 .
drwxr-xr-x 79 root root    4096 Aug  2 06:01 ..
drwxr-xr-x  3 root root    4096 May  6  2019 C.UTF-8
-rw-r--r--  1 root root 3031712 May  6  2019 locale-archive

With this test machine I played with locale at the beginning of the installation.
Currently only de_DE.UTF-8 UTF8 is active in /etc/locale.gen.

'includesections' in section 'openvpn'

line 113: includesections = netbasics
line 115: includesections = netbasics, uidbasics
I will manually disable line 113.

In the file jk_init.ini.dpkg-dist (timestamp from Nov 11 2019), the first line listed is commented out (#includesections = netbasics).

@jnorell

24.08.2020-19:32 - DEBUG - chmod failed: /var/www/clients/client1/web15/bin : 493

This directory is a symlink to usr/bin/
ls -ld /var/www/clients/client1/web15/bin
lrwxrwxrwx 1 root root 7 Aug 24 19:32 /var/www/clients/client1/web15/bin -> usr/bin

/var/www/clients/client1/web15# ls -ld usr/bin
drwxr-xr-x 2 root root 4096 Aug 24 19:32 usr/bin

However, the following output is quite confusing:

/var/www/clients/client1/web15# ls -la usr/bin
total 12512
drwxr-xr-x 2 root  root       4096 Aug 24 19:32 .
drwxr-xr-x 5 root  root       4096 Aug 24 19:32 ..
-rwxr-xr-x 8 web13 client1   39520 Feb 28  2019 basename
-rwxr-xr-x 8 web13 client1   47784 Feb 28  2019 dircolors
-rwxr-xr-x 8 web13 client1   35424 Feb 28  2019 dirname
-rwxr-xr-x 2 root  root    3196368 Jul 27 20:02 git
lrwxrwxrwx 1 root  root          3 Aug 24 19:32 git-receive-pack -> git
lrwxrwxrwx 1 root  root          3 Aug 24 19:32 git-upload-pack -> git
-rwxr-xr-x 8 web13 client1   39584 Feb 28  2019 groups
-rwxr-xr-x 8 web13 client1   43808 Feb 28  2019 id
-rwxr-xr-x 8 web13 client1    8564 May  7  2018 lesspipe
-rwxr-xr-x 2 root  root    4097240 Jul  4 14:31 mysql
-rwxr-xr-x 2 root  root    3921984 Jul  4 14:31 mysqldump
-rwxrwxrwx 8 web13 client1  246160 Jun 12  2019 nano
-rwxr-xr-x 5 web13 client1  187840 Jul 26  2019 patch
lrwxrwxrwx 1 root  root         22 Aug 24 19:32 pico -> /etc/alternatives/pico
-rwxr-xr-x 8 web13 client1   68416 Feb 28  2019 rm
-rwxr-xr-x 8 web13 client1  445560 Apr 23  2019 tar
-rwxr-xr-x 6 web13 client1  183136 Jul 30  2019 unzip
-rwxr-xr-x 8 web13 client1  213136 Aug 16  2015 zip

Here should actually appear "web15" and not "web13".
"web13" was the website, I had made my tests with, until yesterday.

If this is a serious problem, I would first have to clone another machine to verify it.

Shell-User fails to connect

short description

correct behavior

environment

Reproducible action

workaround

possible cause

proposed fix

Designs

Child items ...

Activity