Tag: linux

Blog entries related to Linux

OK, this is driving me crazy.  ISPConfig is a great piece of software.  It can manage your web sites, and related services (such as DNS, FTP, mail, etc.) well, and it handles the config files with pretty good care, and it rarely breaks.  But, the installation experience is so twisted that I can’t bear.  And if I would only have to tolerate this at fresh installs, then I’d just try to live with it, but facing the same old problems once more each time I try to update the installation is way beyond acceptable.

After you download their latest release, and untar it in, let’s say, /root, it creates a folder named install_ispconfig and then you’re just supposed to cd into that directory and issue a ./setup command.  Then the setup application asks you some questions about your system’s config1 (giving you a choice for an expert setup mode, in which it asks you all kinds of questions about all of the services installed on your system) and then it starts compiling Apache/PHP/OpenSSL/etc.  And if you’re doing an upgrade, you only have to choose your language and agree to their licensing terms, no further questions appear on the screen.  That sounds easy, right?

Wrong.  The installer is written with the assumption that everything will work right the first time.  If you answer incorrectly on any of those questions, or if something interrupts the installation process (be it as simple as an SSH session being disconnected, assuming you’re not using screen), then what are you supposed to do?  Just re-run ./setup and hope it picks up from where it left last time, right?

Wrong.  You’re supposed to delete the whole install_ispconfig directory2, and start from scratch!  Now, that doesn’t sound like a big deal?  Consider this.  I was updating ISPConfig on a server today, and nearly at the end of the install session, I got a number of weird shell error messages whining about unrecognized commands, and an Enter Password prompt.   Obviously I didn’t know what password to enter, so I tried the first password coming to my mind, the ISPConfig administrative password.  But it turned out that I was wrong.  Checking out the error messages, I noticed it should be the MySQL password.  Turns out that the ISPConfig installer tries to invoke the mysql utility with the password specified on the command line, and special characters in the password could confuse the shell, causing those error messages.  I found this after retrying the whole setup process (including half an hour to rebuild all the packages from the source) four times in a row.

You’d think that an installer should be able to handle such an error along the way?  You’d think that it should be able to use the results of those steps which were completed successfully in later runs?  You’d think that the compilation and configuration steps could be separated in the installer so that failing one does not abort the other?  If you’re working with the ISPConfig installer, you’d be wrong.  Come on, guys, you could do better.

1 It should be noted that strangely enough, it never asks you where you want ISPConfig to be installed.  It uses /root/ispconfig and /home/admispconfig, and you don’t get a say on that.  I still remember the first time I installed ISPConfig: I renamed the extracted directory to /root/ispconfig, and ran the setup from there, and wasted hours before I figured out that it’s the name of the directory which causes the installer to fail…

2 Actually the newer versions of their installer seems to delete that directory itself when something goes wrong, and doesn’t care to print a single message on the console.  Imagine how surprised you’d be when the installer quits, and you ls inside the install_ispconfig folder to see that the directory is empty!  And if you cd up and ls again, you’ll see there’s no install_ispconfig directory at all!  Hilarious!

Tagged with: ,

Today I had this problem where Fail2ban was keeping on blacklisting an IP address, even though it was in the ignoreip list in /etc/fail2ban/jail.conf.  After double-checking everything on the server, and googling desperately, I found out that up to version 0.8.2, Fail2ban had a bug which caused only the first IP in the ignoreip list to take effect.  And guess what?  Ubuntu versions before gutsy have older versions of Fail2ban.  After a bit of digging, I found out the patch which had fixed the problem in 0.8.2, and I decided to patch my local Fail2ban installation.

In order to do this, you should edit /usr/share/fail2ban/server/filter.py and apply the following patch:

--- filter.py.orig 2008-05-21 02:49:22.000000000 -0500 +++ filter.py 2008-05-21 02:50:12.000000000 -0500 @@ -299,7 +299,7 @@ for i in self.__ignoreIpList: # An empty string is always false if i == "": - return False + continue s = i.split('/', 1) # IP address without CIDR mask if len(s) == 1: @@ -314,7 +314,7 @@ if ip in ips: return True else: - return False + continue if a == b: return True return False

Then, you should restart Fail2ban:

/etc/init.d/fail2ban restart

And it will pick up the fix and process the ignoreip correctly.

Tagged with: ,

If you happen to run Ubuntu Server (at least, up to version 7.04) and use the Courier POP3 or IMAP server, and you’re watching your log files closely, you’ve seen error messages such as the below one in /var/log/syslog:

May 21 01:14:10 server4 authdaemond: PAM [dlerror: /lib/security/pam_foreground.so: undefined symbol: pam_set_data]

Many people have observed this problem.  This happens because libpam-foreground.so has not been compiled with the right options.  This library needs to be linked against libpam.so and glibc.so, but it isn’t).  So, why is that this error only manifests itself in Courier login attempts, and not all the other applications and services which use PAM?  This depends on the order that each application loads its shared objects.  If the application first loads both glibc and libpam, and then tries to load libpam-foreground, then great!  Otherwise (as is the case with Courier) an error such as the above happens.

This is just a nuisance, because in non-interactive sessions, libpam-foreground doesn’t do anything useful after all, but here’s how to fix it.  You should just run the below commands in a login session.  You don’t need to be root (I use sudo where necessary).

cd ~ sudo apt-get install build-essential fakeroot devscripts mkdir libpam-foreground cd libpam-foreground wget http://launchpadlibrarian.net/8187075/libpam-foreground_0.3-0ubuntu1.debdiff apt-get source libpam-foreground sudo apt-get build-dep libpam-foreground cd libpam-foreground-0.3 patch -p4 < ../libpam-foreground_0.3-0ubuntu1.debdiff debuild -uc -us cd .. sudo dpkg -i libpam-foreground_0.3-0ubuntu1_i386.deb

The above commands download the source package for libpam-foreground, apply a patch to correct the compilation options, re-compile the package and install the resulting deb file.  No more annoying log messages from libpam-foreground!

Tagged with: ,

Server monitoring is something server admins put off until, well, it’s too late.  Server admins have all experienced clients calling in to let them know that a service on their server is not working.  This can be unfortunate and embarrassing at the very least.  I thought I’d document a number of utilities I use to monitor Linux servers.

  • Snort.  Snort is a NIDS and NIPS, and one of the most famous network security tools.  As a server admin, you should already know Snort.  I usually use BASE alongside Snort on my servers.
  • OSSEC.  OSSEC is a HIDS, which can perform a number of useful monitoring tasks, such as rootkit detection, automated log analysis, and file integrity checking.  In addition, it provides instant alerting and active response (which means that it can take some action automatically as soon as it faces a problematic situation).  A must-have tool in your toolbox, IMO.
  • Fail2ban.  Fail2ban is a small utility which does one thing, and does it good.  It monitors the system authentication logs, and detects password guessing attempts by counting the number of failed authentications, and bans the IPs of the offenders using the kernel firewall.
  • Monit.  Monit is a tool for managing services, processes, devices, files, etc.  Monit allows full customization by accepting flexible rules on what to monitor, and what to do in case something goes wrong.  It goes beyond simple stuff such as monitoring a port and pid file, and can for example send you an alert when a filesystem is about to run out of space, stop a service when it consumes too much CPU, or monitor an external server.
  • Munin.  Munin is an excellent utility which can collect data on just about anything going on in the server using RRDtool, and generate graphs for both online and offline viewing.  A few examples of the graphs it generates include network interface incoming/outgoing traffic, inode usage, swap usage, memory usage, and filesystem usage.  It also includes facilities to generate service-specific graphs, such as MySQL query count and Postfix mail queue size.

Tagged with: ,