HEP System Administration

Now using iptables as a firewall on the server. This server needs to allow the following services:

ssh – from anywhere
smtp/submission – from anywhere
http/https – from anywhere
imaps/pop3s – from anywhere
nfs – from our subnet only
mysql (though this will soon be turned off) – from localhost only

The one problem that we have is that some parts of nfs are randomly assigned port numbers. I needed to set these parts to a specific port and then allow that port through the firewall.

Edit /etc/sysconfig/nfs

MOUNTD_PORT="10004"
RQUOTAD_PORT="10005"
STATD_OUTGOING_PORT="10003"
STATD_PORT="10002"

Edit /etc/modprobe.conf and add this line

# Set lockd to a port for iptables
options lockd nlm_tcpport=10000 nlm_udpport=10001

Create /etc/sysconfig/iptables with the following:

# 29 May 2007 by MH
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [120351:14706650]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT 
-A FORWARD -j RH-Firewall-1-INPUT 
-A RH-Firewall-1-INPUT -i lo -j ACCEPT

-A RH-Firewall-1-INPUT -p icmp -j ACCEPT
-A RH-Firewall-1-INPUT -p ipv6-crypt -j ACCEPT 
-A RH-Firewall-1-INPUT -p ipv6-auth -j ACCEPT 

# Allow ssh logins from anyplace
-A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

# Allow secure imap from anyplace 
-A RH-Firewall-1-INPUT -p tcp --dport 993 -j ACCEPT

# Allow secure pop from anyplace 
-A RH-Firewall-1-INPUT -p tcp --dport 995 -j ACCEPT

# Allow smtp and submission from anyplace 
-A RH-Firewall-1-INPUT -p tcp --dport 25 -j ACCEPT 
-A RH-Firewall-1-INPUT -p tcp --dport 587 -j ACCEPT

# Allow http and https from anyplace, https is for webmail 
-A RH-Firewall-1-INPUT -p tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp --dport 443 -j ACCEPT

# Allow portmap (111), ,rquotad(10005), mountd(10004), statd(10002 & 10003), nfsd(2049) and 
# lockd(10000 & 10001) from our subnet (for nfs)
# rquotad, mountd and statd are set in /etc/sysconfig/iptables
# lockd is set in /etc/modprobe.conf
-A RH-Firewall-1-INPUT -s 10.135.102.0/255.255.255.0 -p tcp -m tcp --dport 111 -j ACCEPT
-A RH-Firewall-1-INPUT -s 10.135.102.0/255.255.255.0 -p udp -m udp --dport 111 -j ACCEPT

-A RH-Firewall-1-INPUT -s 10.135.102.0/255.255.255.0 -p tcp -m tcp --dport 2049 -j ACCEPT
-A RH-Firewall-1-INPUT -s 10.135.102.0/255.255.255.0 -p udp -m udp --dport 2049 -j ACCEPT

-A RH-Firewall-1-INPUT -s 10.135.102.0/255.255.255.0 -p tcp -m tcp --dport 10000:10005 -j ACCEPT
-A RH-Firewall-1-INPUT -s 10.135.102.0/255.255.255.0 -p udp -m udp --dport 10000:10005 -j ACCEPT

# Allow localhost to use unsecured imap on 143 (this is for squirrelmail)
-A RH-Firewall-1-INPUT -s 127.0.0.1 -p tcp --dport 143  -j ACCEPT

# Allow localhost to use mysql (3306)
-A RH-Firewall-1-INPUT -s 127.0.0.1 -p tcp --dport 3306 -j ACCEPT
-A RH-Firewall-1-INPUT -s 127.0.0.1 -p udp --dport 3306 -j ACCEPT

-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited 
COMMIT

Then I reboot the machine to have the settings take effect.

I took Marty’s old thermal_shutdown script and rewrote it in perl, so that I could use my ups shutdown script with it. I got the stuff for my shutdown script somewhere on the web, but I can’t find it anymore to cite it. So, my apologies to the original author.

#!/usr/bin/perl

# thermal_shutdown - Script to check temperature in glass room
#     If temp too high, email gurus, shutdown computer and shutdown ups
#
#       8 May 2007
#
#       Changed Marty's thermal_shutdown script to perl and added part about ups shutdown
#
#       The perl module Device::SerialPort must be installed.  I installed the rpm
#       perl-Device-SerialPort-1.002-1.2.el4.rf.i386.rpm.  A copy is in 
#       /support/data1/kickstart/additions.

use Device::SerialPort;  # Need to communicate with the serial port ups connection

# Some definitions
$PORT="/dev/ttyS0"; # Port with ups cable
$FILE="/system/monitors/therm_209a.data"; # File with temperature data
$LIMIT=80.0; # Temperature above which everything gets shut down
chop($THISHOST=`hostname -s`); # Hostname of this computer
chop($CURRENT=`cat $FILE`); # Current temperature
chop($DATE=`date`); # Current date

# This entire script only runs if the current temp is greater than the limit

if($CURRENT > $LIMIT)
{
        #################################################################
        # MAIL SECTION:  Send mail to the gurus list with the information
        #################################################################
        my($sendmail) = "/usr/sbin/sendmail";
        my($subject) = "TEMPERATURE SHUTDOWN: $THISHOST";
        my($mailto) = "chiefs@blog";
        my($mailfrom) = "root@$THISHOST";
        my($message) = "                     
    *****  THERMAL SHUTDOWN   *****
    
        Computer $THISHOST and its ups are now being shutdown to protect itself from damage.

        The temperature in the glass room has exceeded $LIMIT degrees F.

        The temperature is $CURRENT degrees F on $DATE.

        Please call Physical Plant to have someone come out to check out the system.  If no 
        one responds in 15 minutes, call again, or call Jim.

        The number for Physical Plant is 123-555-1414.

        Jim's phone number:  123-555-7824.


";



        # Open a stream to mail and send everything
        open(MAIL,"|$sendmail -oi -t");
        print MAIL "From: $mailfrom\n";
        print MAIL "To: $mailto\n";
        print MAIL "Subject: $subject\n\n";
        print MAIL "$message\n";
        close(MAIL);

        ##########################################################################
        # SHUTDOWN SECTION:  Send signal to shutdown the ups and then the computer
        ##########################################################################
        # Connection Settings
        $ob = new Device::SerialPort ($PORT) || die "Can't open $PORT: $!\n";
        $ob->baudrate(2400);
        $ob->parity("none");
        $ob->databits(8);

        # Send Y to put the ups in smart mode
        $pass=$ob->write("Y");

        # Send two Ks with > 1.5s delay between to shut down
        $pass=$ob->write("K");
        sleep 2;
        $pass=$ob->write("K");

        undef $ob;

        # Now shutdown the computer.  Hopefully, it'll be shut down before the ups goes off
       `/sbin/shutdown -h -t0 now`;
}

To prepare for the next time that our glass room overheats, we want to run a script that automatically shuts off the ups anytime the temperature rises above a certain level. We already have the temperature monitor set up, so now we just need to figure out what to do to send the ups the signal to shut itself off.

I’ve installed the apcupsd program on one of our computers and connected it to its ups with the provided serial cable. I’m guessing that I don’t really need apcupsd running, but for now I have it. The rpm I had complained that it wouldn’t install without libcrypto.so.4 and libnetsnmp.so.5. We have libcrypto.so.6 and libnetsnmp.so.10 installed. So I just made links from these newer libraries to the older names.

cd /lib
ln -s libcrypto.so.6 libcrypto.so.4
cd /usr/lib
ln -s libnetsnmp.so.10 libnetsnmp.so.5

I removed the apcupsd program because I didn’t need it.

With the ups connected to the serial port, I can use kermit to talk to it.

kermit

c-kermit> set line /dev/ttyS0

c-kermit> set speed 2400

c-kermit> c

				

Our ntp settings got screwed up (though it’s more accurate to say that I screwed up our ntp settings.) To determine what the current settings are:

$ ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+ns1.uchicago.ed darkcity.cerias  2 u    4   64  377    3.372   -2.833   1.466
*ns4.uchicago.ed tick.uh.edu      2 u  123  128  377    0.389   -4.102   0.163
+ns5.uchicago.ed tick.uh.edu      2 u    -   64  377    0.459   -4.140   0.082

Note that if above it shows 0.0.0.0 as the refid, the system is NOT communicating with the ntp server.

To prepare for websites using php and databasese, mysql-server was installed. Upon starting with /etc/rc.d/init.d/mysqld start, got an error message about group mysql being invalid. Turns out that we didn’t have a mysql group. Normally, the group mysql would get GID 27. However, that GID was already in use for us. So, mysql was given GID 241 because that was the GID of a directory created from the installation.

/etc/rc.d/init.d/mysqld start

mysqladmin -u root password 'new_password'

mysql -u root -p
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| test               |
+--------------------+
3 rows in set (0.00 sec)

mysql> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

mysql> select Host,User,Password from user where user='';
+-----------+------+----------+
| Host      | User | Password |
+-----------+------+----------+
| comp      |      |          |
| localhost |      |          |
+-----------+------+----------+
2 rows in set (0.00 sec)

mysql> select Host,User,Password from user where Password='';
+-----------+------+----------+
| Host      | User | Password |
+-----------+------+----------+
| comp      | root |          |
| comp      |      |          |
| localhost |      |          |
+-----------+------+----------+
3 rows in set (0.00 sec)

Now need to delete those empty users.

mysql> delete from user where User='';
Query OK, 2 rows affected (0.00 sec)

mysql> select Host,User,Password from user where Password='';
+------+------+----------+
| Host | User | Password |
+------+------+----------+
| comp | root |          |
+------+------+----------+
1 row in set (0.00 sec)

mysql> delete from user where Password='';
Query OK, 1 row affected (0.00 sec)

mysql> select Host,User,Password from user;
+-----------+------+------------------+
| Host      | User | Password         |
+-----------+------+------------------+
| localhost | root | 1e3392c069a69a58 |
+-----------+------+------------------+
1 row in set (0.00 sec)

So we are now left with one user (root) on the localhost with a password. New users can be created to take care of different databases.

Currently we have the backup machine set with quotas as shown here:

setquota -u user 100000000 125000000 100000 150000 /local/ls1

When attempting to start httpd on the new server, kept getting the message:

hep1:init.d$ ./httpd start
Starting httpd: execvp: Permission denied

This appears to be selinux blocking the program. So, I changed /etc/sysconfig/selinux to disabled instead of enforcing. I tried permissive for a while, but things still didn’t work and I didn’t want this to be the problem.

I wrote a couple of perl scripts that run on each machine and find out how many files each user has on each data disk. The way this works is that there are two scripts that run on each machine. First is gen_disk_listing, which is just a find command that generates a list of all the files on the data disks. Second is find_data_disk_hogs which separates the files by owner. The last script is run on cdf which creates a webpage from all the data gathered.

The networking problems were caused by a setting that I didn’t make after adding the new storage units. Since the storage units have two ethernet adapters, I put one on the campus 10 subnet and one on my own 192.168 subnet that was going to be set up for gb speeds. The idea was that most of the data transfer would take place on the 192.168 subnet, keeping it off the campus network. Unfortunately, I was unable to get the gb nics working. I decided to let the cdf users use the storage units through the 10 subnet, while I continued to work on the gb network stuff. The problem is that, by default, data on the 128 will go to the switch and then come back on the 10. This basically overwhelmed the switch, causing all our problems. Ron at Network Services told me to add a route to the 10, so that the step of going to the switch would be eliminated. So, I added the following:

route add -net 10.135.102.0 netmask 255.255.255.0 eth0

So, now our route table looks like this:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
128.135.102.0 * 255.255.255.0 U 0 0 0 eth0
10.135.102.0 * 255.255.255.0 U 0 0 0 eth0
169.254.0.0 * 255.255.0.0 U 0 0 0 eth0
default v102router.uchi 0.0.0.0 UG 0 0 0 eth0

To make the route permanent, do the following:

on SLF305, make /etc/sysconfig/network-scripts/route-eth0 (permissions 755):
10.135.102.0/24 dev eth0

The switch in the glass room was replaced this morning by network services. I’ll be keeping an eye on things to see if this fixes our problems.