Archive for the ‘Server’ Category

Installed a new server with RHEL6 and was trying to get the synopsys license working on it. I kept getting this error,

[maryh@eshop bin]$ ./lmgrd
./lmgrd: Command not found.

even though that command was present in that directory.

The problem was I was missing some 32-bit packages on the system. I installed:

# yum install redhat-lsb-core.i686

which installed a bunch of packages. After that, everything worked as expected.

We needed to upgrade our current trac system to RHEL6. Here’s basically how I did it.

-Install RHEL6 Workstation

-Additional packages we were going to need: mod_ssl, mod_wsgi, ImageMagick, graphviz, libpng-static

[ ~]# easy_install Trac
[ web]# pwd
/nfs/net/web
[ web]# cp -Rp lappd-trac lappd-trac-rhel6
[ conf]# tracd --port 8000 /net/web/lappd-trac-rhel6/LAPPD

Opening a browser gave this error:

(The Trac Environment needs to be upgraded. Run “trac-admin /net/web/lappd-trac-rhel6/LAPPD upgrade”)

[ conf]# trac-admin /net/web/lappd-trac-rhel6/LAPPD upgrade
[ conf]# trac-admin /net/web/lappd-trac-rhel6/LAPPD wiki upgrade

Restarting the test on port 8000, works, but got errors:

/nfs/net/web/lappd-trac-rhel6/LAPPD/plugins/graphviz-0.7.5-py2.6.egg/graphviz/graphviz.py:23: DeprecationWarning: the sha module is deprecated; use the hashlib module instead

[ plugins]# easy_install hashlib

Downloaded the graphviz source and unzipped, then ran:

[ 0.11-0.7.5]$ python setup.py bdist_egg
[ 0.11-0.7.5]$ cd dist
[ dist]$ easy_install graphviz-0.7.5dev-py2.6.egg

Other plugins that we’d need.

[ site-packages]# easy_install https://trac-hacks.org/svn/datefieldplugin/0.11/
[ site-packages]# easy_install -U -Z http://trac-hacks.org/svn/customfieldadminplugin/0.11
[ site-packages]# easy_install http://trac-hacks.org/svn/iniadminplugin/0.11
[ site-packages]# easy_install http://trac-hacks.org/svn/citemacro/0.11

We also had been using the TracMath plugin, I installed it, but as soon as it was activated, our system crashed. I looked around to see if we were perhaps missing a required package, but couldn’t find anything. So, I have disabled this plugin.

If we attempt to install this plugin, when we try to upgrade the database, we get this error:

[~]# trac-admin /net/web/lappd-trac/LAPPD upgrade
TypeError: __init__() got an unexpected keyword argument 'unique'

To fix, it depends on where the plugin was installed. For the system, delete it from /usr/local/lib/python2.6/site-packages. For the project, it will be in PROJECT_HOME/plugins. After deleting, restart the webserver.

How to authenticate against and openldap server, in Apache.

Since we’re sending credentials, we want everything to be over https, so everything here is in the /etc/httpd/conf.d/ssl.conf file. I’m calling the openldap system, liberty.example.com and the system running the webserver, wishbone.example.com. The url for the site is then https://wishbone.example.com.

Right after SSL Global Context, add this line:

LDAPTrustedGlobalCert CA_BASE64 /opt/certs/ldap-server.pem

(This file is the certificate for your openldap system. It’s self-signed and a CA as well.)

Make a self-signed certificate for wishbone that you’ll use for https. If you use the Makefile, both your certificate and key will be in the same file.

SSLCertificateFile /etc/pki/tls/certs/wishbone.pem
SSLCertificateKeyFile /etc/pki/tls/certs/wishbone.pem

At the end of the file, just before the </VirtualHost>, add the following:

LDAPTrustedClientCert CERT_BASE64 /opt/certs/ldap-server.pem
LDAPTrustedMode SSL

<Directory /var/www/html>
Order allow,deny
Allow from all
</Directory>

<Location /login>
AuthType Basic
Authname 'Account Info'
AuthBasicProvider 'ldap'
AuthLDAPURL 'ldaps://liberty.example.com:636/ou=people,dc=liberty,dc=example,dc=com?uid?one'
AuthzLDAPAuthoritative off
Require valid-user
</Location>

Had a weird problem come up today. I reboot one of our systems and it would not come up. Actually, it apparently was working, just ridiculously slowly. I thought it had hung at the Starting udev line, but letting it run for an hour, got it a few lines further down. Nonetheless, something was definitely wrong. I tried my usual ideas of adding apm=off acpi=off noapci noapmd, but that didn’t help. It seemed like the disk was bad, but not with the usual messages that I would get.

I brought out a RHEL installation disk and boot into rescue mode. I did get an error about it not being able to find all of the current linux installations. I then went into the bios and found that this computer was set up to have its sata disks act like ide ones. I’m pretty sure that I did this when I originally installed because it wouldn’t recognize the drives as ahci. However, I’m also sure that they should be ahci, which is newer, instead of acting like the old ide. So I changed this to ahci, even though I was sure this was going to cause more problems. (I was right about the last part.)

I then again, did a linux rescue. For some reason, I no longer got the error about not finding all my linux installations. I could run chroot /mnt/sysimage to get my installation mounted properly. Then, I copied /etc/modules.conf to /etc/modules.conf.SAVE. I knew that I needed to change the modules that were loaded to reflect ahci drives instead of ide ones. So I edited this line:

alias scsi_hostadapter1 ata_piix

to

alias scsi_hostadapter1 ahci

This would work for after the kernel is loaded, but I needed to make sure the ahci module was loaded at boot. For this, I needed to make a new initrd image. I used the following:

mkinitrc --preload=ahci \\
/boot/initrd-2.6.9-89.0.3.ELhugemem.mary.img \\
2.6.9-89.0.3.ELhugemem

Then, I edited /etc/grub.conf to take this initrd file instead of the one it had. And lastly, to make sure it was using the proper disk. For some reason, my disk devices would change from /dev/sda and /dev/sdb to /dev/sdb and /dev/sdc. Instead of putting the device names in /etc/grub.conf, I used the labels.

kernel /vmlinuz-2.6.9-89.0.3.ELhugemem ro root=/dev/sda3 rhgb quiet

I also used the labels in /etc/fstab.

LABEL=/1                   /                       ext3    defaults        1 1
LABEL=/boot	        /boot                   ext3    defaults        1 2

After I reboot and everything worked, I renamed the initrd file without the mary bit, thus overwriting the original.

One thing I was worried about was whether I would have to do this each time I updated the kernel. Fortunately, there was a kernel update ready, so I installed it. The initrd file that came in worked perfectly. I didn’t have to do anything special.

Trying to run yum -y upgrade, gave the following errors:

(snip)
--> Finished Dependency Resolution
perl-DBD-mysql-4.010-1.el4.rf.i386 from dag has depsolving problems
  --> Missing Dependency: libmysqlclient.so.14 is needed by package perl-DBD-mysql-4.010-1.el4.rf.i386 (dag)
perl-DBD-mysql-4.010-1.el4.rf.i386 from dag has depsolving problems
  --> Missing Dependency: libmysqlclient.so.14(libmysqlclient_14) is needed by package perl-DBD-mysql-4.010-1.el4.rf.i386 (dag)
subversion-1.4.6-0.1.el4.rf.i386 from dag has depsolving problems
  --> Missing Dependency: libapr-0.so.0 is needed by package subversion-1.4.6-0.1.el4.rf.i386 (dag)
subversion-1.4.6-0.1.el4.rf.i386 from dag has depsolving problems
  --> Missing Dependency: libneon.so.24 is needed by package subversion-1.4.6-0.1.el4.rf.i386 (dag)
1:nfs-utils-1.0.9-40.el5.i386 from installed has depsolving problems
  --> Missing Dependency: libevent-1.1a.so.1 is needed by package 1:nfs-utils-1.0.9-40.el5.i386 (installed)
subversion-1.4.6-0.1.el4.rf.i386 from dag has depsolving problems
  --> Missing Dependency: libaprutil-0.so.0 is needed by package subversion-1.4.6-0.1.el4.rf.i386 (dag)
Error: Missing Dependency: libmysqlclient.so.14 is needed by package perl-DBD-mysql-4.010-1.el4.rf.i386 (dag)
Error: Missing Dependency: libneon.so.24 is needed by package subversion-1.4.6-0.1.el4.rf.i386 (dag)
Error: Missing Dependency: libaprutil-0.so.0 is needed by package subversion-1.4.6-0.1.el4.rf.i386 (dag)
Error: Missing Dependency: libevent-1.1a.so.1 is needed by package 1:nfs-utils-1.0.9-40.el5.i386 (installed)
Error: Missing Dependency: libmysqlclient.so.14(libmysqlclient_14) is needed by package perl-DBD-mysql-4.010-1.el4.rf.i386 (dag)
Error: Missing Dependency: libapr-0.so.0 is needed by package subversion-1.4.6-0.1.el4.rf.i386 (dag)

These all seemed to be problems from the dag repo that I used to get the perl cpan modules. To fix, do the following:

cd /etc/yum.repos.d
mkdir save
mv dag.repo save
yum -y upgrade  (now runs successfully)
mv save/dag.repo .
rmdir save

I recently updated the kernel on our mailserver and found that spamassassin would no longer run. I was getting an error message something like this:

Errno architecture (i386-linux-thread-multi-2.6.9-78.0.1.elsmp) does not match executable architecture (i386-linux-thread-multi-2.6.18-128.1.1.el5) at /usr/lib/perl5/site_perl/5.8.8/Errno.pm line 11.
Compilation failed in require at /usr/lib/perl5/5.8.8/i386-linux-thread-multi/IO/Socket.pm line 17.
BEGIN failed--compilation aborted at /usr/lib/perl5/5.8.8/i386-linux-thread-multi/IO/Socket.pm line 17.
Compilation failed in require.
BEGIN failed--compilation aborted.

The solution was to get rid of everything that I installed with cpan. That means I just renamed /usr/lib/perl5/site_perl to something else and remade the site_perl directory with the following files:

/usr/lib/perl5/site_perl
/usr/lib/perl5/site_perl/5.8.5
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto
/usr/lib/perl5/site_perl/5.8.6
/usr/lib/perl5/site_perl/5.8.6/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.6/i386-linux-thread-multi/auto
/usr/lib/perl5/site_perl/5.8.7
/usr/lib/perl5/site_perl/5.8.7/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.7/i386-linux-thread-multi/auto
/usr/lib/perl5/site_perl/5.8.8
/usr/lib/perl5/site_perl/5.8.8/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.8/i386-linux-thread-multi/auto

After that, spamassassin started with no problems.

So, how to install other cpan modules without using cpan? (Since it can cause problems.) I added the Dag repository (http://dag.wieers.com/) to my setup. Just create a file /etc/yum.repos.d/dag.repo with the following:

#
# DAG (http://dag.wieers.com) additional RPMS repository
#
[main]
[dag]
name=DAG (http://dag.wieers.com) additional RPMS repository
baseurl=http://linuxsoft.cern.ch/dag/redhat/el4/en/$basearch/dag
gpgkey=http://linuxsoft.cern.ch/cern/slc4X/$basearch/docs/RPM-GPG-KEY-dag
gpgcheck=1
enabled=1

They actually have rpms for a lot of the modules. Then, I can simply run:

yum install perl-File-Find-Rule

Tools to use to monitor website traffic and bandwidth usage are webalizer and mrtg, respectively. Webalizer is quite easy to set up, just edit /etc/webalizer.conf to tell it where to write files. I also made those files only accessible to our subnet. And then run a daily cron job to generate the files.

I’m still setting up mrtg and will put my notes up as soon as it’s done.

Started getting this error in our samba logs and no one could mount a samba drive:

failed to bind to server ldap://server.uchicago.edu with dn="cn=Manager,dc=server,dc=uchicago,dc=edu" 
Error: Server is unwilling to perform unauthenticated bind (DN with no password) disallowed

I had just installed all the updates on the server, so I’m assuming this was related. I know that I ran smbpasswd -W to put the ldap password to use to bind in secrets.tdb. So, I just ran the command again and everything was fine. I guess that one of the updates either changed the format of the file or changed the file itself. Either way, we look to be ok now.

In the past, we’ve gotten errors from Cadence software about an incorrectly built binary, but the program would still run ok. The annoying errors were lost by editing /etc/csh.cshrc and adding the line:

setenv LD_ASSUME_KERNEL 2.4.1 # Gets rid of the "Incorrectly built binary" error

However, after the last software upgrade, any simple unix command was giving errors about libc.so.6: cannot open shared object file. These errors weren’t just annoying, they wouldn’t allow the simple programs, like hostname, who, etc. to run. So, the LD_ASSUME_KERNEL variable had to be unset.

On our server, I then got this error:

[maryh@server maryh]$ /net/sw/cad.rh/ic/ic5141/tools/dfII/bin/32bit/icms.exe: error while 
loading shared libraries: libXp.so.6: cannot open shared object file: No such file or directory

This was solved by running:

[root@server ~]# yum install xorg-x11-deprecated-libs

Now things work ok on the server, but as the csh.cshrc file gets pushed out to the other clients, it will have to be modified to put the LD_ASSUME_KERNEL line back in. This is because the clients are running rhel4 (the server is rhel5) and they need that line.