HEP System Administration

Found a new program, called lshw. It just gives a list of the hardware in the computer, much like /etc/sysconfig/hwconf. But, the part I like is that it also gives you the model of the motherboard. So, it saves me a trip of walking to other buildings to see what’s inside a computer.

Example showing motherboard (output is long):

*-core
       description: Motherboard
       product: P4S800
       vendor: ASUSTeK Computer INC.
       physical id: 0
       version: REV 1.xx
       serial: xxxxxxxxxxx
     *-firmware
          description: BIOS
          vendor: Award Software, Inc.
          physical id: 0
          version: ASUS P4S800 ACPI BIOS Revision 1009 (06/08/2004)
          size: 64KiB
          capacity: 192KiB
          capabilities: pci pnp apm upgrade shadowing escd cdboot bootselect socketedrom edd int13floppy360 int13floppy1200 int13floppy720 int13floppy2880 int5printscreen int9keyboard int14serial int17printer int10video acpi usb agp

One of the students wanted this software installed. GENIE Neutrino Monte Carlo Generator

I followed the instructions that were on that website, but here are the details specific to us.

First get the software. Start afs.

cvs -d /afs/rl.ac.uk/user/c/candreop/cvsrep checkout -P GENIE

Move the GENIE directory to /code/GENIE. This will become $GENIE_HOME.

There are also some prerequisites that are needed. libxml2 and log4cpp we already had. Though, later on, it complained about some log4cpp stuff being missing. So we also needed log4cpp-devel, which was not mentioned on the webpage. (yum install log4cpp-devel fixed us up.) Also installed log4cpp in /code/shared, so that it could be run on computers without log4cpp and log4cpp-devel, which in our case means all the Fermi linux computers.

We also needed LDAPDF. It said CERNLIB/PDFLIB could be used, but again, later in the process it complained about missing LDAPDF.

# wget http://www.hepforge.org/archive/lhapdf/lhapdf-5.7.0.tar.gz
untar and unzip
# cd ldapdf
# ./configure --prefix=/code/shared
# make
# make install

Next we needed something called pythia6. There was a script that just needed to be run and would download the library. I put it in /code/pythia6. After running the script, there was a new directory called v6_412 there. That’s all that was required for pythia.

Next, we needed root that had pythia6 enabled.

# ftp root.cern.ch
Connected to lxbuild091.cern.ch.
220 Welcome to ROOT FTP service.
530 Please login with USER and PASS.
530 Please login with USER and PASS.
KERBEROS_V4 rejected as an authentication type
Name (root.cern.ch:root): ftp
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> cd /root
250 Directory successfully changed.
ftp> bin
200 Switching to Binary mode.
ftp> get root_v5.22.00.source.tar.gz
local: root_v5.22.00.source.tar.gz remote: root_v5.22.00.source.tar.gz
227 Entering Passive Mode (128,142,141,37,23,60)
150 Opening BINARY mode data connection for root_v5.22.00.source.tar.gz (26320334 bytes).
226 File send OK.
26320334 bytes received in 45 seconds (5.7e+02 Kbytes/s)
ftp> bye

Unzip and untar it.

# ./configure --prefix=/code/root_v5.22.00 --etcdir=/code/root_v5.22.00/etc --enable-pythia6 --with-pythia6-libdir=/code/pythia6/v6_412/lib
# make
# make install

Lastly, compile GENIE

export GENIE=/code/GENIE 
export ROOTSYS=/code/root_v5.22.00
export PATH=$PATH:$GENIE/bin:$ROOTSYS/bin
# ./configure --prefix=/code/shared \
--with-pythia6-lib=/code/pythia6/v6_412 \
--with-lhapdf-inc=/code/shared/include \
--with-lhapdf-lib=/code/shared/lib \
--with-libxml2-inc=/usr/include/libxml2 \
--with-libxml2-lib=/usr/lib \
--with-log4cpp-lib=/code/shared/lib \
--with-log4cpp-inc=/code/shared/include/log4cpp

# gmake
# gmake install

That’s it. Then to run the software, be sure to set the following environment variables.

export GENIE=/code/shared
export ROOTSYS=/code/root_v5.22.00
export LD_LIBRARY_PATH=$ROOTSYS/lib/root:/code/pythia6/v6_412:/code/shared/lib:$GENIE/lib
export PATH=$PATH:/code/shared/bin:$ROOTSYS/bin:$GENIE/bin:$GENIE/src/stdapp
# genie 

I then could run the samples given on the installation page:

[mary@computer]$ export GEVGL=QEL 
[mary@computer]$ gmkspl -p 14 -t 1000060120 -n 300 -e 30 -o xsec-spline-numuC12-qel.xml 
[mary@computer]$ export GSPLOAD=xsec-spline-numuC12-qel.xml 
[mary@computer]$ gevgen -s -n 1000 -p 14 -t 1000060120 -e 0,15 -f $GENIE/data/flux/t2kflux.root,h30000 

The following are some notes I took while setting things up and problems I had. I tried to edit the above instructions with anything special that I needed to do to get everything working, but in case I didn’t, my notes might help.

NOTES
Problems
1. When I start genie, I get this error message:

Warning in : macro /code/shared/src/scripts/gcint/genie.C not found

and then root starts instead of genie.

That script is located in the src directory that came with the genie source. I’m copying it to /code/shared.

2. Next I got an error about libGVHE missing. I don’t know why it didn’t get compiled along with all the other libraries. It did get compiled in one of my other attempts. So instead of recompiling everything, I just copied it from the old location to the new one.

3. Next problem, I now get this error:

CINT/ROOT C/C++ Interpreter version 5.16.29, Jan 08, 2008
Type ? for help. Commands must be C++ statements.
Enclose multiple statements between { }.
root [0] 
Processing /code/shared/src/scripts/gcint/genie.C...
dlopen error: libLHAPDF.so.0: cannot map zero-fill pages: Cannot allocate memory
Load Error: Failed to load Dynamic link library /code/shared/lib/libGPDF.so
*** Interpreter error recovered ***
genie [1] 

So we’re running out of memory. I tried to run genie on a system with 16GB of ram and everything worked fine. So I need to install more memory to this computer.

I found a Fermi Linux computer to use for this software. Since Fermi Linux does not include log4cpp in its available packages, I had to compile it from scratch.

./configure --prefix=/code/shared
make
make install

I also had to install libgfortran.

yum install libgfortran

Appears to be working now.

Spoke too soon. Constantinos had a problem running the gmkspl and gevgen programs. They are located in $GENIE/src/stdapp. There was a make file in there, but they never got made. When I ran gmake, I got complaints bout Pythia6 missing, even though the full path was included in the command. (It was set with LD_LIBRARY_PATH.) To solve this, I created a link in /code/shared/lib:

# ln -s /code/pythia6/v6_412/lib/libPythia6.so libPythia6.so

Then gmake worked ok.

Also had to copy /code/genie/config to /code/shared/config for some missing files.

Got an error message like this:

1237576361 ERROR PDG : [n]  : Could not load PDG data
Error in : Could not open PDG particle file /etc/root/pdg_table.txt

I found that all the root installations had a directory $ROOTSYS/etc, inside of which, there was a file pdg_table.txt. So I copied the directory from the root download area to my new $ROOTSYS. This worked, but then I got a new error message:

1237576249 ERROR PDG : [n]  : Can't add non-existent particle [pdgc = 1000060120]

So, I think that I have to use the pdg_table.txt that was found in /code/genie/data/pdg. I copied that file to $ROOTSYS/etc/pdg_table.txt.

Now I have a new error, that I don’t really know how to fix.

gmkspl: INukeHadroData.cxx:173: void genie::INukeHadroData::LoadCrossSections(): Assertion `! gSystem->AccessPathName(datafile_NN. c_str())' failed.
Aborted

Wah bought some VMIVME-7648 controllers that looked very much like the VMIVME-7700 that we already had. Here is a lousy picture of one of the controllers.

My job was to install linux on them and get them ready for use in our teststands. First problem was that they did not have any method of booting from usb. Since that was the only way to connect a dvd drive with the software on it, this turned out to be a problem. They did, however, have PXE booting. I set up a little dhcp server on another linux box (using a second network card), but could not get it to boot that way either. In the logs, I could see that it requested an address and that one was offered, but after that it would just timeout.

I finally figured out that the best thing to do was to install linux on the compact flash card on a different computer and then put the compact flash card in the controller. Though when I did a standard installation, the error message that I kept getting was:

UMB upper limit segment address: E884
Fixed Disk 0: CF Card
ERROR:
0200: Failure Fixed Disk

I knew that the disk had really not failed because I had tried three different ones and they worked in other systems, just not that one. At first, I was using a 4GB compact flash card and thought that it was failing because it was too big. The one piece of documentation I found on this board said that it originally came with either a 256MB, 512MB or 1GB compact flash card. So I switched to trying to use a 1GB card. The problem here is that it was quite difficult to get RHEL onto a 1GB card, but it could be done by using a minimal install. (Interesting side note: when I did a text install, I never saw minimum as a choice in the package listings, but I did see it when I did a graphical install. Weird.) I had also downloaded and tried to install Damn Small Linux, but I couldn’t get that installed either.

My first bit of success came when I changed the format of the disk from ext3 to ext2. Now, when I boot, I at least got some error messages. I got the following:

hdc: dma_timer_expiry: dma status == 0x21
hdc: DMA timeout error
hdc: dma timeout error: status=0x58 { DriveReady SeekComplete DataRequest }

ide: failed opcode was: unknown
VFS: Can't find ext2 filesystem on dev hdc.
mount: error 22 mounting ext2
mount: error 2 mounting none
Switching to new root
switchroot: mount failed: 22
umount /initrd/dev failed: 2
Kernel panic - not syncing:  Attempted to kill init!

The problem with everything was in the BIOS of the computer controllers. The clue here is that there’s something to do with DMA. I tried adding

ide=nodma

to the kernel line in the grub.conf file, but that didn’t help too much.

The solution was in the BIOS of the controller. I changed the parameters of the compact flash card from “Auto” to “User”. Then, I could disable the “Ultra DMA Mode”, which solved all problems.

Notice in the image that the compact flash card comes up as the master on the secondary controller. I don’t know why this is, but it followed this behavior on all three controllers that I tried. This meant that I knew what the device name of the hard disk would be in linux (/dev/hdc).

So, how did I install linux on the compact flash card? Fortunately, I had a usb-connected multiple card reader.

I went to a desktop computer and disconnected all the internal drives (except the cdrom) and hooked up the usb card reader with the compact flash card inserted. I then boot off the rhel4 installation cd and entered the following at the boot prompt.

boot:  linux expert askmethod

The expert flag is needed so that I could see the attached usb drive. The askmethod was so that it would let me do a network install. (I needed this because I didn’t have a dvd drive on the computer I used. A network install was easier for me.)

When you get to the page about partitioning the drive and whether to use “Disk Druid” or manually, you have to choose manual. Once you do, it will immediately popup an error message about /dev/sda not being present. After you dismiss that one (click Ignore, I think), another will pop up noting /dev/sdc is missing and then it will happen again with /dev/sdd. The reason for this is because my card reader will actually read four different types of flash cards. It just happens that the compact flash card comes up as the second in the chain, so it’s always /dev/sdb. Since I used a 4gb card, I created a 200mb /boot partition, a 512 mb swap partition and the rest under /. The next page asks about the bootloader. The default is for it to install grub on /dev/sda, which is a problem because there is no /dev/sda. I think I tried putting it on /dev/sdb1, but that didn’t work either. Somewhere on that page, there’s a checkbox for advanced bootloader options. Check that and on the next screen, you’ll be allowed to change the order of the drives. Here, you can switch /dev/sdb to be the first drive, so that grub will be installed on the master boot record on /dev/sdb, which is what we want. After that, there was nothing else very special about the installation. I just unselected most packages because we didn’t need them. (When I was using the 1gb compact flash cards, I just selected “Minimal” install.) Installing to the 4gb compact flash card took a couple of hours.

With linux (in our case RHEL4) installed on the compact flash card, I reconnected all the drives on the pc and boot it. I then mounted the compact flash card and edited /etc/grub.conf. Here’s what my final /etc/grub.conf looked like:

boot=/dev/hdc
default=0
timeout=3
title Red Hat Enterprise Linux AS (2.6.9-78.EL)
	root (hd0,0)
	kernel /vmlinuz-2.6.9-78.EL ro root=/dev/hdc3 ide=nodma
	initrd /initrd-2.6.9-78.EL.img

I plugged that in the crate controller and started it up. When kudzu starts, you’ll be prompted to remove some hardware that was on the computer used for installation and then install hardware specific to the vme computer. After that, you’re all set. The only error message I get is one complaining that RHEL4 requires 256mb of ram and these computers only have 128mb. However, it still runs fine. I’m going to order some ram for these to get rid of that error, but it’s not a big worry.

The last step, for us, was to install the VME software. Since I had downloaded it previously, all I needed to do was copy it to the new computer and compile it. This is where having the 4gb card is great. I had enough space to install all the development tools, so I just had to run make install to get it all set up. Final step is to automatically load the vme module. Add the following to /etc/rc.d/rc.local:

modprobe vme_universe

I also attached a little 2gb usb flash drive for storage. (When I get a bigger one I’ll use that.) I then copied all the programs from our old controller to the new one and checked that they worked. We haven’t run any big tests yet, but so far everything appears to be working.