HEP System Administration

Our VME system is somewhat old, running Scientific Linux 4.7. Unfortunately, I was never able to get a newer version of the operating system working with these computers.

I needed to compile a program on this system. And the first attempt gave me tons of errors:

[root@wahcrate koto]# gcc adclossless.c -o adclossless -lvme
In file included from /usr/include/stdio.h:28,
                 from adclossless.c:1:
/usr/include/features.h:1: error: stray '\31' in program
/usr/include/features.h:1: error: stray '\139' in program
/usr/include/features.h:1: error: stray '\8' in program
/usr/include/features.h:1: error: stray '\2' in program
/usr/include/features.h:1: error: stray '\3' in program

Since I can’t really update this program, my hack to get this working was to copy any header files that were causing errors from a different computer. I copied the following files from a system running Red Hat Enterprise Linux, version 5.

/usr/include/features.h
/usr/include/gconv.h
/usr/include/alloca.h
/usr/include/errno.h
/usr/include/fcntl.h
/usr/include/getopt.h

In case this caused any problems, I renamed the old files first to .OLD. Lastly, the compilation complained about the file /usr/include/endian.h not being present. So I copied that file too.

After that the compilation worked.

[root@wahcrate koto]# gcc adclossless.c -o adclossless -lvme
/tmp/cc5q5khk.o(.text+0x270): In function `main':
: warning: the `gets' function is dangerous and should not be used.

I was also able to start the program.

[root@wahcrate koto]# ./adclossless 

 *************************************************** 
 *****     Beam Test Program for FADC Board     **** 
 *************************************************** 

(Slot:4 type:"none")>>help


    Commands:   read 
    Commands:   write    < value> 
    Commands:   read_vme  slot_number address 
    Commands:   write_vme slot_number address value 

    Commands:   run lb  <2.5, 3.125> 
    Commands:   run dc  
    Commands:   run db  
    Commands:   run ec  
    Commands:   run rnt  

    Commands:   reset_reg     
    Commands:   set slot_number   <1-21>
    Commands:   show
    Commands:   Exit,exit,q 


(Slot:4 type:"none")>>q

Unfortunately, at this point, I have no idea if it will work, since I don’t know how to use it. I will wait and see if it works for the student.

Here are the steps we did:
1. Make the following directories. You might have to be root to make them, but then change the owner and group on /opt/atlas to a regular user.

mkdir -p /opt/atlas/apt/rpmdm
mkdir -p /opt/atlas/apt/rpm
chown -R user:group /opt/atlas

Exit root.

2. Get apt and install in /opt/atlas

cd /opt/atlas
rpm -ivh --nodeps --relocate=/=$PWD/apt --dbpath=$PWD/apt/rpmdb http://atlas-computing.web.cern.ch/atlas-computing/links/reposDirectory/apt/apt-0.5_15lorg3.90-1.slc4.atlas.i386.rpm

3. Edit ~/.rpmmacros (don’t do this to the root account)
%_dbpath /opt/atlas/apt/var/lib/rpm
%_rpmlock_path /opt/atlas/apt/rpm/transaction

4. Get and fix the apt config file

cd /opt/atlas
wget http://pcatd12.cern.ch/releases/download/config/apt.conf
sed s#INSTALL_ROOT#$PWD#g apt.conf > apt/etc/apt.conf

5. Add the sources for these files to the sources list

cd /opt/atlas/apt/etc/apt/sources.list.d
wget http://pcatd12.cern.ch/releases/download/config/atlas.list

6. Get the setup file

cd /opt/atlas/apt
wget http://pcatd12.cern.ch/releases/download/config/setup.sh

7. Source the setup file

source setup.sh

8. Update apt lists

apt-get update

9. Install the tdaq software and source (all dependencies will be installed as well)

apt-get install tdaq-01-09-01_i686_slc4_gcc34_opt
apt-get install tdaq_src

The library in which we are most-interested is in:
/opt/atlas/tdaq/tdaq-01-09-01/installed/i686-slc4-gcc34-opt/lib/libROSslink.so

We copied this file to /usr/lib.

We’ve found that the vme software will in fact compile with a 2.6 kernel. The version of the software that worked for us was Scientific Linux 4.7. Strangely, Red Hat Enterprise Linux 4 did not work at all. We got lots and lots of error messages when running make. On SL47, we only got a few errors about MODULE_PARMS being an error. (I read somewhere that we just had to switch MODULE_PARMS to module_params, but turns out that we didn’t even need to do that.)

The odd error that we did have was after we ran make install, we could load the module (modprobe vme_universe) and things worked fine. However, after a reboot, the module would be loaded, but the hello_vme program would keep giving us the “Cannot initialize VMEbus” error. After examining more closely what make install was doing, we added the following to /etc/rc.d/rc.local:

# Load VME module
# Also need to create the device files
mkdir -p /dev/bus/vme
mknod --mode=666 /dev/bus/vme/ctl c 221 8
modprobe vme_universe

That fixed things. I don’t know why MAKEDEV isn’t making the module, but we got around this issue.

We still have a problem of getting large usb drives to work, but possibly, since we have to make the device files by hand for the vme stuff, that we may also have to do that for the usb stuff. (And we need to look into which modules need to be loaded for usb things.)

We have learned some things about the VMIVME-7648 controllers and the vme software that we have. The main issue is that the vmisft software that we have will only compile on a 2.4 kernel. Thus, our previous installation of SLC4.5 (basically RHEL4) with its 2.6 kernel would not work. So the first step is to install something with a 2.4 kernel. For us, we used SLC306 (derived from RHEL3) because we had the disks.

Our first attempt was to simply do the installation just as we had done previously–connect our multicard reader to a pc with no other drives and install. We had a problem in that our multicard reader always showed a disk for each type of card. This meant that during the installation, an error would come up about there not being /dev/sda present. The compact flash card always came up as /dev/sdb. In the SLC45 installation, we could simply cancel this error and move on to the next disk. However, in the SLC306 installation, canceling this error led to the automatic closing of the installation program. We got lucky in that a new multicard reader we bought did not have this problem. The compact flash card in that reader showed as /dev/sda, which solved all our problems.

For reference, the model we used is the Verbatim Universal Card Reader 15-in-1, manufacturer part number 95343.

Once the operating system was installed, the compact flash card was installed into the vme controller. When grub started, we needed to stop the boot process and edit the boot options. We needed to add ide=nodma to the end of the kernel line.

Then, during the boot process, it will find that a lot of hardware had changed. Just delete anything that it says is missing and configure any new hardware. This will require the network cards to be reconfigured.

Once the system is up. Edit the file /etc/grub.conf with the ide=nodma bit so that this is always present. Also, it’s a good idea to disable the second network card, if it’s not needed. This is done in /etc/sysconfig/network-scripts/ifcfg-eth0 and ifcfg-eth1. Just change ONBOOT=yes to ONBOOT=no for whichever one is not needed.

Next, compile and install the vmisft software. It can be downloaded here.
vmisft software

Unzip and untar the file. Then run make and make install to install the software. Next you need to make sure that the vme_universe module is loaded whenever the computer is turned on. Add the following to /etc/rc.d/rc.local:

# Load vme module
modprobe vme_universe

To check that everything is working, simply type:

modprobe vme_universe

to load the module immediately. Run lsmod to check that the module is loaded. Once it’s loaded, the vmisft directory has a test program. Run:

cd vmisft-7433-3.6/vme_universe/TOOLS
./hello_vme

If everything is working correctly, you’ll get a message like this:

Bus handle value 3
We're ready to start accessing the bus now!

If things are not working, the error message will be:

Cannot initialize the VMEbus

Some other issues that we had with these controllers had to do with the bios. The following are some screenshots showing how things should be set up. Only screens changed from the default are shown.

Wah bought some VMIVME-7648 controllers that looked very much like the VMIVME-7700 that we already had. Here is a lousy picture of one of the controllers.

My job was to install linux on them and get them ready for use in our teststands. First problem was that they did not have any method of booting from usb. Since that was the only way to connect a dvd drive with the software on it, this turned out to be a problem. They did, however, have PXE booting. I set up a little dhcp server on another linux box (using a second network card), but could not get it to boot that way either. In the logs, I could see that it requested an address and that one was offered, but after that it would just timeout.

I finally figured out that the best thing to do was to install linux on the compact flash card on a different computer and then put the compact flash card in the controller. Though when I did a standard installation, the error message that I kept getting was:

UMB upper limit segment address: E884
Fixed Disk 0: CF Card
ERROR:
0200: Failure Fixed Disk

I knew that the disk had really not failed because I had tried three different ones and they worked in other systems, just not that one. At first, I was using a 4GB compact flash card and thought that it was failing because it was too big. The one piece of documentation I found on this board said that it originally came with either a 256MB, 512MB or 1GB compact flash card. So I switched to trying to use a 1GB card. The problem here is that it was quite difficult to get RHEL onto a 1GB card, but it could be done by using a minimal install. (Interesting side note: when I did a text install, I never saw minimum as a choice in the package listings, but I did see it when I did a graphical install. Weird.) I had also downloaded and tried to install Damn Small Linux, but I couldn’t get that installed either.

My first bit of success came when I changed the format of the disk from ext3 to ext2. Now, when I boot, I at least got some error messages. I got the following:

hdc: dma_timer_expiry: dma status == 0x21
hdc: DMA timeout error
hdc: dma timeout error: status=0x58 { DriveReady SeekComplete DataRequest }

ide: failed opcode was: unknown
VFS: Can't find ext2 filesystem on dev hdc.
mount: error 22 mounting ext2
mount: error 2 mounting none
Switching to new root
switchroot: mount failed: 22
umount /initrd/dev failed: 2
Kernel panic - not syncing:  Attempted to kill init!

The problem with everything was in the BIOS of the computer controllers. The clue here is that there’s something to do with DMA. I tried adding

ide=nodma

to the kernel line in the grub.conf file, but that didn’t help too much.

The solution was in the BIOS of the controller. I changed the parameters of the compact flash card from “Auto” to “User”. Then, I could disable the “Ultra DMA Mode”, which solved all problems.

Notice in the image that the compact flash card comes up as the master on the secondary controller. I don’t know why this is, but it followed this behavior on all three controllers that I tried. This meant that I knew what the device name of the hard disk would be in linux (/dev/hdc).

So, how did I install linux on the compact flash card? Fortunately, I had a usb-connected multiple card reader.

I went to a desktop computer and disconnected all the internal drives (except the cdrom) and hooked up the usb card reader with the compact flash card inserted. I then boot off the rhel4 installation cd and entered the following at the boot prompt.

boot:  linux expert askmethod

The expert flag is needed so that I could see the attached usb drive. The askmethod was so that it would let me do a network install. (I needed this because I didn’t have a dvd drive on the computer I used. A network install was easier for me.)

When you get to the page about partitioning the drive and whether to use “Disk Druid” or manually, you have to choose manual. Once you do, it will immediately popup an error message about /dev/sda not being present. After you dismiss that one (click Ignore, I think), another will pop up noting /dev/sdc is missing and then it will happen again with /dev/sdd. The reason for this is because my card reader will actually read four different types of flash cards. It just happens that the compact flash card comes up as the second in the chain, so it’s always /dev/sdb. Since I used a 4gb card, I created a 200mb /boot partition, a 512 mb swap partition and the rest under /. The next page asks about the bootloader. The default is for it to install grub on /dev/sda, which is a problem because there is no /dev/sda. I think I tried putting it on /dev/sdb1, but that didn’t work either. Somewhere on that page, there’s a checkbox for advanced bootloader options. Check that and on the next screen, you’ll be allowed to change the order of the drives. Here, you can switch /dev/sdb to be the first drive, so that grub will be installed on the master boot record on /dev/sdb, which is what we want. After that, there was nothing else very special about the installation. I just unselected most packages because we didn’t need them. (When I was using the 1gb compact flash cards, I just selected “Minimal” install.) Installing to the 4gb compact flash card took a couple of hours.

With linux (in our case RHEL4) installed on the compact flash card, I reconnected all the drives on the pc and boot it. I then mounted the compact flash card and edited /etc/grub.conf. Here’s what my final /etc/grub.conf looked like:

boot=/dev/hdc
default=0
timeout=3
title Red Hat Enterprise Linux AS (2.6.9-78.EL)
	root (hd0,0)
	kernel /vmlinuz-2.6.9-78.EL ro root=/dev/hdc3 ide=nodma
	initrd /initrd-2.6.9-78.EL.img

I plugged that in the crate controller and started it up. When kudzu starts, you’ll be prompted to remove some hardware that was on the computer used for installation and then install hardware specific to the vme computer. After that, you’re all set. The only error message I get is one complaining that RHEL4 requires 256mb of ram and these computers only have 128mb. However, it still runs fine. I’m going to order some ram for these to get rid of that error, but it’s not a big worry.

The last step, for us, was to install the VME software. Since I had downloaded it previously, all I needed to do was copy it to the new computer and compile it. This is where having the 4gb card is great. I had enough space to install all the development tools, so I just had to run make install to get it all set up. Final step is to automatically load the vme module. Add the following to /etc/rc.d/rc.local:

modprobe vme_universe

I also attached a little 2gb usb flash drive for storage. (When I get a bigger one I’ll use that.) I then copied all the programs from our old controller to the new one and checked that they worked. We haven’t run any big tests yet, but so far everything appears to be working.

We have a vme crate running RedHat Enterprise Linux 3. Students are writing programs to access data on the vme crate. In order to do this, we need a computer with compilation tools and drivers for the vme crate. (The crate only has a 1gb flash card in it, which does not have enough room to hold all the compilation programs.) So, we are using a computer running Cern SL 3.0.8 (2.4.21-27.0.2.EL.cernsmp), which is close enough to the kernel on the crate (2.4.21-4.EL).

The driver/libraries were downloaded from here and the documentation came from here

The software was untarred to /cpv/code/vme where it was compiled with make and installed with make install (as root).

It installed:

ll /usr/lib/libvme.so*
lrwxrwxrwx    1 root     root           11 Aug 18 12:15 /usr/lib/libvme.so -> libvme.so.3
lrwxrwxrwx    1 root     root           13 Aug 18 12:15 /usr/lib/libvme.so.3 -> libvme.so.3.6
-rw-r--r--    1 root     root        12629 Aug 18 12:15 /usr/lib/libvme.so.3.6
/lib/modules/2.4.21-27.0.2.EL.cernsmp/kernel/drivers/vme
ll
total 68
-rw-r--r--    1 root     root        63606 Aug 18 12:14 vme_universe.o
cd /usr/include/vme
ll
total 72
-rw-r--r--    1 root     root        33954 Aug 18 12:15 universe.h
-rw-r--r--    1 root     root        10516 Aug 18 12:15 vme_api.h
-rw-r--r--    1 root     root        12804 Aug 18 12:15 vme.h
-rw-r--r--    1 root     root         5682 Aug 18 12:15 vmivme.h

Then go to the vme_universe/test directory and run make and make install. This creates the following:

cd /usr/bin
ll vme*
-rwxr-xr-x    1 root     root        10748 Aug 18 13:16 vme_acquire_bus*
-rwxr-xr-x    1 root     root        13198 Aug 18 13:16 vme_catch_interrupt*
-rwxr-xr-x    1 root     root        12982 Aug 18 13:16 vme_dma_read*
-rwxr-xr-x    1 root     root        13366 Aug 18 13:16 vme_dma_write*
-rwxr-xr-x    1 root     root        13231 Aug 18 13:16 vme_endian*
-rwxr-xr-x    1 root     root        11461 Aug 18 13:16 vme_generate_interrupt*
-rwxr-xr-x    1 root     root        13034 Aug 18 13:16 vme_peek*
-rwxr-xr-x    1 root     root        13234 Aug 18 13:16 vme_poke*
-rwxr-xr-x    1 root     root        10748 Aug 18 13:16 vme_release_bus*
-rwxr-xr-x    1 root     root        12205 Aug 18 13:16 vme_rmw*
-rwxr-xr-x    1 root     root        13035 Aug 18 13:16 vme_slave_peek*
-rwxr-xr-x    1 root     root        13327 Aug 18 13:16 vme_slave_poke*
-rwxr-xr-x    1 root     root        10708 Aug 18 13:16 vme_sysreset*