Boot failure, related to module symbols?
I'm trying to build Scientific Linux 5.8 appliances. In the past week or so, while the appliances have built normally, they've failed at boot.
There seems to be a problem with the module symbols, meaning that block and filesystem modules aren't loaded and so the kernel can't find /dev/root.
Here's an example boot log:
Linux version 2.6.18-308.20.1.el5 (brewbuilder@norob.fnal.gov) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-52)) #1 SMP Tue Nov 13 11:54:02 EST 2012
Command line: ro root=LABEL=79d3d2d4 console=tty0 console=ttyS0
BIOS-provided physical RAM map:
BIOS-e820: 0000000000010000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000003fffe000 (usable)
BIOS-e820: 000000003fffe000 - 0000000040000000 (reserved)
BIOS-e820: 00000000feffc000 - 00000000ff000000 (reserved)
BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
DMI 2.4 present.
kvm-clock: cpu 0, msr 7eff:804a7401, boot clock
No NUMA configuration found
Faking a node at 0000000000000000-000000003fffe000
Bootmem setup node 0 0000000000000000-000000003fffe000
Memory for crash kernel (0x0 to 0x0) notwithin permissible range
disabling kdump
ACPI: PM-Timer IO Port: 0xb008
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 6:2 APIC version 20
ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
Setting APIC routing to physical flat
ACPI: HPET id: 0x8086a201 base: 0xfed00000
Using ACPI (MADT) for SMP configuration information
Nosave address range: 000000000009f000 - 00000000000a0000
Nosave address range: 00000000000a0000 - 00000000000f0000
Nosave address range: 00000000000f0000 - 0000000000100000
Allocating PCI resources starting at 50000000 (gap: 40000000:beffc000)
SMP: Allowing 1 CPUs, 0 hotplug CPUs
kvm-clock: cpu 0, msr 0:1735401, primary cpu clock
Built 1 zonelists. Total pages: 256944
Kernel command line: ro root=LABEL=79d3d2d4 console=tty0 console=ttyS0
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
Using TSC for driving interrupts
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
Checking aperture...
ACPI: DMAR not present
Memory: 1023408k/1048568k available (2620k kernel code, 24708k reserved, 1673k data, 224k init)
Calibrating delay loop (skipped), value calculated using timer frequency.. 6784.58 BogoMIPS (lpj=3392292)
Security Framework v1.0.0 initialized
SELinux: Initializing.
selinux_register_security: Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 256
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 4096K
SMP alternatives: switching to UP code
Freeing SMP alternatives: 32k freed
ACPI: Core revision 20060707
Using local APIC timer interrupts.
Detected 62.500 MHz APIC timer.
Brought up 1 CPUs
time.c: Using 100.000000 MHz WALL KVM GTOD KVM timer.
time.c: Detected 3392.292 MHz processor.
checking if image is initramfs... it is
Freeing initrd memory: 2471k freed
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: No dock devices found.
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI quirk: region b000-b03f claimed by PIIX4 ACPI
PCI quirk: region b100-b10f claimed by PIIX4 SMB
ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
ACPI: PCI Interrupt Link [LNKS] (IRQs 9) *0, disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 7 devices
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: protocols = UNLABELED CIPSOv4
NetLabel: unlabeled traffic allowed by default
hpet0: at MMIO 0xfed00000 (virtual 0xffffffffff5fe000), IRQs 2, 8, 0
hpet0: 3 64-bit timers, 100000000 Hz
ACPI: DMAR not present
PCI-GART: No AMD northbridge found.
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 6, 262144 bytes)
TCP established hash table entries: 131072 (order: 9, 2097152 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
audit: initializing netlink socket (disabled)
type=2000 audit(1354799734.324:1): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Initializing Cryptographic API
alg: No test for crc32c (crc32c-generic)
ksign: Installing public key data
Loading keyring
- Added public key 6E36AF2ED02D2A5A
- User ID: Red Hat, Inc. (Kernel Module GPG key)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Limiting direct PCI/PCI transfers.
PCI: PIIX3: Enabling Passive Release on 0000:00:01.0
Activating ISA DMA hang workarounds.
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Real Time Clock Driver v1.12ac
Non-volatile memory driver v1.2
Linux agpgart interface v0.101 (c) Dave Jones
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
�serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
brd: module loaded
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
PIIX3: IDE controller at PCI slot 0000:00:01.1
PIIX3: chipset revision 0
PIIX3: not 100% native mode: will probe irqs later
ide0: BM-DMA at 0xc0e0-0xc0e7, BIOS settings: hda:pio, hdb:pio
ide1: BM-DMA at 0xc0e8-0xc0ef, BIOS settings: hdc:pio, hdd:pio
ide-floppy driver 0.99.newide
usbcore: registered new driver hiddev
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
TCP bic registered
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
input: AT Translated Set 2 keyboard as /class/input/input0
ACPI: (supports S5)
Initalizing network drop monitor service
Freeing unused kernel memory: 224k freed
Write protecting the kernel read-only data: 531k
Red Hat nash version 5.1.19.6 starting
Mounting proc filesystem
Mounting sysfs filesystem
Creating /dev
Creating initial device nodes
Setting up hotplug.
input: ImExPS/2 Generic Explorer Mouse as /class/input/input1
Creating block device nodes.
Loading mptspi.ko module
mptspi: Unknown symbol mpt_deregister
mptspi: Unknown symbol mpt_reset_deregister
mptspi: Unknown symbol mptscsih_host_attrs
mptspi: Unknown symbol mptscsih_qcmd
mptspi: Unknown symbol mptscsih_bios_param
mptspi: Unknown symbol mptscsih_io_done
mptspi: Unknown symbol mptscsih_slave_configure
mptspi: Unknown symbol mpt_config
mptspi: Unknown symbol mptscsih_taskmgmt_complete
mptspi: Unknown symbol mptscsih_ioc_reset
mptspi: Unknown symbol mpt_free_msg_frame
mptspi: Unknown symbol mptscsih_remove
mptspi: Unknown symbol mptscsih_bus_reset
mptspi: Unknown symbol mptscsih_is_phys_disk
mptspi: Unknown symbol spi_display_xfer_agreement
mptspi: Unknown symbol mptscsih_proc_info
mptspi: Unknown symbol __scsi_iterate_devices
mptspi: Unknown symbol mptscsih_host_reset
mptspi: Unknown symbol spi_release_transport
mptspi: Unknown symbol mpt_put_msg_frame
mptspi: Unknown symbol mpt_register
mptspi: Unknown symbol mpt_event_deregister
mptspi: Unknown symbol mpt_findImVolumes
mptspi: Unknown symbol mptscsih_scandv_complete
mptspi: Unknown symbol scsi_scan_host
mptspi: Unknown symbol mptscsih_resume
mptspi: Unknown symbol mptscsih_raid_id_to_num
mptspi: Unknown symbol mpt_event_register
mptspi: Unknown symbol mptscsih_event_process
mptspi: Unknown symbol scsi_add_host
mptspi: Unknown symbol mptscsih_suspend
mptspi: Unknown symbol scsi_is_host_device
mptspi: Unknown symbol mptscsih_slave_destroy
mptspi: Unknown symbol scsi_print_command
mptspi: Unknown symbol mptscsih_change_queue_depth
mptspi: Unknown symbol scsi_scan_target
mptspi: Unknown symbol spi_attach_transport
mptspi: Unknown symbol mptscsih_shutdown
mptspi: Unknown symbol mpt_get_msg_frame
mptspi: Unknown symbol mptscsih_dev_reset
mptspi: Unknown symbol mpt_attach
mptspi: Unknown symbol mpt_reset_register
mptspi: Unknown symbol mptscsih_info
mptspi: Unknown symbol mptscsih_abort
mptspi: Unknown symbol mpt_HardResetHandler
mptspi: Unknown symbol scsi_host_alloc
mptspi: Unknown symbol spi_dv_device
mptspi: Unknown symbol scsi_device_lookup_by_target
mptspi: Unknown symbol mptscsih_IssueTaskMgmt
insmod: error inserting '/lib/mptspi.ko': -1 Unknown symbol in module
Loading virtio_pci.ko module
virtio_pci: Unknown symbol vring_transport_features
virtio_pci: Unknown symbol vring_interrupt
virtio_pci: Unknown symbol vring_new_virtqueue
virtio_pci: Unknown symbol unregister_virtio_device
virtio_pci: Unknown symbol register_virtio_device
virtio_pci: Unknown symbol vring_del_virtqueue
insmod: error inserting '/lib/virtio_pci.ko': -1 Unknown symbol in module
Loading virtio_blk.ko module
virtio_blk: Unknown symbol register_virtio_driver
virtio_blk: Unknown symbol unregister_virtio_driver
virtio_blk: Unknown symbol virtio_check_driver_offered_feature
insmod: error inserting '/lib/virtio_blk.ko': -1 Unknown symbol in module
Loading ehci-hcd.ko module
Loading ohci-hcd.ko module
Loading uhci-hcd.ko module
USB Universal Host Controller Interface driver v3.0
ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
ACPI: PCI Interrupt 0000:00:01.2[D] -> Link [LNKD] -> GSI 11 (level, high) -> IRQ 11
uhci_hcd 0000:00:01.2: UHCI Host Controller
uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1
uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c040
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
Loading ext4.ko module
ext4: Unknown symbol jbd2_journal_forget
ext4: Unknown symbol jbd2_journal_stop
ext4: Unknown symbol jbd2_journal_restart
ext4: Unknown symbol jbd2_journal_load
ext4: Unknown symbol jbd2_journal_check_available_features
ext4: Unknown symbol jbd2_journal_set_features
ext4: Unknown symbol jbd2_journal_unlock_updates
ext4: Unknown symbol jbd2_journal_file_inode
ext4: Unknown symbol jbd2_journal_get_create_access
ext4: Unknown symbol jbd2_journal_init_dev
ext4: Unknown symbol jbd2_journal_clear_features
ext4: Unknown symbol jbd2_journal_init_jbd_inode
ext4: Unknown symbol jbd2_journal_get_write_access
ext4: Unknown symbol jbd2_log_start_commit
ext4: Unknown symbol jbd2_journal_force_commit_nested
ext4: Unknown symbol jbd2_journal_clear_err
ext4: Unknown symbol jbd2_journal_init_inode
ext4: Unknown symbol jbd2_journal_abort
ext4: Unknown symbol jbd2_journal_blocks_per_page
ext4: Unknown symbol jbd2_journal_update_format
ext4: Unknown symbol jbd2_journal_release_jbd_inode
ext4: Unknown symbol jbd2_journal_revoke
ext4: Unknown symbol crc16
ext4: Unknown symbol jbd2_journal_start_commit
ext4: Unknown symbol jbd2_journal_extend
ext4: Unknown symbol jbd2_journal_dirty_metadata
ext4: Unknown symbol jbd2_journal_try_to_free_buffers
ext4: Unknown symbol jbd2_journal_wipe
ext4: Unknown symbol jbd2_journal_start
ext4: Unknown symbol jbd2_journal_destroy
ext4: Unknown symbol jbd2_journal_release_buffer
ext4: Unknown symbol jbd2_journal_force_commit
ext4: Unknown symbol jbd2_journal_get_undo_access
ext4: Unknown symbol jbd2_log_wait_commit
ext4: Unknown symbol jbd2_journal_invalidatepage
ext4: Unknown symbol jbd2_journal_lock_updates
ext4: Unknown symbol jbd2_journal_errno
ext4: Unknown symbol jbd2_journal_begin_ordered_truncate
ext4: Unknown symbol jbd2_journal_flush
insmod: error inserting '/lib/ext4.ko': -1 Unknown symbol in module
Loading dm-mem-cache.ko module
Loading dm-region_hash.ko module
dm_region_hash: Unknown symbol dm_dirty_log_destroy
insmod: error inserting '/lib/dm-region_hash.ko': -1 Unknown symbol in module
Loading dm-message.ko module
Loading dm-raid45.ko module
dm_raid45: Unknown symbol dm_dirty_log_create
dm_raid45: Unknown symbol rh_dec
dm_raid45: Unknown symbol rh_delay_by_region
dm_raid45: Unknown symbol rh_start_recovery
dm_raid45: Unknown symbol rh_exit
dm_raid45: Unknown symbol dm_put_device
dm_raid45: Unknown symbol dm_dirty_log_destroy
dm_raid45: Unknown symbol dm_io_client_create
dm_raid45: Unknown symbol rh_get_region_key
dm_raid45: Unknown symbol rh_region_to_sector
dm_raid45: Unknown symbol rh_update_states
dm_raid45: Unknown symbol rh_stop_recovery
dm_raid45: Unknown symbol rh_recovery_start
dm_raid45: Unknown symbol rh_state
dm_raid45: Unknown symbol rh_init
dm_raid45: Unknown symbol dm_unregister_target
dm_raid45: Unknown symbol dm_table_event
dm_raid45: Unknown symbol dm_put
dm_raid45: Unknown symbol rh_recovery_prepare
dm_raid45: Unknown symbol dm_io_client_destroy
dm_raid45: Unknown symbol dm_disk
dm_raid45: Unknown symbol dm_register_target
dm_raid45: Unknown symbol rh_get_region_size
dm_raid45: Unknown symbol dm_table_get_mode
dm_raid45: Unknown symbol dm_table_get_md
dm_raid45: Unknown symbol dm_io
dm_raid45: Unknown symbol rh_sector_to_region
dm_raid45: Unknown symbol dm_get_device
dm_raid45: Unknown symbol rh_recovery_end
dm_raid45: Unknown symbol rh_flush
dm_raid45: Unknown symbol rh_inc
insmod: error inserting '/lib/dm-raid45.ko': -1 Unknown symbol in module
Scanning and configuring dmraid supported devices
Creating root device.
Mounting root filesystem.
mount: could not find filesystem '/dev/root'
Setting up other filesystems.
Setting up new root fs
setuproot: moving /dev failed: No such file or directory
no fstab.sys, mounting internal defaults
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
switchroot: mount failed: No such file or directory
Kernel panic - not syncing: Attempted to kill init!
The template file is:
name: glidein-bg-3.6
summary: Just Enough Operating System based on centos5
version: 3
release: 6
# remove swap partition
os:
name: sl
version: 5
hardware:
cpus: 8
memory: 16384
partitions:
"/":
size: 10
repos:
- name: "EPEL5-x86_64"
baseurl: "http://mirror.bytemark.co.uk/fedora/epel/5/x86_64/"
- name: "EMI1-base"
baseurl: "http://emisoft.web.cern.ch/emisoft/dist/EMI/1/sl5/#BASE_ARCH#/base"
- name: "EMI1-updates"
baseurl: "http://emisoft.web.cern.ch/emisoft/dist/EMI/1/sl5/#BASE_ARCH#/updates"
- name: "EMI1-third-party"
baseurl: "http://emisoft.web.cern.ch/emisoft/dist/EMI/1/sl5/$basearch/third-party"
- name: "CVMFS"
baseurl: "http://cvmrepo.web.cern.ch/cvmrepo/yum/cvmfs/x86_64"
- name: "glideinWMS-vm"
baseurl: "http://heppc48.hep.ph.ic.ac.uk/glideinWMS"
- name: "EMI1-trustanchors"
baseurl: "http://repository.egi.eu/sw/production/cas/1/current/"
packages:
- @core
- curl
- dhclient
- fuse
- iputils
- libxslt
- ntp
- openssh
- openssh-server
- openssh-clients
- puppet
- python
- sysklogd
- tar
- vim-enhanced
- util-linux
- wget
- xterm
- xorg-x11-xauth
- yum
- yum-priorities
- ca-policy-egi-core
- fetch-crl
- cvmfs
- glideinwms-vm-core
- glideinwms-vm-ec2
files:
"/etc/rc.d":
- "rc.local"
"/etc/ssh":
- "sshd_config"
"/etc/sysconfig":
- "ephemeral_disks"
- "clock"
"/etc/init.d":
- "mount_ephemeral"
"/etc/cvmfs":
- "default.local"
"/etc/cvmfs/config.d":
- "cms.cern.ch.conf"
"/etc":
- "fuse.conf"
- "autofs.master"
"/root":
- "cvmfs.pp"
- "storage.xml"
post:
base:
- "chmod +x /etc/rc.d/rc.local"
- "/sbin/chkconfig mount_ephemeral on"
- "chmod +w /etc/sudoers; sed -i 's/Defaults requiretty/#Defaults requiretty/g' /etc/sudoers; chmod -w /etc/sudoers"
- "cp -f /usr/share/zoneinfo/Europe/Zurich /etc/localtime"
- "yum -y install emi-wn"
- "mkdir /data"