1 Reply Latest reply on Dec 7, 2012 7:10 AM by Adam Huffman

    Boot failure, related to module symbols?

    Adam Huffman Newbie

      I'm trying to build Scientific Linux 5.8 appliances.  In the past week or so, while the appliances have built normally, they've failed at boot.


      There seems to be a problem with the module symbols, meaning that block and filesystem modules aren't loaded and so the kernel can't find /dev/root.


      Here's an example boot log:


      Linux version 2.6.18-308.20.1.el5 (brewbuilder@norob.fnal.gov) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-52)) #1 SMP Tue Nov 13 11:54:02 EST 2012

      Command line: ro root=LABEL=79d3d2d4 console=tty0 console=ttyS0

      BIOS-provided physical RAM map:

      BIOS-e820: 0000000000010000 - 000000000009fc00 (usable)

      BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)

      BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)

      BIOS-e820: 0000000000100000 - 000000003fffe000 (usable)

      BIOS-e820: 000000003fffe000 - 0000000040000000 (reserved)

      BIOS-e820: 00000000feffc000 - 00000000ff000000 (reserved)

      BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)

      DMI 2.4 present.

      kvm-clock: cpu 0, msr 7eff:804a7401, boot clock

      No NUMA configuration found

      Faking a node at 0000000000000000-000000003fffe000

      Bootmem setup node 0 0000000000000000-000000003fffe000

      Memory for crash kernel (0x0 to 0x0) notwithin permissible range

      disabling kdump

      ACPI: PM-Timer IO Port: 0xb008

      ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)

      Processor #0 6:2 APIC version 20

      ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])

      ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])

      IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23

      ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)

      ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)

      ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)

      ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)

      ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)

      Setting APIC routing to physical flat

      ACPI: HPET id: 0x8086a201 base: 0xfed00000

      Using ACPI (MADT) for SMP configuration information

      Nosave address range: 000000000009f000 - 00000000000a0000

      Nosave address range: 00000000000a0000 - 00000000000f0000

      Nosave address range: 00000000000f0000 - 0000000000100000

      Allocating PCI resources starting at 50000000 (gap: 40000000:beffc000)

      SMP: Allowing 1 CPUs, 0 hotplug CPUs

      kvm-clock: cpu 0, msr 0:1735401, primary cpu clock

      Built 1 zonelists.  Total pages: 256944

      Kernel command line: ro root=LABEL=79d3d2d4 console=tty0 console=ttyS0

      Initializing CPU#0

      PID hash table entries: 4096 (order: 12, 32768 bytes)

      Using TSC for driving interrupts

      Console: colour VGA+ 80x25

      Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)

      Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)

      Checking aperture...

      ACPI: DMAR not present

      Memory: 1023408k/1048568k available (2620k kernel code, 24708k reserved, 1673k data, 224k init)

      Calibrating delay loop (skipped), value calculated using timer frequency.. 6784.58 BogoMIPS (lpj=3392292)

      Security Framework v1.0.0 initialized

      SELinux:  Initializing.

      selinux_register_security:  Registering secondary module capability

      Capability LSM initialized as secondary

      Mount-cache hash table entries: 256

      CPU: L1 I cache: 32K, L1 D cache: 32K

      CPU: L2 cache: 4096K

      SMP alternatives: switching to UP code

      Freeing SMP alternatives: 32k freed

      ACPI: Core revision 20060707

      Using local APIC timer interrupts.

      Detected 62.500 MHz APIC timer.

      Brought up 1 CPUs

      time.c: Using 100.000000 MHz WALL KVM GTOD KVM timer.

      time.c: Detected 3392.292 MHz processor.

      checking if image is initramfs... it is

      Freeing initrd memory: 2471k freed

      NET: Registered protocol family 16

      ACPI: bus type pci registered

      PCI: Using configuration type 1

      ACPI: Interpreter enabled

      ACPI: Using IOAPIC for interrupt routing

      ACPI: No dock devices found.

      ACPI: PCI Root Bridge [PCI0] (0000:00)

      PCI quirk: region b000-b03f claimed by PIIX4 ACPI

      PCI quirk: region b100-b10f claimed by PIIX4 SMB

      ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)

      ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)

      ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)

      ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)

      ACPI: PCI Interrupt Link [LNKS] (IRQs 9) *0, disabled.

      Linux Plug and Play Support v0.97 (c) Adam Belay

      pnp: PnP ACPI init

      pnp: PnP ACPI: found 7 devices

      usbcore: registered new driver usbfs

      usbcore: registered new driver hub

      PCI: Using ACPI for IRQ routing

      PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report

      NetLabel: Initializing

      NetLabel:  domain hash size = 128

      NetLabel:  protocols = UNLABELED CIPSOv4

      NetLabel:  unlabeled traffic allowed by default

      hpet0: at MMIO 0xfed00000 (virtual 0xffffffffff5fe000), IRQs 2, 8, 0

      hpet0: 3 64-bit timers, 100000000 Hz

      ACPI: DMAR not present

      PCI-GART: No AMD northbridge found.

      NET: Registered protocol family 2

      IP route cache hash table entries: 32768 (order: 6, 262144 bytes)

      TCP established hash table entries: 131072 (order: 9, 2097152 bytes)

      TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)

      TCP: Hash tables configured (established 131072 bind 65536)

      TCP reno registered

      audit: initializing netlink socket (disabled)

      type=2000 audit(1354799734.324:1): initialized

      Total HugeTLB memory allocated, 0

      VFS: Disk quotas dquot_6.5.1

      Dquot-cache hash table entries: 512 (order 0, 4096 bytes)

      Initializing Cryptographic API

      alg: No test for crc32c (crc32c-generic)

      ksign: Installing public key data

      Loading keyring

      - Added public key 6E36AF2ED02D2A5A

      - User ID: Red Hat, Inc. (Kernel Module GPG key)

      io scheduler noop registered

      io scheduler anticipatory registered

      io scheduler deadline registered

      io scheduler cfq registered (default)

      Limiting direct PCI/PCI transfers.

      PCI: PIIX3: Enabling Passive Release on 0000:00:01.0

      Activating ISA DMA hang workarounds.

      pci_hotplug: PCI Hot Plug PCI Core version: 0.5

      Real Time Clock Driver v1.12ac

      Non-volatile memory driver v1.2

      Linux agpgart interface v0.101 (c) Dave Jones

      Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled

      �serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A

      00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A

      brd: module loaded

      Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2

      ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx

      PIIX3: IDE controller at PCI slot 0000:00:01.1

      PIIX3: chipset revision 0

      PIIX3: not 100% native mode: will probe irqs later

          ide0: BM-DMA at 0xc0e0-0xc0e7, BIOS settings: hda:pio, hdb:pio

          ide1: BM-DMA at 0xc0e8-0xc0ef, BIOS settings: hdc:pio, hdd:pio

      ide-floppy driver 0.99.newide

      usbcore: registered new driver hiddev

      usbcore: registered new driver usbhid

      drivers/usb/input/hid-core.c: v2.6:USB HID core driver

      PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12

      serio: i8042 KBD port at 0x60,0x64 irq 1

      serio: i8042 AUX port at 0x60,0x64 irq 12

      mice: PS/2 mouse device common for all mice

      md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27

      md: bitmap version 4.39

      TCP bic registered

      Initializing IPsec netlink socket

      NET: Registered protocol family 1

      NET: Registered protocol family 17

      input: AT Translated Set 2 keyboard as /class/input/input0

      ACPI: (supports S5)

      Initalizing network drop monitor service

      Freeing unused kernel memory: 224k freed

      Write protecting the kernel read-only data: 531k

      Red Hat nash version starting

      Mounting proc filesystem

      Mounting sysfs filesystem

      Creating /dev

      Creating initial device nodes

      Setting up hotplug.

      input: ImExPS/2 Generic Explorer Mouse as /class/input/input1

      Creating block device nodes.

      Loading mptspi.ko module

      mptspi: Unknown symbol mpt_deregister

      mptspi: Unknown symbol mpt_reset_deregister

      mptspi: Unknown symbol mptscsih_host_attrs

      mptspi: Unknown symbol mptscsih_qcmd

      mptspi: Unknown symbol mptscsih_bios_param

      mptspi: Unknown symbol mptscsih_io_done

      mptspi: Unknown symbol mptscsih_slave_configure

      mptspi: Unknown symbol mpt_config

      mptspi: Unknown symbol mptscsih_taskmgmt_complete

      mptspi: Unknown symbol mptscsih_ioc_reset

      mptspi: Unknown symbol mpt_free_msg_frame

      mptspi: Unknown symbol mptscsih_remove

      mptspi: Unknown symbol mptscsih_bus_reset

      mptspi: Unknown symbol mptscsih_is_phys_disk

      mptspi: Unknown symbol spi_display_xfer_agreement

      mptspi: Unknown symbol mptscsih_proc_info

      mptspi: Unknown symbol __scsi_iterate_devices

      mptspi: Unknown symbol mptscsih_host_reset

      mptspi: Unknown symbol spi_release_transport

      mptspi: Unknown symbol mpt_put_msg_frame

      mptspi: Unknown symbol mpt_register

      mptspi: Unknown symbol mpt_event_deregister

      mptspi: Unknown symbol mpt_findImVolumes

      mptspi: Unknown symbol mptscsih_scandv_complete

      mptspi: Unknown symbol scsi_scan_host

      mptspi: Unknown symbol mptscsih_resume

      mptspi: Unknown symbol mptscsih_raid_id_to_num

      mptspi: Unknown symbol mpt_event_register

      mptspi: Unknown symbol mptscsih_event_process

      mptspi: Unknown symbol scsi_add_host

      mptspi: Unknown symbol mptscsih_suspend

      mptspi: Unknown symbol scsi_is_host_device

      mptspi: Unknown symbol mptscsih_slave_destroy

      mptspi: Unknown symbol scsi_print_command

      mptspi: Unknown symbol mptscsih_change_queue_depth

      mptspi: Unknown symbol scsi_scan_target

      mptspi: Unknown symbol spi_attach_transport

      mptspi: Unknown symbol mptscsih_shutdown

      mptspi: Unknown symbol mpt_get_msg_frame

      mptspi: Unknown symbol mptscsih_dev_reset

      mptspi: Unknown symbol mpt_attach

      mptspi: Unknown symbol mpt_reset_register

      mptspi: Unknown symbol mptscsih_info

      mptspi: Unknown symbol mptscsih_abort

      mptspi: Unknown symbol mpt_HardResetHandler

      mptspi: Unknown symbol scsi_host_alloc

      mptspi: Unknown symbol spi_dv_device

      mptspi: Unknown symbol scsi_device_lookup_by_target

      mptspi: Unknown symbol mptscsih_IssueTaskMgmt

      insmod: error inserting '/lib/mptspi.ko': -1 Unknown symbol in module

      Loading virtio_pci.ko module

      virtio_pci: Unknown symbol vring_transport_features

      virtio_pci: Unknown symbol vring_interrupt

      virtio_pci: Unknown symbol vring_new_virtqueue

      virtio_pci: Unknown symbol unregister_virtio_device

      virtio_pci: Unknown symbol register_virtio_device

      virtio_pci: Unknown symbol vring_del_virtqueue

      insmod: error inserting '/lib/virtio_pci.ko': -1 Unknown symbol in module

      Loading virtio_blk.ko module

      virtio_blk: Unknown symbol register_virtio_driver

      virtio_blk: Unknown symbol unregister_virtio_driver

      virtio_blk: Unknown symbol virtio_check_driver_offered_feature

      insmod: error inserting '/lib/virtio_blk.ko': -1 Unknown symbol in module

      Loading ehci-hcd.ko module

      Loading ohci-hcd.ko module

      Loading uhci-hcd.ko module

      USB Universal Host Controller Interface driver v3.0

      ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11

      ACPI: PCI Interrupt 0000:00:01.2[D] -> Link [LNKD] -> GSI 11 (level, high) -> IRQ 11

      uhci_hcd 0000:00:01.2: UHCI Host Controller

      uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1

      uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c040

      usb usb1: configuration #1 chosen from 1 choice

      hub 1-0:1.0: USB hub found

      hub 1-0:1.0: 2 ports detected

      Loading ext4.ko module

      ext4: Unknown symbol jbd2_journal_forget

      ext4: Unknown symbol jbd2_journal_stop

      ext4: Unknown symbol jbd2_journal_restart

      ext4: Unknown symbol jbd2_journal_load

      ext4: Unknown symbol jbd2_journal_check_available_features

      ext4: Unknown symbol jbd2_journal_set_features

      ext4: Unknown symbol jbd2_journal_unlock_updates

      ext4: Unknown symbol jbd2_journal_file_inode

      ext4: Unknown symbol jbd2_journal_get_create_access

      ext4: Unknown symbol jbd2_journal_init_dev

      ext4: Unknown symbol jbd2_journal_clear_features

      ext4: Unknown symbol jbd2_journal_init_jbd_inode

      ext4: Unknown symbol jbd2_journal_get_write_access

      ext4: Unknown symbol jbd2_log_start_commit

      ext4: Unknown symbol jbd2_journal_force_commit_nested

      ext4: Unknown symbol jbd2_journal_clear_err

      ext4: Unknown symbol jbd2_journal_init_inode

      ext4: Unknown symbol jbd2_journal_abort

      ext4: Unknown symbol jbd2_journal_blocks_per_page

      ext4: Unknown symbol jbd2_journal_update_format

      ext4: Unknown symbol jbd2_journal_release_jbd_inode

      ext4: Unknown symbol jbd2_journal_revoke

      ext4: Unknown symbol crc16

      ext4: Unknown symbol jbd2_journal_start_commit

      ext4: Unknown symbol jbd2_journal_extend

      ext4: Unknown symbol jbd2_journal_dirty_metadata

      ext4: Unknown symbol jbd2_journal_try_to_free_buffers

      ext4: Unknown symbol jbd2_journal_wipe

      ext4: Unknown symbol jbd2_journal_start

      ext4: Unknown symbol jbd2_journal_destroy

      ext4: Unknown symbol jbd2_journal_release_buffer

      ext4: Unknown symbol jbd2_journal_force_commit

      ext4: Unknown symbol jbd2_journal_get_undo_access

      ext4: Unknown symbol jbd2_log_wait_commit

      ext4: Unknown symbol jbd2_journal_invalidatepage

      ext4: Unknown symbol jbd2_journal_lock_updates

      ext4: Unknown symbol jbd2_journal_errno

      ext4: Unknown symbol jbd2_journal_begin_ordered_truncate

      ext4: Unknown symbol jbd2_journal_flush

      insmod: error inserting '/lib/ext4.ko': -1 Unknown symbol in module

      Loading dm-mem-cache.ko module

      Loading dm-region_hash.ko module

      dm_region_hash: Unknown symbol dm_dirty_log_destroy

      insmod: error inserting '/lib/dm-region_hash.ko': -1 Unknown symbol in module

      Loading dm-message.ko module

      Loading dm-raid45.ko module

      dm_raid45: Unknown symbol dm_dirty_log_create

      dm_raid45: Unknown symbol rh_dec

      dm_raid45: Unknown symbol rh_delay_by_region

      dm_raid45: Unknown symbol rh_start_recovery

      dm_raid45: Unknown symbol rh_exit

      dm_raid45: Unknown symbol dm_put_device

      dm_raid45: Unknown symbol dm_dirty_log_destroy

      dm_raid45: Unknown symbol dm_io_client_create

      dm_raid45: Unknown symbol rh_get_region_key

      dm_raid45: Unknown symbol rh_region_to_sector

      dm_raid45: Unknown symbol rh_update_states

      dm_raid45: Unknown symbol rh_stop_recovery

      dm_raid45: Unknown symbol rh_recovery_start

      dm_raid45: Unknown symbol rh_state

      dm_raid45: Unknown symbol rh_init

      dm_raid45: Unknown symbol dm_unregister_target

      dm_raid45: Unknown symbol dm_table_event

      dm_raid45: Unknown symbol dm_put

      dm_raid45: Unknown symbol rh_recovery_prepare

      dm_raid45: Unknown symbol dm_io_client_destroy

      dm_raid45: Unknown symbol dm_disk

      dm_raid45: Unknown symbol dm_register_target

      dm_raid45: Unknown symbol rh_get_region_size

      dm_raid45: Unknown symbol dm_table_get_mode

      dm_raid45: Unknown symbol dm_table_get_md

      dm_raid45: Unknown symbol dm_io

      dm_raid45: Unknown symbol rh_sector_to_region

      dm_raid45: Unknown symbol dm_get_device

      dm_raid45: Unknown symbol rh_recovery_end

      dm_raid45: Unknown symbol rh_flush

      dm_raid45: Unknown symbol rh_inc

      insmod: error inserting '/lib/dm-raid45.ko': -1 Unknown symbol in module

      Scanning and configuring dmraid supported devices

      Creating root device.

      Mounting root filesystem.

      mount: could not find filesystem '/dev/root'

      Setting up other filesystems.

      Setting up new root fs

      setuproot: moving /dev failed: No such file or directory

      no fstab.sys, mounting internal defaults

      setuproot: error mounting /proc: No such file or directory

      setuproot: error mounting /sys: No such file or directory

      Switching to new root and running init.

      unmounting old /dev

      unmounting old /proc

      unmounting old /sys

      switchroot: mount failed: No such file or directory

      Kernel panic - not syncing: Attempted to kill init!


      The template file is:


      name: glidein-bg-3.6

      summary: Just Enough Operating System based on centos5

      version: 3

      release: 6

      # remove swap partition



        name: sl

        version: 5



        cpus: 8

        memory: 16384



            size: 10



        - name: "EPEL5-x86_64"

          baseurl: "http://mirror.bytemark.co.uk/fedora/epel/5/x86_64/"

        - name: "EMI1-base"

          baseurl: "http://emisoft.web.cern.ch/emisoft/dist/EMI/1/sl5/#BASE_ARCH#/base"

        - name: "EMI1-updates"

          baseurl: "http://emisoft.web.cern.ch/emisoft/dist/EMI/1/sl5/#BASE_ARCH#/updates"

        - name: "EMI1-third-party"

          baseurl: "http://emisoft.web.cern.ch/emisoft/dist/EMI/1/sl5/$basearch/third-party"

        - name: "CVMFS"

          baseurl: "http://cvmrepo.web.cern.ch/cvmrepo/yum/cvmfs/x86_64"

        - name: "glideinWMS-vm"

          baseurl: "http://heppc48.hep.ph.ic.ac.uk/glideinWMS"

        - name: "EMI1-trustanchors"

          baseurl: "http://repository.egi.eu/sw/production/cas/1/current/"


        - @core

        - curl

        - dhclient

        - fuse

        - iputils

        - libxslt

        - ntp

        - openssh

        - openssh-server

        - openssh-clients

        - puppet

        - python

        - sysklogd

        - tar

        - vim-enhanced

        - util-linux

        - wget

        - xterm

      - xorg-x11-xauth

        - yum

        - yum-priorities

        - ca-policy-egi-core

        - fetch-crl

        - cvmfs

        - glideinwms-vm-core

        - glideinwms-vm-ec2




          - "rc.local"



          - "sshd_config"



          - "ephemeral_disks"

          - "clock"



          - "mount_ephemeral"



          - "default.local"



          - "cms.cern.ch.conf"



          - "fuse.conf"

          - "autofs.master"



          - "cvmfs.pp"

          - "storage.xml"





          - "chmod +x /etc/rc.d/rc.local"

          - "/sbin/chkconfig mount_ephemeral on"

          - "chmod +w /etc/sudoers; sed -i 's/Defaults    requiretty/#Defaults    requiretty/g' /etc/sudoers; chmod -w /etc/sudoers"

          - "cp -f /usr/share/zoneinfo/Europe/Zurich /etc/localtime"

          - "yum -y install emi-wn"

          - "mkdir /data"