umount: /sysroot/proc: device is busy.

[bfulton]

I'm seeing problems at the end of running boxgrinder-build while trying to umount /sysroot/proc:

{code}

D, [2010-09-01T11:44:39.294862 #22644] DEBUG -- : Repositories installed.

I, [2010-09-01T11:44:39.294970 #22644]  INFO -- : Post operations executed.

T, [2010-09-01T11:44:39.295117 #22644] TRACE -- : Closing guestfs...

T, [2010-09-01T11:44:39.295399 #30828] TRACE -- : GFS:

T, [2010-09-01T11:44:39.295566 #30828] TRACE -- : GFS: sync

T, [2010-09-01T11:44:39.295822 #30828] TRACE -- : GFS: send_to_daemon: 0x27e8020 g->state = 3, n = 28

T, [2010-09-01T11:44:39.295952 #30828] TRACE -- : GFS: recv_from_daemon: 0x27e8020 g->state = 3, size_rtn = 0x7fff1616f57c, buf_rtn = 0x7fff1616f570

T, [2010-09-01T11:44:39.807575 #30828] TRACE -- : GFS: proc 2 (sync) took 0.33 seconds

T, [2010-09-01T11:44:39.843633 #30828] TRACE -- : GFS: umount_all

T, [2010-09-01T11:44:39.843758 #30828] TRACE -- : GFS: send_to_daemon: 0x27e8020 g->state = 3, n = 28

T, [2010-09-01T11:44:39.843871 #30828] TRACE -- : GFS: recv_from_daemon: 0x27e8020 g->state = 3, size_rtn = 0x7fff1616f57c, buf_rtn = 0x7fff1616f570

T, [2010-09-01T11:44:39.845242 #30828] TRACE -- : GFS: mount

T, [2010-09-01T11:44:39.901017 #30828] TRACE -- : GFS: umount /sysroot/proc

T, [2010-09-01T11:44:39.945086 #30828] TRACE -- : GFS: umount: /sysroot/proc: device is busy.

T, [2010-09-01T11:44:39.947244 #30828] TRACE -- : GFS: (In some cases useful info about processes that use

T, [2010-09-01T11:44:39.954370 #30828] TRACE -- : GFS: the device is found by lsof(8) or fuser(1))

T, [2010-09-01T11:44:39.958635 #30828] TRACE -- : GFS: guestfsd: error: umount: /sysroot/proc: umount: /sysroot/proc: device is busy.

T, [2010-09-01T11:44:39.960792 #30828] TRACE -- : GFS: (In some cases useful info about processes that use

T, [2010-09-01T11:44:39.962644 #30828] TRACE -- : GFS: the device is found by lsof(8) or fuser(1))

T, [2010-09-01T11:44:39.967018 #30828] TRACE -- : GFS: proc 47 (umount_all) took 0.12 seconds

error: umount_all: umount: /sysroot/proc: umount: /sysroot/proc: device is busy.

        (In some cases useful info about processes that use

         the device is found by lsof(8) or fuser(1)). Use --trace to view backtrace

closing guestfs handle 0x27e8020 (state 2)

kill_subprocess

sending SIGTERM to process 30833

[root@domU-XX-XX-XX-XX-XX-XX boxgrinder]# mount

/dev/xvda1 on / type ext3 (rw)

none on /proc type proc (rw)

none on /sys type sysfs (rw)

none on /dev/pts type devpts (rw,gid=5,mode=620)

none on /dev/shm type tmpfs (rw)

/dev/xvdb on /mnt type ext3 (rw)

/dev/xvdc on /data type ext3 (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

{code}

It looks like everything went fine during the build -- no obvious errors and when I use guestfish on the tmp .raw, it looks correct -- but perhaps there is a race in the process cleanup and unmounting?

Any thoughts?

[goldmann]

What libguestfs version you use? What OS you use?

--Marek

[rwmjones]

It would be good to see the trace and/or debug outputs (trace in particular).  See:

http://libguestfs.org/FAQ.html#debug

This seems to be down to some sort of leaked resource in the appliance.  Perhaps a daemon gets started during some rpm install or other run command which holds open something in /proc, preventing it from getting unmounted after a command runs.  You can see the logic we use around running commands here:

http://git.annexia.org/?p=libguestfs.git;a=blob;f=daemon/command.c;h=48bed2de0262760f49e944b119632f6907c6e459;hb=HEAD#l50

[goldmann]

Thanks Rich. I think the info above is a trace output from libguestfs, but, Bright, could you please paste the full log for this build located in log/boxgrinder.log? Thanks!

--Marek

[bfulton]

Thanks Rich.  Thanks Markek.  Rich's hunch was correct: the Sun JDK RPM %post installs and starts /etc/init.d/jexec which mounts /proc/sys/fs/binfmt_misc, hampering /proc umount.

In case anyone else runs into this specific issue, one might "grind" the Sun JDK like:

post:

  base:

    - |

      rpm -Uvh jdk-6u21-linux-amd64.rpm

      /etc/init.d/jexec stop

      umount /proc/sys/fs/binfmt_misc

...where jdk-6u21-linux-amd64.rpm is the path or URL to the RPM which unbundles when you accept the license agreement of the downloaded .rpm.bin.

[goldmann]

Thank you Bright for lettng us know!

[rwmjones]

Good bit of detective work there :-)