Another story that kept me awake like this one will be the todays topic. By some funny coincidence it’s also about the same customer. This time I was called because a very important ACFS filesystem could not be mounted. The customer tried to change the group- and user-ID of the Grid Infrastructure Home which was not a good idea at all. So they reverted all the changes but an ACFS filesystem did not come up again. That was the moment they called me.
So I first checked the ASM volumes. There was only one volume in “disabled” state.
[oracle@server02 ~]$ asmcmd volinfo --all
Diskgroup Name: DATA
Volume Name: U27
Volume Device: /dev/asm/u27-510
State: DISABLED
Size (MB): 31815680
Resize Unit (MB): 64
Redundancy: UNPROT
Stripe Columns: 8
Stripe Width (K): 1024
Usage: ACFS
Mountpath: /u27
The obvious idea was to re-enable to volume. But that didn’t succeed.
[oracle@server02 ~]$ asmcmd volenable --all
ORA-15032: not all alterations performed
ORA-15489: conflicting volume already open on this node (DBD ERROR: OCIStmtExecute)
[oracle@server02 ~]$ oerr ora 15489
15489, 00000, "conflicting volume already open on this node"
// *Cause: An ASM volume was not closed prior to instance shutdown
// on node specified. It is conflicting with the volume that
// is to be enabled.
// *Action: Close the ASM volume either via unmount or node restart and
// retry the operation.
//
Ok, we have a disabled volume which is still in use somehow. It really is, see this:
[root@server02 ~]# mount|grep u27
/dev/asm/u27-510 on /u27 type acfs (rw,relatime,device,ordered)
[root@server02 ~]# ll /u27
ls: cannot open directory /u27: Input/output error
[root@server02 ~]# umount /u27
umount: /u27: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount.acfs: CLSU-00107: operating system function: OfsWaitProc; failed with error data: 32; at location: OWPR_1
umount.acfs: CLSU-00101: operating system error message: Broken pipe
umount.acfs: ACFS-04151: unmount of mount point /u27 failed
I can’t use the filesystem and cannot unmount the filesystem either. So what is preventing me from unmounting the filesytem?
[root@server02 ~]# fuser /u27
[root@server02 ~]# lsof |grep /u27
[root@server02 ~]# umount /u27
umount: /u27: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount.acfs: CLSU-00107: operating system function: OfsWaitProc; failed with error data: 32; at location: OWPR_1
umount.acfs: CLSU-00101: operating system error message: Broken pipe
umount.acfs: ACFS-04151: unmount of mount point /u27 failed
[root@server02 ~]# lsof |grep u27-510
At this point the customer and I agreed to simply reboot the system. Beside the fact, that the system did not come up in the first place due to some misconfiguration in the grub.conf, this finally solved the issue.
I just needed to load the kernel modules since this was a stand-alone installation, enable the volume and finally mount the filesysem again.
[root@server02 ~]# lsmod|grep advm
[root@server02 ~]# $GRID_HOME/bin/acfsload start
[root@server02 ~]# su - oracle
[oracle@server02 ~]$ asmcmd volenable --all
[root@server02 ~]# /sbin/fsck -a -y -t acfs /dev/asm/u27-510
[root@server02 ~]# mount -t acfs /dev/asm/u27-510 /u27
Customer happy, their Golden Gate Trails where back again.
Marco's DBA Blog 