Marco Pachaly-Mischke 9:17 am on June 13, 2016
Tags: clusters ( 22 ), RAC ( 8 )

Using ASM Filter Driver right from the beginning

Preface

If you are running Oracle RAC, then configuring the shared storage is one of the main preinstallation tasks that needs to be done. You need to configure multipathing and make sure that the device name that will be used for ASM is always the same. And you must set permissions and ownership for these devices. On Linux you can use ASMlib for that. It stamps the devices so that it can identify them, provides an unique and consistent name for ASM and sets propper permissions for the devices. But it still possible for other processes to write to these devices, using “dd” for instance.

Now there is Oracle Grid Infrastructure 12c which introduces a replacement for ASMlib called ASM Filter Driver (AFD). Basically it does the same things as ASMlib but in addition to that it is able to block write operations from other processes than Oracle’s own ones.

So that is a good thing and I wanted to use it for a new cluster that I should set up. And that is where the trouble starts. Beside the fact that there were some bugs in the initial versions of AFD from which most got fixed by the April 2016 PSU, AFD is installed as part of Grid Infrastructure. You can read that in the Automatic Storage Management Docs. It states the following:

After installation of Oracle Grid Infrastructure, you can optionally configure Oracle ASMFD for your system.

What? After installation? But I need it right from the beginning to use it for my initial disk group. How about that? There is a MOS note How to Install ASM Filter Driver in a Linux Environment Without Having Previously Installed ASMLIB (Doc ID 2060259.1) but this Whitepaper also asumes that Grid Infrastructure is already installed.

But as you can read from this blog posts title, there is a way to use AFD from scratch, but it is not really straight forward.

1. Install Grid Infrastructure Software

First step is to install Grid Infrastructure as a software only installation. That implies that you have to do it on all nodes that should form the future cluster. I did that on the first node, saved the response file and did a silent install on the other nodes.

[oracle@vm140 ~] ./runInstaller -silent -responseFile /home/oracle/stage/grid/grid.rsp -ignorePrereq

At the end of the installation you need to run the “orainstRoot.sh” script which itself provides two other root scripts which configure either a cluster or a stand alone server:

[root@vm140 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@vm140 ~]# /u01/app/12.1.0.2/grid/root.sh
Performing root user operation.

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/12.1.0.2/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

To configure Grid Infrastructure for a Stand-Alone Server run the following command as the root user:
/u01/app/12.1.0.2/grid/perl/bin/perl -I/u01/app/12.1.0.2/grid/perl/lib -I/u01/app/12.1.0.2/grid/crs/install /u01/app/12.1.0.2/grid/crs/install/roothas.pl

To configure Grid Infrastructure for a Cluster execute the following command as oracle user:
/u01/app/12.1.0.2/grid/crs/config/config.sh
This command launches the Grid Infrastructure Configuration Wizard. The wizard also supports silent operation, and the parameters can be passed through the response file that is available in the installation media.

For the moment, we do not run any of these scripts.

2. Patching Grid Infrastructure software

Next step is to patch GI software to get the latest version for AFD. Simply update OPatch on all nodes and use “opatchauto” to patch GI home. You need to specify the ORACLE_HOME path using “-oh” parameter to patch an unconfigured Grid Infrastructure home.

[root@vm140 ~]# export ORACLE_HOME=/u01/app/12.1.0.2/grid
[root@vm140 ~]# export PATH=$ORACLE_HOME/OPatch:$PATH
[root@vm140 ~]# opatch version
OPatch Version: 12.1.0.1.12

OPatch succeeded.

[root@vm140 ~]# opatchauto apply /home/oracle/stage/22646084 -oh $ORACLE_HOME

[...]

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:vm140
CRS Home:/u01/app/12.1.0.2/grid
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /home/oracle/stage/22646084/21436941
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-06-10_13-54-11PM_1.log

Patch: /home/oracle/stage/22646084/22291127
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-06-10_13-54-11PM_1.log

Patch: /home/oracle/stage/22646084/22502518
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-06-10_13-54-11PM_1.log

Patch: /home/oracle/stage/22646084/22502555
Log: /u01/app/12.1.0.2/grid/cfgtoollogs/opatchauto/core/opatch/opatch2016-06-10_13-54-11PM_1.log


OPatchAuto successful.

You see that with the latest OPatch version there is no need to create an ocm.rsp resopnse file anymore.

3. Configure Restart

Configure Restart? Why? Because it sets up everything we need to use AFD but does not need any shared storage or other cluster related things like virtual IPs, SCANs and so on.
Therefore you use the script that was provided earlier by the “orainstRoot.sh” script. Do that on all nodes of the future cluster.

[root@vm140 ~]# /u01/app/12.1.0.2/grid/perl/bin/perl -I/u01/app/12.1.0.2/grid/perl/lib -I/u01/app/12.1.0.2/grid/crs/install /u01/app/12.1.0.2/grid/crs/install/roothas.pl

4. Deconfigure Restart

After Restart was configured, you can deconfigure it right away. Everything that is needed for AFD is being kept. The documentation for that is here.

[root@vm140 ~]# cd /u01/app/12.1.0.2/grid/crs/install/
[root@vm140 install]# ./roothas.sh -deconfig -force

5. Confiure ASM Filter Driver

Now you can finally start configuring AFD. The whitepaper from the MOS note mentioned at the beginning provides a good overview of what has to be done. Simply connect as “root”, set the environment and run the following:

[root@vm140 install]# $ORACLE_HOME/bin/asmcmd afd_configure
Connected to an idle instance.
AFD-627: AFD distribution files found.
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
Modifying resource dependencies - this may take some time.
ASMCMD-9524: AFD configuration failed 'ERROR: OHASD start failed'
[root@vm140 install]# $ORACLE_HOME/bin/asmcmd afd_state
Connected to an idle instance.
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'DISABLED' on host 'vm140'

Don’t care about the error and the message that is telling it failed. That is simply because there is no cluster at all at the moment.
As a final configuration step you need to set the discovery string for AFD so that it can find the disks you want to use. This is defined inside “/etc/afd.conf”:

[root@vm140 install]# cat /etc/afd.conf
afd_diskstring='/dev/xvd*'

The above steps need to be done on all servers of the future cluster.
Now that AFD is configured, you can start labeling your disks. Do this on only one node:

[root@vm140 install]# $ORACLE_HOME/bin/asmcmd afd_label GI /dev/xvdb1
Connected to an idle instance.
[root@vm140 install]# $ORACLE_HOME/bin/asmcmd afd_label DATA /dev/xvdc1
Connected to an idle instance.
[root@vm140 install]# $ORACLE_HOME/bin/asmcmd afd_label FRA /dev/xvdd1
Connected to an idle instance.

[root@vm140 install]# $ORACLE_HOME/bin/asmcmd afd_lsdsk
Connected to an idle instance.
--------------------------------------------------------------------------------
Label                     Filtering   Path
================================================================================
GI                         DISABLED   /dev/xvdb1
DATA                       DISABLED   /dev/xvdc1
FRA                        DISABLED   /dev/xvdd1

On all the other nodes just do a rescan of the disks:

[root@vm141 install]# $ORACLE_HOME/bin/asmcmd afd_scan
Connected to an idle instance.
[root@vm141 install]# $ORACLE_HOME/bin/asmcmd afd_lsdsk
Connected to an idle instance.
--------------------------------------------------------------------------------
Label                     Filtering   Path
================================================================================
GI                         DISABLED   /dev/xvdb1
DATA                       DISABLED   /dev/xvdc1
FRA                        DISABLED   /dev/xvdd1

That’s it.

6. Configure cluster with AFD

Finally, you can start configuring your new cluster and use AFD disks right from the beginning. You can now use the Cluster Configuration Assistant that was mentioned by “orainstRoot.sh” to set up your cluster.

[oracle@vm140 ~]$ /u01/app/12.1.0.2/grid/crs/config/config.sh

Follow the steps and you will see the well-known screens for setting up a cluster. At the point when you define the initial Grid Inftrastructure diskgroup you can now specify the “Discovery String”:

And, voila, you see the previously labeled disks:

And after you run the root scripts on all nodes, you’ll get a running cluster:

[root@vm140 bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
			   ONLINE  ONLINE       vm140                    STABLE
			   ONLINE  ONLINE       vm141                    STABLE
			   ONLINE  ONLINE       vm142                    STABLE
			   ONLINE  ONLINE       vm143                    STABLE
ora.GI.dg
			   ONLINE  ONLINE       vm140                    STABLE
			   ONLINE  ONLINE       vm141                    STABLE
			   ONLINE  ONLINE       vm142                    STABLE
			   OFFLINE OFFLINE      vm143                    STABLE
ora.LISTENER.lsnr
			   ONLINE  ONLINE       vm140                    STABLE
			   ONLINE  ONLINE       vm141                    STABLE
			   ONLINE  ONLINE       vm142                    STABLE
			   ONLINE  ONLINE       vm143                    STABLE
ora.net1.network
			   ONLINE  ONLINE       vm140                    STABLE
			   ONLINE  ONLINE       vm141                    STABLE
			   ONLINE  ONLINE       vm142                    STABLE
			   ONLINE  ONLINE       vm143                    STABLE
ora.ons
			   ONLINE  ONLINE       vm140                    STABLE
			   ONLINE  ONLINE       vm141                    STABLE
			   ONLINE  ONLINE       vm142                    STABLE
			   ONLINE  ONLINE       vm143                    STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
	  1        ONLINE  ONLINE       vm140                    STABLE
ora.MGMTLSNR
	  1        ONLINE  ONLINE       vm140                    169.254.231.166 192.
															 168.1.1,STABLE
ora.asm
	  1        ONLINE  ONLINE       vm140                    Started,STABLE
	  2        ONLINE  ONLINE       vm142                    Started,STABLE
	  3        ONLINE  ONLINE       vm141                    Started,STABLE
ora.cvu
	  1        ONLINE  ONLINE       vm140                    STABLE
ora.mgmtdb
	  1        ONLINE  ONLINE       vm140                    Open,STABLE
ora.oc4j
	  1        ONLINE  ONLINE       vm140                    STABLE
ora.scan1.vip
	  1        ONLINE  ONLINE       vm140                    STABLE
ora.vm140.vip
	  1        ONLINE  ONLINE       vm140                    STABLE
ora.vm141.vip
	  1        ONLINE  ONLINE       vm141                    STABLE
ora.vm142.vip
	  1        ONLINE  ONLINE       vm142                    STABLE
ora.vm143.vip
	  1        ONLINE  ONLINE       vm143                    STABLE
--------------------------------------------------------------------------------

And that’s it. Nothing more to do. Beside creating more disk groups and setting up databases. But that is simple compared to what we’ve done till now.

Ahmed Abdel Fattah 1:38 am on June 17, 2016

Hi Marco,

Is AFD available only for Linux?

LikeLike
- Marco Mischke 5:22 am on June 17, 2016
  
  Hi Ahmed,
  Yes, this is only available on Linux starting with 12.1.0.2: https://docs.oracle.com/database/121/OSTMG/GUID-2F5E344F-AFC2-4768-8C00-6F3C56302123.htm#OSTMG95729
  
  LikeLike
Peter 10:23 am on June 24, 2016

Hi Marco,
after applying CPU Jan2016 AFD won’t load on reboot.
I run in that issue after migrate from ASMLib to ASM Filter Driver. That’s realy heavy stuff!

see:
Doc ID 2115515.1
and
Patch 22717225: AFD_CONFGURE DOES NOT ADD START RC SCRIPT

Regards Peter

LikeLike
- Marco Mischke 10:41 am on June 24, 2016
  
  Hi Peter,
  that’s true. The fix for this bug was silently included in PSU 160419 but there is no hint in the list of fixed bugs.
  Regards,
  Marco
  
  LikeLike
Pingback: ASM Filter Driver CPU load – don’t care | Marco's DBA Blog
Dejan 4:33 pm on September 5, 2016

Hi Marco,

thanks for the great article!
I was following your instructions, but unfortunately I got this error message:

[root@dummy ~]# /u01/app/12.1.0/grid/bin/asmcmd afd_label DATA1_1 /dev/mapper/data_1_1p1
Connected to an idle instance.
ASMCMD-9513: ASM disk label set operation failed.

I am using Multipath and UDEV rules.
Do you maybe have an idea, what could be the cause of it?

Thanks & kind regards
Dejan

LikeLike
- Marco Mischke 7:06 am on September 6, 2016
  
  Hi Dejan,
  please check if the AFD driver is loaded using “asmcmd afd_state”. If it is loaded, are there already some disks? –> “asmcmd afd_lsdsk”. Did you change ownership and/or permissions of the devices in your UDEV rules? If yes, that’s not necessary for AFD.
  Are there any errors before the ASMCMD-9513 is raised? Should look like this:
  
  [root@vm101 ~]# asmcmd afd_label dings /dev/xvdx
  ORA-15227: could not perform label set/clear operation
  ORA-15031: disk specification ‘/dev/xvdx’ matches no disks (DBD ERROR: OCIStmtExecute)
  ASMCMD-9513: ASM disk label set operation failed.
  
  HTH,
  Marco
  
  LikeLike
Dejan 7:48 am on September 7, 2016

Hi Marco,

well, it seems, there was no problem with Oracle. 🙂 More likely there is a problem with the network between SAN storage and RAC nodes. I found these error messages in the logs and forwarded them to my colleagues (System & Storage Admins):

err(-9) – Error 27072, OS Error (Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 4294967295)

Sep 5 14:59:36 REPDWH iscsid: connect to 192.168.20.20:3260 failed (No route to host)

As soon as this issue is solved, I’ll proceed with the configuration and give you a feedback if it worked or not.

Thanks again & kind regards from Vienna!

Dejan

LikeLiked by 1 person
- Marco Mischke 7:59 am on September 7, 2016
  
  Hi Dejan,
  Thanks for the reply. So good luck and I hope you get things up and running.
  Kind regards,
  Marco
  
  LikeLike
Dejan 12:29 pm on September 8, 2016

Hi again,

after the issue with the network has been solved, I was able to configure the ASM Filter Driver.
But now, I am struggling with another error message after I run config.sh, at step “Prerequisites checks”:

PRVG-10524 failed to determine whether disk is managed by ASMLib

– Cause: An attempt to validate whether the indicated disk was managed by ASMLib failed.
– Action: Ensure that the ASMLib is correctly configured on all the cluster nodes and that the indicated disk is listed by ”oracleasm” on the Linux operating system platform.

Google search: 0 points
MyOracleSupport search: 0 points

Is it allowed to ignore it? 🙂

Kind regards
Dejan

LikeLike
- Marco Mischke 1:31 pm on September 8, 2016
  
  I tend to ignore warnings during test installations 🙂
  Can’t remember if there was such a warning in my setup. I would say, give it a try. Since you are not using ASMlib at all I am pretty shire it will work.
  Regards,
  Marco
  
  LikeLike
Peter 5:46 pm on October 24, 2016

Hi Marco,
If I label a disk with “asmcmd afd_label”, the disk is attached to “/dev/oracleafd/disks”. But the permissions are root:root instead of grid:asmadmin.
Only “systemctl restart afd.service” will apply the correct permissions from udev rules to labeled disk. “udev control –reload” and “udev trigger” are not enough. Is that expected or not relevant ?
see my post:
https://community.oracle.com/thread/3984318
Regards Peter

LikeLike
Marco Mischke 6:39 am on October 25, 2016

Hi Peter,
I saw that thread on OTN but as of now I can’t answer that completely. Currently I am in the process of setting up a new cluster, I already labeled the disks and as you said, they still are all owned by root. But I am able to choose those disks from the Cluster Assistant, all the labeled disks show up as “candidate”. Unfortunately I cannot proceed since there are some other prerequisites still missing. I’ll comment again and write an answer on OTN as soon as I know something new.
Cheers
Marco

LikeLike
Marco Mischke 10:10 am on October 25, 2016

Hi again,
just tried that on a customers cluster, when you label the disks, they are still owned by root. But you can add them to ASM diskgroups and use the newly labeled disks. Once the server reboots, the ownership changes to “oracle”.
Cheers,
Marco

LikeLike
Pingback: AFD label operation throws ORA-15031 | Marco's DBA Blog
Marick 2:25 pm on December 22, 2016

thank yooouuuu! from the beginning!!!

LikeLike
terry golden 11:34 pm on May 11, 2017

Great article! We’re trying to use oracle restart not rac as that would be a bit of over kill. Followed the steps x 6 and redid 2 to bring up a bare bones si has stack. The problem I’m having is that asm doesn’t see the afd disks despite setting up asm_diskstring = ‘AFD:*’ and the asmcmd afd_state saying the driver is loaded and afd_lsdks showing the drives as enabled. this is on 12.1.0.2 w jul ’16 bp. Looking at 12.2 system I see AFD shows up as a service on restart so maybe that’s the root issue, but then again I don’t see that in your example. Any thoughts would be greatly welcome.

sleepless in seattle,

terry

LikeLike
Norbert 5:15 pm on June 14, 2017

Hi Terry,

I only used ASMFD version 12.2. It seems that AFD became a cluster resource in 12.2 for the first time. I’ve had eight failed installations of Oracle Restart so far. The only way that it worked on my test system was to put a temporary rules file into the temporary rules directory /dev/.udev/rules.d that defined my two disks before configuring Restart (as root user):

KERNEL==”/dev/xvdc1″, OWNER=”hpasmdba”, GROUP=”hpasmadm”, MODE=”660″
KERNEL==”/dev/xvdd1″, OWNER=”hpasmdba”, GROUP=”hpasmadm”, MODE=”660″

Then have udevd set the permissions by:
# udevadm test /block/xvdc/xvdc1
# udevadm test /block/xvdd/xvdd1

At this point devices are set up correctly:
brw-rw—- 1 hpasmdba hpasmadm 202, 33 May 24 18:28 /dev/xvdc1
brw-rw—- 1 hpasmdba hpasmadm 202, 49 May 24 18:28 /dev/xvdd1

In 12.2 I completed the steps above before invoking gridSetup.sh.

Only then OUI finished without errors. Essentially the setup was migrated from pure udev to ASMFD. I could see the provisioned disks. It seems to me that gridSetup.sh will only succeed if rw permission is set for the GI owner before you run gridSetup.sh. Afterwards I noticed this file, also a temporary udev rules file obviously from ASMFD:
# cat /etc/udev/rules.d/53-afd.rules
#
# AFD devices
KERNEL==”oracleafd/.*”, OWNER=”hpasmdba”, GROUP=”hpasmadm”, MODE=”0775″
KERNEL==”oracleafd/*”, OWNER=”hpasmdba”, GROUP=”hpasmadm”, MODE=”0775″
KERNEL==”oracleafd/disks/*”, OWNER=”hpasmdba”, GROUP=”hpasmadm”, MODE=”0664″

# crsctl stat res -t
——————————————————————————–
Name Target State Server State details
——————————————————————————–
Local Resources
——————————————————————————–
ora.DATA.dg
ONLINE ONLINE dp-test-asdb03 STABLE
ora.asm
ONLINE ONLINE dp-test-asdb03 Started,STABLE
ora.ons
OFFLINE OFFLINE dp-test-asdb03 STABLE
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.cssd
1 ONLINE ONLINE dp-test-asdb03 STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.driver.afd
1 ONLINE ONLINE dp-test-asdb03 STABLE
ora.evmd
1 ONLINE ONLINE dp-test-asdb03 STABLE
——————————————————————————–

srvctl config asm
ASM home:
Password file: +DATA/orapwasm
Backup of Password file:
ASM listener:
Spfile: +DATA/ASM/ASMPARAMETERFILE/registry.253.944851183
ASM diskgroup discovery string: AFD:*

Maybe this helps with 12.1. Or you might use 12.2 GI instead of 12.1. It supports a 12.2 database I assume. Would need to check the compatibility matrix but normally GI can have a higher release than the RDBMS.

LikeLike
Sven Illert 11:19 am on June 15, 2017

Terry,

I would not recommend using UDEV rules to modify permissions of AFD devices as Norbert suggested. How did you set the asm diskstring? Did you use “asmcmd dsset”? This is the way I’d recommend.

Best Regards,
Sven

LikeLiked by 1 person
Sven Illert 11:40 am on June 15, 2017

For AFD installation in 12.2 have a look at the oracle database installation documentation:

http://docs.oracle.com/database/122/LADBI/installing-oracle-grid-infrastructure-for-a-standalone-server-with-a-new-database-installation.htm#LADBI-GUID-0B1CEE8C-C893-46AA-8A6A-7B5FAAEC72B3

AFD is now supported during installation.

LikeLike
terry golden 1:22 pm on June 15, 2017

yes.

The problem seemed to be related to lack of acess to /opt subdirectory down. Granting access cured that problem, and that’s when the fun started. Almost immediately I coudn’t create disk groups, restarted asm a few times then traced sql+ a couple of times then all of a sudden the create disk group script worked. But then dbca failed writing to +fra, but step walking the create in sql+ I was able to create a db. Then trying the same operation in dbca writes to the +dat failed even though they worked the day before, only to work the next day (literally exact same script zero changes anywhere).

As to 12.2 I followed the docs to a tea and setup the afd disks in oui but the assistant job failed to create asm, even using redo feature in the configuration scripts. Set it up by hand everything works but oracle account gets a nasty tns-12547 error that seems to be again priv error related to grid account asm even after trying every step in every related mos article.

As to this article and standalone gi, I would say if you left out the step of deconfig asm and went to setting up afd disks etc then manually create asm, listener, dsset, etc then it might just work in 12.1.

LikeLike
Peter 5:20 am on September 16, 2017

Hi Marco,
If I want to check the state of afd driver with “afddriverstate supported” or “asmcmd afd_state” it will runs very long for 60 seconds but with successful result.
The problem is that ora.driver.afd CRS resource uses this afdstate action on startup and has a timeout of 60s. So sometimes there is an error reported from orarootagent in crs alert log on startup. However the CRS stack started and ora.driver.afd comes ONLINE.

env:
RAC 12.2 with OL 7.4, actual patches are applied

I’ve already post that in OTN
https://community.oracle.com/message/14524041#14524041
I’ll open a SR, too. But maybe it’s for your interest and you have an idea.

Thank you very much.

Regards Peter

LikeLike
Marco Mischke 10:27 am on September 22, 2017

Hi Peter,
thanks for your comment. I did not experience such a long time until now. Maybe you can share your insights once the issue gets fixed.
Cheers,
Marco

LikeLike
oracle12cdatabase 9:59 am on June 27, 2018

Is it mandatory to use ASMFD in 12.2 GI or ASMLIB still works?

LikeLiked by 1 person
- Marco Mischke 11:18 am on June 27, 2018
  
  It is not mandatory but is much more easy to use it right from the start. I recommend using AFD instead of ASMLib.
  
  LikeLike
Janie 10:00 pm on August 2, 2018

Hi Marco! I am new to ASM Filter driver concept and trying to setup GI Oracle restart RAC 12.2.0.1. Server admin has prepared multipath disks for ASM with size and alias name provided by me. (Such as ASMD001, ASMD002, ASMD003 — for data DG 100 GB each, ASMA001, ASMA002 –disks for ARCH DG 50 GB each, etc). The purpose of getting alias created was to easily identify device names while assiging it to different diskgroups.

So when I do “multipath -ll” on Redhat Linux 7 server, I see those alias names with specified size. But when I go to /dev, disk names are listed as /sdcb, /sg85, /sgcb, etc etc all owned by root:root. Shouldn’t alias be display here as well under /dev? Are we doing something wrong here?

brw-rw—-. 1 root disk 69, 0 Jul 31 11:42 sdcc
crw-rw—-. 1 root disk 21, 82 Jul 31 11:42 sg82
brw-rw—-. 1 root disk 69, 16 Jul 31 11:42 sdcd
crw-rw—-. 1 root disk 21, 83 Jul 31 11:42 sg83
crw-rw—-. 1 root disk 21, 84 Jul 31 11:42 sg84
brw-rw—-. 1 root disk 69, 32 Jul 31 11:42 sdce
brw-rw—-. 1 root disk 69, 48 Jul 31 11:42 sdcf

If above problem gets fixed, then my second question would be: “Since its already labeled at storage level, do I still need to create ASMFD labels”?
# cd /u01/app/oracle/product/12.2.0/grid/bin
# ./asmcmd afd_label ASMD001 /dev/ASMD001 –init
# ./asmcmd afd_label ASMD002 /dev/ASMD002 –init
# ./asmcmd afd_label ASMD003 /dev/ASMD003 –init
# ./asmcmd afd_label ASMA001 /dev/ASMA001 –init
# ./asmcmd afd_label ASMA002 /dev/ASMA002 –init

Reference:
https://docs.oracle.com/en/database/oracle/oracle-database/12.2/ladbi/installing-oracle-grid-infrastructure-for-a-standalone-server-with-a-new-database-installation.html#GUID-0B1CEE8C-C893-46AA-8A6A-7B5FAAEC72B3

We are still preparing storage for ASM and are new to 12c Grid/ASMFD concepts. My questions might sound silly, but I would appreciate any response! 🙂

Thanks much!

LikeLike
Marco Mischke 8:59 am on August 3, 2018

Hi Janie,
if you specify a name for the multipath device, it should appear under /dev/mapper/. This is just a name for the OS level, you still need to label it. The label is being stored in the disk header separately.
Cheers,
Marco

LikeLiked by 1 person
Janie 3:38 pm on August 3, 2018

Wonderful! Thank you so much Marco! I was struggling to find that path and couldn’t move ahead. Now I know I can provide ASM_DISKSTRING as /dev/mapper/*.

Right now disks under /dev/mapper/ are owned by root:root and are symbolic links. Should I get ownership of disks changed to grid:dba or grid:asmdba? Also, could symbolic link be a problem for ASM to discover disks?

lrwxrwxrwx. 1 root root 8 Aug 1 16:05 ASMOCR08 -> ../dm-13
lrwxrwxrwx. 1 root root 8 Aug 1 16:05 ASMD007 -> ../dm-17
lrwxrwxrwx. 1 root root 8 Aug 1 16:05 ASMR104 -> ../dm-18

My users are like this per Oracle documentation:
#/usr/sbin/useradd -g oinstall -G asmadmin,asmdba,asmoper -d /home/grid -m grid
#/usr/sbin/useradd -g oinstall -G dba,asmdba -d /home/oracle -m oracle

I sincerely appreciate all your help! Cheers!

LikeLike
- Marco Mischke 3:54 pm on August 3, 2018
  
  No need to set permissions. That is what AFD was made for. And as long as the symbolic link points to a valid device everything should be fine.
  
  LikeLiked by 1 person
Janie 8:08 pm on August 7, 2018

Hello Marco! Thank you so much for your last update. Need help with another issue.
I have labeled disks using ASMFD afd_label commands and verified it using afd_lslbl.
But when I run ./gridSetup.sh from Grid Home, on the second wizard page (step 2 of 10), no disks showed up. By default discovery string is /dev/sd* and even after changing it to /dev/mapper/ASM* nothing show up.

grid@usdbdevn grid_1]$ ls -ltr /dev/mapper/ASM*
lrwxrwxrwx. 1 root root 7 Aug 7 14:54 /dev/mapper/ASMR101 -> ../dm-7
lrwxrwxrwx. 1 root root 8 Aug 7 14:54 /dev/mapper/ASMR102 -> ../dm-14
lrwxrwxrwx. 1 root root 7 Aug 7 14:54 /dev/mapper/ASMR103 -> ../dm-6

What else could I be missing? Thanks again in advance!

LikeLiked by 1 person
- Marco Mischke 8:14 pm on August 7, 2018
  
  Hi,
  Did you use the -init option for afd_label?
  Cheers,
  Marco
  
  LikeLike
Janie 10:25 pm on August 7, 2018

Yes I did use –init and verification was also successful. For example:

asmcmd afd_label ASMR101 /dev/mapper/ASMR101 –init
asmcmd afd_label ASMR102 /dev/mapper/ASMR102 –init
asmcmd afd_label ASMR103 /dev/mapper/ASMR103 –init

and then
+ asmcmd afd_lslbl /dev/mapper/ASMR101
——————————————————————————–
Label Duplicate Path
================================================================================
ASMR101 /dev/mapper/ASMR101
+ asmcmd afd_lslbl /dev/mapper/ASMR102
——————————————————————————–
Label Duplicate Path
================================================================================
ASMR102 /dev/mapper/ASMR102
+ asmcmd afd_lslbl /dev/mapper/ASMR103
——————————————————————————–
Label Duplicate Path
================================================================================
ASMR103 /dev/mapper/ASMR103

LikeLike
Marco Mischke 10:16 am on August 10, 2018

Hm.. looks good so far. Actually I think that should work.

LikeLike
Janie 10:20 pm on August 10, 2018

It did not work while discovering, checked with Oracle and they asked to change ownership of disks to grid:oinstall. Did that, and gridsetup.sh was able to identify disks. Now we have different errors says “AFD is not supported”, trying to work it out with Oracle. I will update once we fix it. Thank you!

LikeLike
Harshini 2:14 pm on April 19, 2023

Hi Marco, Though we can check AFD is loaded with ( $ORACLE_HOME/bin/asmcmd afd_state ) , but How can we check if correct version is loaded or not? Please suggest.

LikeLike
- Marco Pachaly-Mischke 4:03 pm on April 19, 2023
  
  Hi,
  let me check that once I am back from vacation.
  Cheers, Marco
  
  LikeLike