Oct 24 2010

Migration of the 11g R1 OCR and Voting disks to different paths when Cluster Ready Service (CRS) is off-line

Category: 11g,Database,Multipathing,Solarisittichai @ 5:15 pm

During a new setup of our 11g R1 (11.1.0.7) on Solaris platform, initially the system is provided with only single path to disks (LUNs) on SAN. Later the sys admins would like to enable the multipathing. :-( Obviously, it is expected that the disk paths will be changed. This will not only affect the ASM disk groups, but also the OCR and Vote disks.

The path that is, for example, c4t50060E80045C5930d17s6, after multipathing, will become something like this c4t60060E80056FB30000006FB300001019d17s6.

The disk path change for the ASM disk groups is not a major concern due to the fact that all needed information is already stored in the disk header  (so the disk path name is irrelevant here). Merely, update the ASM’s disk discovery path parameter (asm_diskstring) to the new path, everything should be fine. (Reference: Metalink Doc ID 466231.1 – ASM Survive Change Of Disc Path?)

The main concern is with cluster registry information stored in the OCR and Voting disks which obviously their paths will not be the same after multipathing. It is expected that CRS won’t start up because the ocr.loc still contains the old paths which will no longer exist after reboot.

We could not run ocrconfig -replace ocr and ocrconfig -replace ocrmirror before multipathing as the  new disk paths are not there yet, or after that as CRS will not be available. Note that to move the OCR location for Oracle RAC 10gR2 or later, use “ocrconfig -replace” command. Manual edit of the ocr.loc at /var/opt/oracle is not recommended.

Fortunately there is a way to perform disk path change while CRS is off-line.

Before multipating
Backup all OCR and Voting disks.

For OCR, use:

# ocrconfig -manualbackup

For Voting disks, use:

# dd if={VOTING_PATH} of={VOTING_BACKUP_PATH}

After multipathing (and server reboot), the CRS is not available.

1. Even with CRS down, ensure that there is no other CRS-related processes (evmd or cssd) running.

Some processes will be restarted automatically even they’re terminated manually with using kill -9.

To ensure that no process running, backup and comment the last three line of the /etc/inittab.

#h1:3:respawn:/etc/init.d/init.evmd run >/dev/null 2>&1 </dev/null
#h2:3:respawn:/etc/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null
#h3:3:respawn:/etc/init.d/init.crsd run >/dev/null 2>&1 </dev/null

Run init q to take change into effect.

Don’t forget to un-comment them when all tasks are completed.

2. On all nodes, run the following commands as root to point to the new disk paths for the master and the mirror:

# ocrconfig -repair ocr /dev/rdsk/c5t60060E80045C590000005C5900000C03d0s6

# ocrconfig -repair ocrmirror /dev/rdsk/c5t60060E80045C590000005C5900000C04d0s6

3. As root, on one node, run:

# ocrconfig -overwrite

4. Run ocrcheck on all nodes to verify new paths:

# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version                  :          2
Total space (kbytes)     :    1036532
Used space (kbytes)      :       5408
Available space (kbytes) :    1031124
ID                       :  582752590
Device/File Name         : /dev/rdsk/c5t60060E80045C590000005C5900000C03d0s6
Device/File integrity check succeeded
Device/File Name         : /dev/rdsk/c5t60060E80045C590000005C5900000C04d0s6
Device/File integrity check succeeded

Cluster registry integrity check succeeded

5. Next, we’ll also change the paths for voting disk.

Delete the old paths and add new ones in the order to ensure that at least one disk is available. The option -force will be required in this case.

# crsctl delete css votedisk /dev/rdsk/c3t50060E80045C5920d15s6 -force
# crsctl add css votedisk /dev/rdsk/c5t60060E80045C590000005C5900000C05d0s6 -force
# crsctl delete css votedisk /dev/rdsk/c3t50060E80045C5920d16s6 -force
# crsctl delete css votedisk /dev/rdsk/c3t50060E80045C5920d17s6 -force
# crsctl add css votedisk /dev/rdsk/c5t60060E80045C590000005C5900000C06d0s6 -force
# crsctl add css votedisk /dev/rdsk/c5t60060E80045C590000005C5900000C07d0s6 –force

Now we should be able to start CRS.

Reference: Method 2 (Offline) in Note 428681.1 :OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE)

Tags: , , , , , , , , , , , , , ,


Oct 05 2009

Online SAN Storage Migration for Oracle 11g RAC database with ASM

Category: 11g,Database,Migrationittichai @ 7:30 am

It has been awhile since my last post. My pathetic excuses are all pretty much mentioned here. :-)

Last month we’ve worked with the storage team to migrate the SAN storage of our Oracle 11gR1 database to a new one. The drive of migration is mainly for SAN consolidation which is, of course, ultimately for cost saving. In addition to migrating the ASM disk groups storing database’s data files, all clusterware files (OCR and voting disk) must be migrated too. The rebalance feature in Oracle ASM makes data migration very easy and seamless. And since the clusterware files have redundancy, they can be seamlessly migrated as well. With 11gR1, all migration tasks can be performed online.

Prerequisites/Assumptions:

- New SAN LUNs/disks are already visible to all RAC nodes. In case of the disks for ASM diskgroups, they are already discovered by ASM. The minimum numbers and permissions of the OCR and voting disks must be met.

- It is recommended to perform the migration tasks during off-peak hours or even better if during planned maintenance window period.

Note that the sample shown here is specific to my environment (11.1.0.7 on Solaris 10 with dual-pathing to Hitachi SAN, and OCR and voting disks are on raw devices).

SAN Migration of the ASM diskgroups

If you’re more comfortable with GUI, all tasks here can be accomplished using the Enterprise Manager.

1. Add new disks to ASM diskgroups.

ALTER DISKGROUP PMDW_DG1 ADD DISK
  '/dev/rdsk/c4t60060E80056FB30000006FB300000823d0s6' NAME PMDW_DG1_0003,
  '/dev/rdsk/c4t60060E80056FB30000006FB300000826d0s6' NAME PMDW_DG1_0004,
  '/dev/rdsk/c4t60060E80056FB30000006FB300000829d0s6' NAME PMDW_DG1_0005
REBALANCE POWER 11;

We go with the rebalance power of 11 which is full throttle because it is planned maintenance.

2. Check rebalance status from Enterprise Manager or v$ASM_OPERATION.

3. When rebalance completes, drop the old disks.

ALTER DISKGROUP PMDW_DG1 DROP DISK
   PMDW_DG1_0000,
   PMDW_DG1_0001,
   PMDW_DG1_0002
REBALANCE POWER 11;

When adding or removing several disks, it is recommend to add or remove all disks at once. This is to reduce the number of the rebalance operations that are needed for storage changes.

SAN Migration of the OCR Files

1. Backup all OCR-related files.

# {CRS_HOME}/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :     921332
         Used space (kbytes)      :       4548
         Available space (kbytes) :     916784
         ID                       :  776278942
         Device/File Name         : /dev/rdsk/c4t50060E800000000000002892000003F8d0s6
                      Device/File integrity check succeeded
         Device/File Name         : /dev/rdsk/c4t50060E800000000000002892000003F9d0s6
                       Device/File integrity check succeeded

Backup the /var/opt/oracle/ocr.loc file:

# cp ocr.loc ocr.loc.old

Manually backup OCR:

# {CRS_HOME}/bin/ocrconfig -manualbackup

2. As root, run the following commands to replace OCR files. This change can be performed on-line, and will be reflected across the entire cluster.

# {CRS_HOME}/bin/ocrconfig -replace ocr /dev/rdsk/c4t60060E80056FB30000006FB300001014d0s6

# {CRS_HOME}/bin/ocrconfig -replace ocrmirror /dev/rdsk/c4t60060E80056FB30000006FB300001015d0s6

3. Verify the new configuration.

Check new ocr.loc file updated:

# cat /var/opt/oracle/ocr.loc
#Device/file /dev/rdsk/c4t50060E800000000000002892000003F9d0s6 getting replaced by device /dev/rdsk/c4t60060E80056FB30000006FB300001015d0s6
ocrconfig_loc=/dev/rdsk/c4t60060E80056FB30000006FB300001014d0s6
ocrmirrorconfig_loc=/dev/rdsk/c4t60060E80056FB30000006FB300001015d0s6

Check OCR:

# {CRS_HOME}/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version                  :          2
Total space (kbytes)     :     921332
Used space (kbytes)      :       4548
Available space (kbytes) :     916784
ID                       :  776278942
Device/File Name         : /dev/rdsk/c4t60060E80056FB30000006FB300001014d0s6
Device/File integrity check succeeded
Device/File Name         : /dev/rdsk/c4t60060E80056FB30000006FB300001015d0s6
Device/File integrity check succeeded

Cluster registry integrity check succeeded

Logical corruption check succeeded

SAN Migration of the voting disks

1. Backup the voting disks.

Query the original locations:

# /opt/oracrs/bin/crsctl query css votedisk
0.     0    /dev/rdsk/c4t50060E800000000000002892000003FBd0s6
1.     0    /dev/rdsk/c4t50060E800000000000002892000003FCd0s6
2.     0    /dev/rdsk/c4t50060E800000000000002892000003FFd0s6

Backup voting disks using dd:

dd if={voting_disk_name} of={backup_file_name}

Example,
dd if=/dev/rdsk/c4t50060E800000000000002892000003FBd0s6 of=/tmp/voting1

2. Move voting disks.

Starting with 11.1 onwards, the voting disk migration can be performed on-line.

# /opt/oracrs/bin/crsctl delete css votedisk /dev/rdsk/c4t50060E800000000000002892000003FBd0s6
# /opt/oracrs/bin/crsctl add  css votedisk /dev/rdsk/c4t60060E80056FB30000006FB300001017d0s6

# /opt/oracrs/bin/crsctl delete css votedisk  /dev/rdsk/c4t50060E800000000000002892000003FCd0s6
# /opt/oracrs/bin/crsctl add css votedisk  /dev/rdsk/c4t60060E80056FB30000006FB300001018d0s6

# /opt/oracrs/bin/crsctl delete css votedisk  /dev/rdsk/c4t50060E800000000000002892000003FFd0s6
# /opt/oracrs/bin/crsctl add css votedisk  /dev/rdsk/c4t60060E80056FB30000006FB300001019d0s6

3. Verify the new configuration.

# /opt/oracrs/bin/crsctl query css votedisk
0.     0    /dev/rdsk/c4t60060E80056FB30000006FB300001017d0s6
1.     0    /dev/rdsk/c4t60060E80056FB30000006FB300001018d0s6
2.     0    /dev/rdsk/c4t60060E80056FB30000006FB300001019d0s6

Reference:
Metalink #428681.1:  OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE), including moving from RAW Devices to Block Devices

Tags: , , , , , ,