Troubleshooting Metadb replicas (SVM)

 Problem: State database corrupted or unavailable. Recovering from Stale State

Database Replicas

Cause: Disk failure , Disk I/O error.

Symptom: Error message at the booting time if databases are <= 50% of total

database. System comes to Single user mode.

ok boot

Hostname: host1

metainit: Host1: stale databases

Insufficient metadevice database replicas located.

Use metadb to delete databases which are broken.

Ignore any "Read-only file system" error messages.

Reboot the system when finished to reload the metadevice database.

After reboot, repair any broken database replicas which were deleted.

Type Ctrl-d to proceed with normal startup, (or give root password for system

maintenance): <root-password>

Entering System Maintenance Mode.

1. Use the metadb command to look at the metadevice state database and see which

state database replicas are not available. Marked by unknown and M flag.

# /usr/opt/SUNWmd/metadb -i

flags first blk block count

a m p lu 16 1034 /dev/dsk/c0t3d0s3

a p l 1050 1034 /dev/dsk/c0t3d0s3

M p unknown unknown /dev/dsk/c1t2d0s3

M p unknown unknown /dev/dsk/c1t2d0s3

2. Delete the state database replicas on the bad disk using the -d option to the metadb

(1M) command. At this point, the root (/) file system is read-only. You can ignore the

mddb.cf error messages:

# /usr/opt/SUNWmd/metadb -d -f c1t2d0s3

metadb: demo: /etc/opt/SUNWmd/mddb.cf.new: Read-only file system .

Verify deletion

# /usr/opt/SUNWmd/metadb -i

flags first blk block count

a m p lu 16 1034 /dev/dsk/c0t3d0s3

a p l 1050 1034 /dev/dsk/c0t3d0s3

3. Reboot.

4. Use the metadb command to add back the state database replicas and to see that the

state database replicas are correct.

# /usr/opt/SUNWmd/metadb -a -c 2 c1t2d0s3

# /usr/opt/SUNWmd/metadb

flags first blk block count a m

p luo 16 1034 dev/dsk/c0t3d0s3

a p luo 1050 1034 dev/dsk/c0t3d0s3

a u 16 1034 dev/dsk/c1t2d0s3

a u 1050 1034 dev/dsk/c1t2d0s3

SVM in Solaris

Root Mirroring – Solaris Volume Manager

What to do?

1. Ensure that the alternate disk has equal geometry & size.

2. Take backup of /etc/system and /etc/vfstab file.

3. Copy VTOC from root (booting) disk to the alternate disk.

4. Ensure that the state database is created.

5. Convert the root slice as a logical component forcefully.

6. Create another metadevice for duplicating root slice.

7. Convert the swap slice as a logical component forcefully.

8. Create another metadevce for duplicating the swap slice.

9. Associate first sub-mirror (for root) to mirror root.

10. Associate first sub-mirror (for swap) to mirror swap.

11. Update the system & vfstab file by running 'metaroot' command.

12. Reboot the system.

13. Associate the second sub-mirror to mirror root.

14. Associate the second sub-mirror to mirror swap.

15. Install boot block (in case of SPARC machine) or grub (X86 machine) in the alternate root slice

16. See the physical path for the alternate disk

17 Set alias name in the OK prompt.

18. Set boot sequence in OK prompt.

How to do?

1. # format

Create the slices manually.

2. # cp /etc/system /etc/system.orig

   # cp /etc/vfstab /etc/vfstab.orig

3. # prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c0t12d0s2

Note:

fmthard -> populate label on the new hard disk drive

4. # metadb -afc3 c0t8d0s7 c0t10d0s7 c0t12d0s7

(if the replicas are existing, this step can be avoided)

5. # metainit -f d5 1 1 c0t8d0s0  

Converting forcefully the root slice as a metadevice

6. # metainit d10 1 1 c0t12d0s0

Creating another metadevice for root

7. # metainit -f d25 1 1 c0t8d0s1

Converting forceful the swap slice as a metadevice

8. # metainit d30 1 1 c0t12d0s1

Creating another metadevice for swap

9. # metainit d15 -m d5

Associating d5 with d15

Here d15 = main mirror for root

10. # metainit d35 -m d25

Associate d35 with d25

Here d35 = main mirror for swap

11. # metaroot d15

a. 'metaroot' edits the file /etc/system and /etc/vfstab so that the system may be booted with the root filesystem on a meta device.

b. 'metaroot' may also be used to edit the files so that the system may be booted with root file system on a conventional disk device.

c. Observe the changes to the files /etc/vfstab and /etc/system.

12. # init 6

Note:

Make sure the sync is completed before rebooting the system

by executing the command # metastat | grep %. It consume some time.

13. # metattach d15 d10

For root adding sub-mirror

14. # metattach d35 d30

For swap adding sub-mirror

15. # cd /usr/platform/`uname -m`/lib/fs/ufs

    # installboot bootblk /dev/rdsk/c0t12d0s0

Installing the boot block to the SPARC machine

# installgrub -fm /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c1d0s0

Installing the grub in the X 86 machines.

16. # ls -l /dev/dsk/c0t12d0s0

Will display the physical path of the logical device.

Pls make a note of the physical path.

17. OK nvalias

18. OK setenv boot-device

                                               

BREAKING THE MIRROR:

# metadetach d120 d100

# metaroot c0t0d0s0

c0t0d0s0 = raw disk of source disk which is running with OS.

Will revert the /etc/system & /etc/vfstab to the default status.

# init 6

# metaclear d100

# metaclear -r d120

Amazing Fact