How to recover VxVM volumes after SAN erasures

This a reminder of how I recovered a re-initialized SAN filesystem’s vxVM config without the original SAN config info.

This procedure will work if the storage layout and architecture is managed on the SAN level and all vxVM volumes are made of 1 single disk at OS level.

I didn’t know that when you export, modify and re-import config of a storageTek it reinitializes itself with the imported config :-(

Dump current config for each disk group in vxprint format

Copy the /etc/vx/cbr/bk/ to a safe location, then for each group, cd into the disk group config folder print the output like this :

[root@otasrv1 dg_osglb.1258621615.102.otasrv1]# pwd

/product/MAINTENANCE/VXVM/etc/vx/cbr/bk/dg_osglb.1258621615.102.otasrv1

cat 1258621615.102.otasrv1.cfgrec|vxprint -ht -D -

the 1258621615.102.otasrv1 part will vary per disk group and per project.

Save all the outputs to text files named .txt copy all the text files to tmp/ folder.

Here is my bash history:

Check that disks are still seen in OS at same location as before failure

fdisk -l >ha

[root@otasrv1 dg_osglb.1258621615.102.otasrv1]# more ha

Disk /dev/sda: 145.9 GB, 145999527936 bytes

255 heads, 63 sectors/track, 17750 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

/dev/sda1 * 1 131 1052226 83 Linux

/dev/sda2 132 4308 33551752+ 82 Linux swap

/dev/sda3 4309 7572 26218080 83 Linux

…

Ignore the above Linux filesystems which were never managed by vxVM. You will get a few unformatted drives like the bellow:

Disk /dev/sdb: 21.4 GB, 21474836480 bytes

64 heads, 32 sectors/track, 20480 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/sdc: 21.4 GB, 21474836480 bytes

64 heads, 32 sectors/track, 20480 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

Disk /dev/sdd: 5368 MB, 5368709120 bytes

166 heads, 62 sectors/track, 1018 cylinders

Units = cylinders of 10292 * 512 = 5269504 bytes

Disk /dev/sde: 5368 MB, 5368709120 bytes

166 heads, 62 sectors/track, 1018 cylinders

Units = cylinders of 10292 * 512 = 5269504 bytes

… more disks

To find out if sdb has changed location, I check if the size has changed. I change to tmp/ folder and grep for sdb

grep ^dm *|grep sdb

sg_db1.txt:dm disk00 sdb auto 2048 41932544 –

here I see that sdb used to be 41932544*512=21469462528 almost equal to new size, difference = 5373952 which is only 5Mb, so it is only a difference of reading between vxVM and fdisk. This disk still has the same name.

Must repeat above check operation for every disk in list. If any disk has changed its name, you will need to manually adjust sections when you add disks to disks group and rename them.

Generate commands to initialize disk with vxvm cds

grep ^dm *|sort -k3|while read d_g d_sk_n d_sk au_t p_rv p_ub dash; do echo '/usr/lib/vxvm/bin/vxdisksetup -if '$d_sk" privlen="$p_rv" publen="$p_ub;done>ESD118_vxVM.sh

the shell script ESD118_vxVM.sh will now contain commands to initialize all the disks previously used for Veritas shared fses.

Generate command to init each disk group and add its disks to it

grep ^dm *|sort -k3|while read d_g d_sk_n d_sk r_est p_rv size dash; do disk_g=`echo $d_g|cut -d\. -f1`; echo 'vxdg init '$disk_g" "$d_sk_n"="$d_sk;done>>ESD118_vxVM.sh

Generate commands to recreate the volumes

grep -e'^v ' *|while read dg_n vol_n dash state st size sel dosh fsg;do disk_g=`echo $dg_n|cut -d\. -f1`; echo 'vxassist -g '$disk_g" make "$vol_n" "$size; done>>ESD118_vxVM.sh

you should end up with 3 sets of configuration commands in the shell script.

Note that vxVM sees disks with a funny name. I had to change the disk initialization section like this:

/usr/lib/vxvm/bin/vxdisksetup -if sdp privlen=2048 publen=398440192

/usr/lib/vxvm/bin/vxdisksetup -if sdq privlen=2048 publen=83875584

To

/usr/lib/vxvm/bin/vxdisksetup -if sdap privlen=2048 publen=398440192

/usr/lib/vxvm/bin/vxdisksetup -if sdaq privlen=2048 publen=83875584

I ended up with this one:

Now, for the disk group creation lines, you need to put all the disks on the same line

vxdg init dg_backup disk14=sdap

vxdg init dg_backup disk15=sdaq

becomes

vxdg init dg_backup disk14=sdap disk15=sdaq

Also, I had 2 errors with disks that were too small to match requested sizes:

[root@otasrv1 dg_osglb.1258621615.102.otasrv1]# /usr/lib/vxvm/bin/vxdisksetup -if sdad privlen=2048 publen=10475264

VxVM vxdisksetup ERROR V-5-2-2480 Disk is too small for supplied parameters

[root@otasrv1 dg_osglb.1258621615.102.otasrv1]# /usr/lib/vxvm/bin/vxdisksetup -if sdae privlen=2048 publen=10475264

VxVM vxdisksetup ERROR V-5-2-2480 Disk is too small for supplied parameters

after removing a few kilobytes to size it works:

/usr/lib/vxvm/bin/vxdisksetup -if sdad privlen=2048 publen=10470000

For some reason, I also had to do the following renamings:

vxdg -n dg_db1 import sg_db1

vxedit -g dg_db2 rename vol_sgdb vol_sgbd

vxedit -g dg_db1 rename vol_sgdb vol_sgbd

final shell script with good values is:

Final result

[root@otasrv1 ~]# vxdisk list

DEVICE TYPE DISK GROUP STATUS

sda auto:none - - online invalid

sdab auto:cdsdisk - - online

sdac auto:cdsdisk - - online

sdad auto:cdsdisk - - online

sdae auto:cdsdisk - - online

sdaf auto:cdsdisk - - online

sdag auto:cdsdisk - - online

sdah auto:cdsdisk - - online

sdai auto:cdsdisk - - online

sdaj auto:cdsdisk - - online

sdak auto:cdsdisk - - online

sdal auto:cdsdisk - - online

sdam auto:cdsdisk - - online

sdan auto:cdsdisk - - online

sdao auto:cdsdisk - - online

sdap auto:cdsdisk - - online

sdaq auto:cdsdisk - - online

sdar auto:cdsdisk - - online

sdas auto:cdsdisk - - online

sdat auto:cdsdisk - - online

sdau auto:cdsdisk - - online

sdav auto:cdsdisk - - online

sdaw auto:cdsdisk - - online

sdax auto:cdsdisk - - online

sday auto:cdsdisk - - online

sdaz auto:cdsdisk - - online

The commands in the shell script should run fine but run them manually and use your integrator skills to troubleshoot.

Format the vxVM filesystems for usage

[root@otasrv1 ~]# for f_s in `find /dev/vx/dsk/ -type b`; do mkfs.ext3 $f_s;done

Bring up the mount points and restore databases

for m_p in `hares -display -group sg_backup|grep backup_fs|awk '{print $1}'|sort -u`; do hares -online $m_p -sys otasrv4; done

for m_p in `hares -display -group sg_db2|grep db2_fs|awk '{print $1}'|sort -u`; do hares -online $m_p -sys otasrv3; done

for m_p in `hares -display -group sg_db1|grep db1_fs|awk '{print $1}'|sort -u`; do hares -online $m_p -sys otasrv4; done

After the disks come up on different hosts, freeze the associated groups before starting cold restore.

hagrp –freeze sg_backup; hagrp –freeze sg_db1; hagrp –freeze sg_db1

Troubleshoot.

[Oracle DBA] Kill all sessions for a specific user/application

Select them like:

collumn TOT format A60

COLUMN USERNAME FORMAT A10

select 'Alter system kill session '||CHR(39)||SID||','||SERIAL#||CHR(39) TOT, username, osuser, machine from v$session where username='PMSEEADMIN';

Then

run all the resulting commands

Alter system kill session '55,558' ;

Alter system kill session '66,3003' ;

Alter system kill session '68,3822' ;

Alter system kill session '69,1648' ;

Alter system kill session '73,1320' ;

Alter system kill session '97,3689' ;

Alter system kill session '120,2671';

This blog is public but use at your own risk.

[Network Traces] how to print full hexdump of packets on one ligne using perl and tcpdump.

Here is the shell script:

#! /usr/bin/perl -w
sub concat{
#To ressemble fragmented packets and handle lost fragments.
print "entering concat";
}
$last_time="00:00:00.00000";
$fragment="FFFF";
while (<>)
{
#Sample TCPDUMP with -x -n output would be:
#
#04:09:29.989335 IP 10.0.0.32.55238 > 209.85.227.113.80: Flags [.], ack 2233, win 81, options [nop,nop,TS val 256065 ecr 48692], length 0
# 0x0000: 4500 0034 d413 4000 4006 a7c9 0a00 0020
# 0x0010: d155 e371 d7c6 0050 f5f8 0b98 f76a 480d
# 0x0020: 8010 0051 f7eb 0000 0101 080a 0003 e841
# 0x0030: 0000 be34
#
if(/^([[:digit:]]{2}):([[:digit:]]{2}):([[:digit:]]{2})\.([[:digit:]]+)\s(.*)/){
#new fragment
#*****Here we could check address/lengths in the hash for consistency, but I guess the regex builds the check in.
print "$last_time:$fragment\n";
$fragments{$last_time}="$fragment";
$last_time="$1:$2:$3.$4";
$fragment="";
}
elsif(/^[\s+]0x([[:xdigit:]]{4}):(\s+)([[:xdigit:]]{4})\s+([[:xdigit:]]{4})\s+([[:xdigit:]]{4})\s+([[:xdigit:]]{4})\s+([[:xdigit:]]{4})\s+([[:xdigit:]]{4})\s+([[:xdigit:]]{4})\s+([[:xdigit:]]{4})(.*)/)
{
#new piece of the fragment
$fragment="$fragment$3$4$5$6$7$8$9$10";
}

}

$fragments{$last_time}="$fragment";
print "$last_time:$fragment\n";

[Cluster, Veritas] Configuring timings for startup and fail-over

After spending 3 hours trying to figure out a way to fine tune timings on a large Veritas cluster to allow for a fail-over during a special and tricky failure use case, I put for myself a reminder on all the parameters and their meaning here.

Configuring the failure behavior

To define the number of times a resource attempts to recover before giving up.
RestartLimit is the number of times VCS attempts to restart the failed resource on the same host. When it is exhausted, the resource faults. If the resource is critical, the service group fails over to the best available node.

To define the number of times a resource attempts to Online before giving up.
OnlineRetryLimit is the number of times VCS attempts to Online the resource initially. When it is exhausted, the resource faults and the service group fails-over.

To define how long Veritas waits between monitoring attempts.
MonitorInterval is in seconds the duration between 2 resource status checks. To be combined with ToleranceLimit to define overall VCS retry policy for a specific resource.

To define after how many failure results from monitoring checks on a specific resource VCS must consider the status as faulted.
MonitorTimeout is the interval in seconds to wait for the monitoring script to return a result and exit.

How long should VCS allow the monitoring script to run before killing it and declaring monitor time-out?
OnlineWaitLimit is the number of times the monitoring agent must try to check whether a resource that was started by VCS during normal startup is indeed ONLINE before considering that the startup attempt is unsuccessful.

To define what happens if the monitoring agent is taking too long to return status(think overloaded service with applicative test).
FaultOnMonitorTimeouts is the number of times the monitoring agent must time-out before VCS considers that the monitored resource is faulted. But it is a bad design to let this in VCS. It is better to make sure you manage monitoring time out via monitor time-out instead. and make sure the agent completes within the MonitorTimeout interval.

How long should VCS allow a startup script to run before declaring online time-out

OnlineTimeout is the interval in seconds to wait for the startup script to return a result and exit.

During resource startup, how many times do we check to see if the startup is successful?
OnlineWaitLimit is the number of times the monitoring agent must try to check whether a resource that was started by VCS during normal startup is indeed ONLINE before considering that the startup attempt is unsuccessful. In between monitor attempts, it waits for MonitorInterval(?)

[HTTP][Apache] How to take a value from a header and put it in a new header using mod_proxy, mod_headers and mod_rewrite

Today I decided after 4 hours trying to figure out something I did a year ago. That from now on when I research a problem and find a solution, I will blog the solution so that I do not have waste time in future. To my surprise, I had already made a blog post in 2007 and my old blogger account still works...

I have tested that the following config in apache config file rewrites the header properly from a COOKIE to a new HTTP-HEADER.

RewriteEngine on
RewriteLog "/tmp/rewrite.log"
RewriteLogLevel 9
RewriteCond %{HTTP_COOKIE} (.*)x-up-calling-line-id=(.*);(.*)
RewriteRule .* - [E=in-msisdn:%2]
RequestHeader set "OUT-MSISDN" %{in-msisdn}e

The cookie was
curl http://localhost/ -H 'Cookie: JSESSIONID=FD92481E01852F97386A9B1F163534B7;x-up-calling-line-id=23387964521; Path=/towap'

And I received the msisdn in proper header OUT-MSISDN in server. Apache version is 2.2.8_Ubuntu with modules mod_headers and mod_rewrite

Ok, maybe I start blogging

Got here by mistake, trying to comment on gtalk