Oracle RAC, ASM, Exadata, Cloud, and More..: January 2014

Infiniband verification

Verify IB cards are present

# lspci | grep Infi

Should be 1 card per system.

Verify IB ports are UP

# ibstatus | grep phys

phys state: 5: LinkUp

Should show 2 ports up per system.

Verify IB ports are at QDR

# ibstatus | grep rate

rate: 40 Gb/sec (4X QDR)

Should show 2 ports at 40 Gb/sec (4x QDR) per system.

ibcheckerrors (to check network error)

ibdiagnet -r (to check cable)

List Flash Storage on Exadata Servers

You can query SCSI flash device information using lsscsi or list your PCI flash module using flash_dom to understand how flash storage is configured and presented on an Exadata Storage Server

lsscsi -v | grep MARVEL

Note: Flash devices are show four 8, 9, 10, 11. This is because of four flash cards have four FMods. Therefore there are 16 flash devices and you can use flash_dom to display the details for the PCI flash devices

flash_dom –l

HBA# Port Name Chip Vendor/Type/Rev MPT Rev Firmware Rev IOC WWID Serial Number

1. /proc/mpt/ioc0 LSI Logic SAS1068E C0 105 011b5c00 0 50310020001116a80 465769T+1209A4ADFA

Current active firmware version is 011b5c00 (1.27.92)

Firmware image's version is MPTFW-01.27.92.00-IT

x86 BIOS image's version is MPTBIOS-6.26.00.00 (2008.10.14)

FCode image's version is MPT SAS FCode Version 1.00.49 (2007.09.21)

D# B___T Type Vendor Product Rev Operating System Device Name

1. 0 0 Disk ATA MARVELLXXXXXXXD21Y /dev/sdn [8:0:0:0]

2. 0 1 Disk ATA MARVELLXXXXXXXD21Y /dev/sdo [8:0:1:0]

3. 0 2 Disk ATA MARVELLXXXXXXXD21Y /dev/sdp [8:0:2:0]

4. 0 3 Disk ATA MARVELLXXXXXXXD21Y /dev/sdq [8:0:3:0]

Validate FMods

You can see the flash devices grouped into sets of four on PCI slots 1, 2, 4, 5. Each device per PCI slot residing in FDOM 0, 1, 2,OR 3. Each FMOD is 24GB slide of storage x 16 = 284GB. In X2 models, there are 4x 96GB pci FLASH cards per storage cell. Each PCI flash car has a dvice partitioned per FDOM so there are 16 flash devices. These flash devices are the flash disks and used for Smart flash cache and smart flash logging.

You should see 16 FMODs. To validate all FMODs are present at the OS level, you can perform the following steps:

cellcli > list physicaldisk where disktype='FlashDisk' attributes name, disktype, physicalSize, slotNumber

flash_dom –l

cellcli > list flashcache detail

id: cf20a022-a381-428e-bf6f-ee54dbcdcb56

size: 364.75G

status: normal

cellcli > list flashlog detail

id: e7cbe8c7-d2f3-4d01-9f87-da83861c21d1

size: 512M

status: normal

To verify hardware profile

/opt/oracle.SupportTools/CheckHWnFWProfile -d

Flash Storage

You can query SCSI flash device using lsscsi, list your PCI flash modedule using flash_dom

lsscsi -v | grep MARVEL (to view your flash devices)

[8:0:0:0] disk ATA MARVELL SD88SA02 D20Y /dev/sdn

…

[11:0:3:0] disk ATA MARVELL SD88SA02 D20Y /dev/sdac

flash_dom -l (flash devices are split into groups of four: 8, 9, 10, 11). Each of flash cars have 4 FMods. Therefore each cell storage have 16 flash devices (4 x 4)

1. /proc/mpt/ioc0 LSI Logic SAS1068E C0 105 011b5c00 0 5080020001116a80 465769T+1209A40TXF

Current active firmware version is 011b5c00 (1.27.92)

Firmware image's version is MPTFW-01.27.92.00-IT

x86 BIOS image's version is MPTBIOS-6.26.00.00 (2008.10.14)

FCode image's version is MPT SAS FCode Version 1.00.49 (2007.09.21)

D# B___T Type Vendor Product Rev Operating System Device Name

1. 0 0 Disk ATA MARVELL SD88SA02 D20Y /dev/sdn [8:0:0:0]

2. 0 1 Disk ATA MARVELL SD88SA02 D20Y /dev/sdo [8:0:1:0]

3. 0 2 Disk ATA MARVELL SD88SA02 D20Y /dev/sdp [8:0:2:0]

4. 0 3 Disk ATA MARVELL SD88SA02 D20Y /dev/sdq [8:0:3:0]

Display physical disk, Megaraid device information, mdadm software RAID configuration

lsscsi can show both physical SAS drives and flash devices. There are 12 SAS devices and

lsscsi -v | grep MARVEL

[8:0:0:0] disk ATA MARVELL SD88SA02 D20Y /dev/sdn

[8:0:1:0] disk ATA MARVELL SD88SA02 D20Y /dev/sdo

….

[11:0:3:0] disk ATA MARVELL SD88SA02 D20Y /dev/sdac

lsscsi -v | grep LSI show 12 physical devices

/opt/MegaRAID/MegaCli/MegaCli64 -ShowSummary –aALL (you should see 12 disks)

Connector : Port 0 - 3<Internal><Encl Pos 0 >: Slot 0

Vendor Id : HITACHI

Product Id : HQWERASDFA.0T

State : Online

Disk Type : SAS,Hard Disk Device

Capacity : 2.727 TB

Power State : Active

Each Exadata Storage Server has 12 physical SAS disks and 4 96GB PCIe Sun Flash Accelerator flash cards. Each card partition into 4 x 24 GB partitions.

On Compute Node, there are 4 x 300 GB SAS – 1 hot spare (slot 3) and 3 active (slot 0,1,2)

/opt/MegaRAID/MegaCli/MegaCli64 -ShowSummary –aALL

[root@r22dwdbs06h-adm MegaCli]# pvdisplay

--- Physical volume ---

PV Name /dev/sda2

VG Name VGExaDb

PV Size 834.89 GB / not usable 3.84 MB

Allocatable yes

PE Size (KByte) 4096

Total PE 213731

Free PE 153827

Allocated PE 59904

PV UUID TMKLyW-2e2u-2BFq-xf4l-jdne-Vnb7-LOmcHd

[root@r22dwdbs06h-adm MegaCli]# lvdisplay

--- Logical volume ---

LV Name /dev/VGExaDb/LVDbSys1

VG Name VGExaDb

--- Logical volume ---

LV Name /dev/VGExaDb/LVDbSys2

VG Name VGExaDb

--- Logical volume ---

LV Name /dev/VGExaDb/LVDbSwap1

VG Name VGExaDb

--- Logical volume ---

LV Name /dev/VGExaDb/LVDbOra1

VG Name VGExaDb

--- Logical volume ---

LV Name /dev/VGExaDb/sysback

VG Name VGExaDb

LV UUID kAKBpc-2I2n-xeJh-8sZy-OPxD-hklZ-ykaBsv

lvdisplay -v /dev/VGExaDb/LVDbSys1 (shows attributes of that logical volume)

ls -ltr /dev/VGExaDb/LVDb* (show the map to /dev/mapper devices)

Display Storage and Compute Server architecture

You’re assigned to manage the Exadata Machine. You want to display overall architecture, storage configuration, network, and OS.

uname –a (To display the kernel version for Oracle enterprise linux)

dmidecode –s system-product-name (To display the server model and serial number)

[root@glscell01 ~]# dmidecode -s system-product-name

SUN FIRE X4170 M2 SERVER

[root@glscell01 ~]# dmidecode -s system-serial-number

1308FMM0MG

imageinfo (To see cell version, kernel version, active system partition)

imagehistory (image installed/patched overtime)

[root@r22dwdbs06h-adm ~]# imagehistory

Version : 11.2.3.2.0.120713

Image activation date : 2012-1-17 15:23:49 -0500

Imaging mode : fresh

Imaging status : success

Version : 11.2.3.2.1.130302

Image activation date : 2013-12-04 21:02:09 -0600

Imaging mode : patch

Imaging status : success

Each storage cell contains either:

12 physical 600GB 15000 RPM high performance of delivering up to 1.8 GB per second of raw per cell or

3 TB 7200 RPM High Capacity SAS disk of delivery up to 1.3GB per second of raw data bandwidth

Compute server:

NET0 allow ssh, it uses eth0 interface

NET1, NET2, NET1-2 RAC VIP and SCAN

IB Connects two ports on the compute servers to both of the InfiniBand leaf switches in the rac. All storage servers and RAC interconnect use this network

NET3 is built on eth3 Use for Backup/external traffic