_MGMTDB Management database (CHM repository) resource

Starting 12c, the management repository single instance database is created if the option is selected during GI installation.  It’s managed by GI and is a single instance that will fail over to the other node in case of the hosting node goes down.  The purpose of the management database is the central repository to store cluster health monitor (CHM/ora.crf).  Its data files are on the same shared storage as OCR and Voting file.  If the option is not configured during the installation or upgrade, all features of Cluster Health Monitor will be disabled.

To display the management repository configuration

Database unique name: _mgmtdb
Database name:
Oracle home: /u01/oracle/product/12.1.0.1/grid
Oracle user: oracle
Spfile: +GI/_mgmtdb/spfile-MGMTDB.ora
Password file:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Database instance: -MGMTDB
Type: Management

To display the configuration for management listener resource:
srvctl config mgmtlsnr
Name: MGMTLSNR
Owner: oracle
Home: <CRS home>
End points: TCP:1531

Some useful commands to check out the management repository database

oclumon manage -get MASTER
srvctl status mgmtdb
srvctl config mgmtdb
srvctl enable  mgmtdb
srvctl disable mgmtdb
srvctl disable mgmtlsnr
srvctl disable mgmtdb -node glsnd015 (To disable the management repository database from running on node glsnd015)
srvctl enable mgmtdb -node glscind01
srvctl relocate mgmtdb -n glscind01 (relocate from node glscind02 to glsind01)

select file_name from dba_data_files union select member file_name from v$logfile
oclumon manage -get reppath

oclumon manage -repos changerepossize 4000

11.2.0.3 to 12.1.0.1 - Manually upgrade Oracle RAC Database

Install perl-TermReadKey-2.30-4.el5.x86_64.rpm
rpm –Uvh perl-TermReadKey-2.30-4.el5.x86_64.rpm

Change compatible for the ASM diskgroup where your database is belonged to
alter diskgroup DBA_DATA501 set attribute 'compatible.asm'='12.1.0.1.0';
 alter diskgroup DBA_DATA501 set attribute 'compatible.rdbms'='11.2.0.0.0';

Copy the parameter file to 12.1.0.1 Oracle home
cp init.ora /apps/oracle/product/12.1.0.1/racdb/dbs/

Database Upgrade
alter system set cluster_database=FALSE scope=spfile sid=’*’;
startup upgrade
cd $ORACLE_HOME/rdbms/admin
$ORACLE_HOME/perl/bin/perl catctl.pl -n  6 -l $ORACLE_HOME/diagnostics catupgrd.sql

There are 56 phases.  Running with parallel 6

Using 6 processes.
Serial   Phase #: 0 Files: 1      Time: 327s
Serial   Phase #: 1 Files: 3      Time: 105s
Restart  Phase #: 2 Files: 1      Time: 0s
Parallel Phase #: 3 Files: 18     Time: 22s
Restart  Phase #: 4 Files: 1      Time: 1s
Serial   Phase #: 5 Files: 5      Time: 43s
Serial   Phase #: 6 Files: 1      Time: 24s
Serial   Phase #: 7 Files: 3      Time: 15s
Restart  Phase #: 8 Files: 1      Time: 0s
Parallel Phase #: 9 Files: 60     Time: 58s
Restart  Phase #:10 Files: 1      Time: 1s
Serial   Phase #:11 Files: 1      Time: 30s
Restart  Phase #:12 Files: 1      Time: 0s
Parallel Phase #:13 Files: 199    Time: 38s
Restart  Phase #:14 Files: 1      Time: 0s
Serial   Phase #:15 Files: 3      Time: 1s
Restart  Phase #:16 Files: 1      Time: 0s
Parallel Phase #:17 Files: 33     Time: 44s
Restart  Phase #:18 Files: 1      Time: 0s
Serial   Phase #:19 Files: 3      Time: 11s
Restart  Phase #:20 Files: 1      Time: 0s
Parallel Phase #:21 Files: 23     Time: 144s
Restart  Phase #:22 Files: 1      Time: 0s
Parallel Phase #:23 Files: 11     Time: 78s
Restart  Phase #:24 Files: 1      Time: 1s
Serial   Phase #:25 Files: 1      Time: 81s
Restart  Phase #:26 Files: 1      Time: 0s
Parallel Phase #:27 Files: 0      Time: 0s
Serial   Phase #:28 Files: 1      Time: 0s
Parallel Phase #:29 Files: 130    Time: 8s
Serial   Phase #:30 Files: 1      Time: 0s
Restart  Phase #:31 Files: 1      Time: 0s
Serial   Phase #:32 Files: 1      Time: 0s
Parallel Phase #:33 Files: 122    Time: 11s
Serial   Phase #:34 Files: 1      Time: 0s
Restart  Phase #:35 Files: 1      Time: 0s
Serial   Phase #:36 Files: 4      Time: 100s
Restart  Phase #:37 Files: 1      Time: 0s
Parallel Phase #:38 Files: 13     Time: 101s
Restart  Phase #:39 Files: 1      Time: 0s
Parallel Phase #:40 Files: 10     Time: 15s
Restart  Phase #:41 Files: 1      Time: 0s
Serial   Phase #:42 Files: 1      Time: 12s
Restart  Phase #:43 Files: 1      Time: 0s
Serial   Phase #:44 Files: 1      Time: 9s
Serial   Phase #:45 Files: 1      Time: 0s
Restart  Phase #:46 Files: 1      Time: 1s
Serial   Phase #:47 Files: 2      Time: 408s
Restart  Phase #:48 Files: 1      Time: 1s
Serial   Phase #:49 Files: 2      Time: 560s
Restart  Phase #:50 Files: 1      Time: 1s
Serial   Phase #:51 Files: 2      Time: 2950s
Restart  Phase #:52 Files: 1      Time: 0s
Serial   Phase #:53 Files: 1      Time: 2s
Serial   Phase #:54 Files: 1      Time: 296s
Serial   Phase #:55 Files: 1      Time: 509s
Serial   Phase #:56 Files: 1      Time: 34s
Grand Total Time: 6042s

Upgrade to the new Oracle home in the cluster
srvctl upgrade database -d DBA12C -o /apps/oracle/product/12.1.0.1/racdb
alter system set cluster_database=TRUE scope=spfile sid=’*’;
startup
@utlu121s.sql
@catuppst.sql

@utlrp.sql

Steps to upgrade 11.2.0.3 to 12.1.0.1 Grid Infrastructure


Below are the summary steps to upgrade 11.2.0.3 to 12.1.0.1 Grid Infrastructure

./runcluvfy.sh stage -pre crsinst -upgrade -n gls101,gls102 -rolling -src_crshome /apps/oracle/product/11.2.0.3/grid -dest_crshome /apps/oracle/product/12.1.0.1/grid -dest_version 12.1.0.1.0 -fixup–verbose

mkdir /apps/oracle/product/12.1.0.1/grid
chown –R oracle:oinstall /apps/oracle/product/12.1.0.1/grid
chmod -R 775 /apps/oracle/product/12.1.0.1/grid
unset ORACLE_BASE
unset ORACLE_HOME
unset ORACLE_SID
unset ORA_CRS_HOME
gls101:/data/oracle/bkup1/software/12.1.0.1/grid
> ./runInstaller
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 1803 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 6453 MB    Passed
Checking monitor: must be configured to display at least 256 colors.    Actual 65536    Passed

Installation Option --> UPgrade Oracle Grid Infrastructure or Oracle Automatic Storage Management
Node Selection --> Select the Cluster nodes (in addition to the local node) in the hardware clustere where the Installer should upgrade Grid Infrastructure
Grid Infrastructure Management --> Configure Grid Infrasture Management Repository --> Yes
Select the name of the operating system group, that you want to use for operating system autehntication to Oracle Automatic Storage Management --> Choose orainstall or dba
Specify the Oracle Base

If you get a warning on the package nfs-utils-1.0.9-60.el5.x86_64.rpm, you can install/update it

rpm -Uvh nfs-utils-1.0.9-60.el5.x86_64.rpm
warning: nfs-utils-1.0.9-60.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 192a7d7d
Preparing...                ########################################### [100%]
   1:nfs-utils              ########################################### [100%]

gls101:/data/oracle/bkup1/software/12.1.0.1
> /apps/oracle/product/12.1.0.1/grid/rootupgrade.sh
Performing root user operation for Oracle 12c 

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /apps/oracle/product/12.1.0.1/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /apps/oracle/product/12.1.0.1/grid/crs/install/crsconfig_params

ASM upgrade has started on first node.

OLR initialization - successful
2013/07/17 17:30:03 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2013/07/17 17:34:22 CLSRSC-343: Successfully started Oracle clusterware stack

clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2013/07/17 17:35:45 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

On the second node, run rootupgrade.sh


gls102:/data/oracle/bkup1/software/12.1.0.1
> /apps/oracle/product/12.1.0.1/grid/rootupgrade.sh
Performing root user operation for Oracle 12c 

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /apps/oracle/product/12.1.0.1/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /apps/oracle/product/12.1.0.1/grid/crs/install/crsconfig_params
OLR initialization - successful
2013/07/17 17:40:12 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2013/07/17 17:45:01 CLSRSC-343: Successfully started Oracle clusterware stack

clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 1.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Start upgrade invoked..
Started to upgrade the Oracle Clusterware. This operation may take a few minutes.
Started to upgrade the OCR.
Started to upgrade the CSS.
The CSS was successfully upgraded.
Started to upgrade Oracle ASM.
Started to upgrade the CRS.
The CRS was successfully upgraded.
Oracle Clusterware operating version was successfully set to 12.1.0.1.0
2013/07/17 17:48:28 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

When creating Grid Infrastructure  management database, you might run into issue of the OCR/Vote diskgroup runing out of space.  For example,   +GI does not have enough space. Required space is 3358 MB , available space is 2771 MB.

To add more disk to GI diskgroup 

SQL> alter diskgroup GI add disk 'ORCL:GI_DISK01';

Diskgroup altered.