Transaction Guard and Application Continuity

Transaction Guard:  A new feauture in version 12c.  Transaction guard provides a way to keep track of transaction state by using a logical transaction identifier which maps to a dataabase transactio_id.  If there is a failure, the applciation connection can query the transaction status using logical transaction_id to determine if a transaction is commited or not.  There are two parameters when creating a service as transaciton guard.  commit_outcome:  to determine if a logical transaction is tracked or not.  retention how long to retend in the database (default 86400 seconds = 1 day).

srvctl add service -d dba12c -service TGDBA12C -preferred DBA12C1 -available DBA12C2 -commit_outcome TRUE -retention 86400

Application Continuity:  
Application Continuity is a new feature in 12c using the transaction guard.  It attempts to mask the outage by rebuilding the database session and resubmitting the pending work.  It used client-side JDBC driver 12c, Universal connection Pool 12c, and Oracle WeblOGIC sERVER 12.1.2 class.

-failovertype TRANSACTION
-commit_outcome  TRUE

srvctl add service -d dba12c -service ACDBA12C -preferred DBA12C1 -available DBA12C2 -commit_outcome TRUE -retention 86400 -failovertype TRANSACTION -failoverretry 10 -failoverdelay 5 -replay_int_time 1200

Creating Services with Pluggable database in RAC


For applications to access the Pluggable database, you need to create services for the PDB.  

Example:  Create order_oltp service associated with pocpdb1 (PDB) prefered on poccdb1(Container DB instance 1) and available on poccdb2 (Container DB instance 2)

srvctl add service -db poccdb -service order_oltp -pdb pocpdb1 -preferred poccdb1 -available poccdb2

srvctl start service -d poccdb -s order_oltp

SQL> connect sys@pocpdb1 as sysdba
Enter password:
Connected.
SQL> show con_name
POCPDB1

To check services of the pluggable database

select name, pdb from all_services;

From clusterware, you can check the service configuration of a container database using srvctl command

poccdb1 > srvctl config service -db poccdb
Service name: order_oltp
Service is enabled
Server pool: poccdb_order_oltp
Cardinality: 1
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Global: false
Commit Outcome: false
Failover type:
Failover method:
TAF failover retries:
TAF failover delay:
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name: pocpdb1
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Replay Initiation Time: 300 seconds
Session State Consistency:
Preferred instances: poccdb1
Available instances: poccdb2

Creating order_oltp Net Service name for Pluggale database

After define the service with property of the pluggable database, you can create the net service name order_oltp in the tnsnames.ora file for applications to access the PDBs.  The connection string should look like this..

ORDER =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 12cracpoc-scan)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = order_oltp)
    )
  )

sqlplus > connect sys@ORDER as sysdba
Enter password:

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

Oracle 12c ASM New Features Notes

ASM Fast Mirror Resync
Enabled when COMPATIBLE.RDBMS >=11.1
ASM fast mirror resync is used to deal with transient disk failures.  When a disk goes offline, ASM tracks the extents that are modified during the outage.  When the transient failure is repaired, ASM can quickly resynchronize only the disk extents that have been modified during the outage.  Note that the tracking mechanism uses one bit for each modified extent and is very efficient.

When using ASM fast mirror resync, the failed disk is taken offline and if you set the DISK_REPAIRE_TIME attribute for the correponding disk group, the disk won't be dropped.  The setting for this attribute determines the duration of disk outages that ASM will tolerate while still being able to resynchronize after the failed disk is repaired.  The default setting for the DISK_REPAIR_TIME attribute is 3.6 hours.  If a disk remains offline longer than the DISK_REPAIR_TIME setting, the disk is dropped from the diskgroup and the diskgroup is rebalanced.

Controlling the Resources Used by Resync

Power limit an set for disk resync operations:  It is conceptually similar to the power limit setting for disk group rebalance.  The Range is 1 to 1024, and if not specified the default setting s 1

alter diskgroup DATA ONLINE DISK data_000 POWER 99;
asmcmd > online -g DATA -D data_000 --power 99
alter diskgroup DATA ONLINE ALL POWER 500

More Efficient Disk Replacement
Administrators have the option to replace disk as a fast and efficient operation.  The replacement disk is populated with mirror copies of the ASM extents from from other disks and there is no need for reorganization or rebalancing of the disk group.  The replacement disk takes the same name as the original disk and becomes part of the same failure group as the original disk.

alter diskgroup GLS_DATA replace disk DATA_0001 with '/dev/disk002';

Dealing with Transient Falure on a Failure Group
SQL> alter diskgroup DBA_GLS set attribute 'failgroup_repair_time'='48h';
Diskgroup altered.
With Oracle 12c, ASM provides administrators with the ability to specify a failure group repair time in addition to the disk repair time.  The failure group repair time can be set by using a new disk group attribuute, FAILGROUP_REPAIRT_TIME.  The default failure group repair time is 24 hours.  Now by default, if all disks in a failure group fail simultaneously, ASM will not drop the disks when the disk repair time expires after 3.6 hrs.  Rather, the administrator will have 24 hours to correct the failure.  If the failure is corrected before the time expires, the disks are updated wusing a fast mirror resync.  If the failure is not corrected in time, ASM must automatically drop all disks in the failure group.

Resync Time Estimate

Check v$ASM_OPERTION for resync operation estimate
select pass, state, est_minutes from v$asm_operation;

Resync Checkpoint and Auto-Restart
With Oracle 12c ASM interrupted resync operations are automatically restarted.  The resync operations are broken into phases:  A checkpoint marks the end of each resync phase, and stale extent metadata is cleared.  Interrupted resync operations restart from the last checkpoint prior to the interruption.

Rebalance Work Estimates
Oracle 12c ASM, a more detailed and more accurate work plan is created at the begining of each rebalance operation.  The admin can generate the work plan by using the EXPLAN WORK command and the work can be viewed in v$ASM_ESTIMATE view (EST_WORK column shows the ASM allocateion units to be moved).

SQL> explain work set statement_id='Drop DATA_DISK15' for alter diskgroup DBA_DATA501 drop disk DATA_DISK15;
Explained.

SQL> select est_work from v$asm_estimate where statement_id='Drop DATA_DISK15';
  EST_WORK
----------
    111168

Priority Ordered Rebalance
 With Oracle 12c ASM, priority ordered rebalance is implemented.  This capability helps to restoring the redundancy of critical files like control files and online redo logs to ensure that they're protected in case of a secondary failure occurs.

Proactively Validating Data Integrity
ASM 12c provides proactive scrubbing capabilities that check for logical corruptions and automatically repair them where possible.  Two ways of scrubbing:  On-demand  by administrator on specific area, Occur as part of rebalance operation

SQL> alter diskgroup DBA_GLS501 scrub repair;
Diskgroup altered.

SQL> alter diskgroup DBA_DATA501 scrub file '+DATA_DISK/GLSLAB/DATAFILE/system.553.826023617' repair wait;
Diskgroup altered.

SQL> alter diskgroup DBA_DATA501 scrub disk DATA_DISK15 repair power max force;
Diskgroup altered

REPAIR:  If the repair option is not specified, ASM only check and report logical corruption
POWER:  LOW, HIGH, or MAX.  If power is not specified, the scrubbing power is controlled based on the system I/O load
FORCE:  Command is processed immediately regarless of system load

Proactive Content Checking during rebalance
Checks are automatic with automatic error correction for mirrored data and checks are enabled with the disk group attribute content.check

SQL> alter diskgroup DBA_DATA501 set attribute 'content.check' = 'TRUE';
Diskgroup altered.

Managing Password Files in ASM

Create ASM password file
+ASM1 > orapwd asm=y file='+GLS_DATA501' password='Password4dba'

Create database password file
ASMCMD> pwcreate --dbuniquename GLSLAB  +DBA_DATA laga4me

Locate password file
+ASM1 > crsctl stat res ora.glslab.db -f | grep PWFILE
PWFILE=+DATA_DISK/GLSLAB/PASSWORD/pwdglslab.464.827084721

Specifying Content Type for a Disk Group
Possible values:  data, recovery or system

Flex ASM - Migrate standard ASM to Flex ASM

Perform the following steps to migrate a standard ASM to flex ASM

export DISPLAY=glsdb01.global:1
 
asmca -> 
 
Under ASM Istances ->ASM listener Port, Select the network(s) to be used for ASM, Click OK, Click Convert to Oracle Flex ASM
 
gls01:/apps/oracle/grid/cfgtoollogs/asmca/scripts

-rwxr-xr-x 1 oracle oinstall 620 Sep 24 13:19 converttoFlexASM.sh

+ASM1 > cat converttoFlexASM.sh | more
#!/bin/sh

su oracle -c "/apps/oracle/product/12.1.0.1/grid/bin/srvctl add asm -proxy"
/apps/oracle/product/12.1.0.1/grid/srvm/admin/srvmhelper -updateDepASM
for node in glsdb01 glsdb02
do
        /apps/oracle/product/12.1.0.1/grid/bin/crsctl stop cluster -n $node
        /apps/oracle/product/12.1.0.1/grid/bin/crsctl start cluster -n $node

        stat=1
        cmd="/apps/oracle/product/12.1.0.1/grid/bin/crsctl status server $node"
        while [ $stat = 1 ]
        do
                sleep 5
                op=`$cmd`
                op=`echo $op | /bin/cut -d' ' -f2  | /bin/cut -d'=' -f2`
                if [ "$op" = "ONLINE" ]
                then
                        stat=0
                        echo "Cluster bounced in node $node"
                fi
        done
done

gls01:/apps/oracle/grid/cfgtoollogs/asmca/scripts 
 
As root:
 
/apps/oracle/grid/cfgtoollogs/asmca/scripts/converttoFlexASM.sh

CRS-2673: Attempting to stop 'ora.crsd' on 'glsdb01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'glsdb01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'glsdb01'
CRS-2673: Attempting to stop 'ora.dba12c.db' on 'glsdb01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'glsdb01'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'glsdb01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.glsdb01.vip' on 'glsdb01'
CRS-2677: Stop of 'ora.dba12c.db' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'glsdb01'
CRS-2673: Attempting to stop 'ora.dbatools.db' on 'glsdb01'
CRS-2677: Stop of 'ora.scan1.vip' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'glsdb02'
CRS-2677: Stop of 'ora.glsdb01.vip' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.glsdb01.vip' on 'glsdb02'
CRS-2677: Stop of 'ora.dbatools.db' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.DBA_DATA501.dg' on 'glsdb01'
CRS-2676: Start of 'ora.scan1.vip' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'glsdb02'
CRS-2677: Stop of 'ora.DBA_DATA501.dg' on 'glsdb01' succeeded
CRS-2676: Start of 'ora.glsdb01.vip' on 'glsdb02' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'glsdb02' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.GI.dg' on 'glsdb01'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'glsdb01'
CRS-2677: Stop of 'ora.MGMTLSNR' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'glsdb02'
CRS-2677: Stop of 'ora.GI.dg' on 'glsdb01' succeeded
CRS-2676: Start of 'ora.MGMTLSNR' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'glsdb02'
CRS-5017: The resource action "ora.mgmtdb start" encountered the following error: 
CRS-2679: Attempting to clean 'ora.mgmtdb' on 'glsdb02'
CRS-2681: Clean of 'ora.mgmtdb' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'glsdb01'
CRS-2677: Stop of 'ora.asm' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'glsdb01'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'glsdb01'
CRS-2677: Stop of 'ora.ons' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'glsdb01'
CRS-2677: Stop of 'ora.net1.network' on 'glsdb01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'glsdb01' has completed
CRS-2677: Stop of 'ora.crsd' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'glsdb01'
CRS-2673: Attempting to stop 'ora.evmd' on 'glsdb01'
CRS-2673: Attempting to stop 'ora.storage' on 'glsdb01'
CRS-2677: Stop of 'ora.storage' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'glsdb01'
CRS-2677: Stop of 'ora.ctssd' on 'glsdb01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'glsdb01' succeeded
CRS-2677: Stop of 'ora.asm' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'glsdb01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'glsdb01'
CRS-2677: Stop of 'ora.cssd' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'glsdb01'
CRS-2676: Start of 'ora.evmd' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'glsdb01'
CRS-2676: Start of 'ora.cssdmonitor' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'glsdb01'
CRS-2672: Attempting to start 'ora.diskmon' on 'glsdb01'
CRS-2676: Start of 'ora.diskmon' on 'glsdb01' succeeded
CRS-2676: Start of 'ora.cssd' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'glsdb01'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'glsdb01'
CRS-2676: Start of 'ora.ctssd' on 'glsdb01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'glsdb01'
CRS-2676: Start of 'ora.asm' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'glsdb01'
CRS-2676: Start of 'ora.storage' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'glsdb01'
CRS-2676: Start of 'ora.crsd' on 'glsdb01' succeeded
Cluster bounced in node glsdb01
CRS-2673: Attempting to stop 'ora.crsd' on 'glsdb02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'glsdb02'
CRS-2673: Attempting to stop 'ora.oc4j' on 'glsdb02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'glsdb02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'glsdb02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'glsdb02'
CRS-2673: Attempting to stop 'ora.cvu' on 'glsdb02'
CRS-2677: Stop of 'ora.cvu' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'glsdb01'
CRS-2676: Start of 'ora.cvu' on 'glsdb01' succeeded
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'glsdb02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'glsdb02'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'glsdb02'
CRS-2677: Stop of 'ora.scan2.vip' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'glsdb01'
CRS-2677: Stop of 'ora.scan3.vip' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'glsdb01'
CRS-2676: Start of 'ora.scan2.vip' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'glsdb01'
CRS-2673: Attempting to stop 'ora.dba12c.db' on 'glsdb02'
CRS-2673: Attempting to stop 'ora.dbatools.db' on 'glsdb02'
CRS-2673: Attempting to stop 'ora.oggdb.db' on 'glsdb02'
CRS-2673: Attempting to stop 'ora.glsdb02.vip' on 'glsdb02'
CRS-2676: Start of 'ora.scan3.vip' on 'glsdb01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'glsdb01'
CRS-2677: Stop of 'ora.glsdb02.vip' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.glsdb02.vip' on 'glsdb01'
CRS-2677: Stop of 'ora.dba12c.db' on 'glsdb02' succeeded
CRS-2677: Stop of 'ora.dbatools.db' on 'glsdb02' succeeded
CRS-2676: Start of 'ora.glsdb02.vip' on 'glsdb01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'glsdb01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'glsdb01' succeeded
CRS-2677: Stop of 'ora.oggdb.db' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.DBA_DATA501.dg' on 'glsdb02'
CRS-2677: Stop of 'ora.DBA_DATA501.dg' on 'glsdb02' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'glsdb01'
CRS-2673: Attempting to stop 'ora.GI.dg' on 'glsdb02'
CRS-2677: Stop of 'ora.GI.dg' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.dba12c.dba12c_rac.svc' on 'glsdb01'
CRS-2673: Attempting to stop 'ora.dba12c.dba12c_rpt.svc' on 'glsdb01'
CRS-2677: Stop of 'ora.dba12c.dba12c_rac.svc' on 'glsdb01' succeeded
CRS-2677: Stop of 'ora.dba12c.dba12c_rpt.svc' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.dba12c.db' on 'glsdb01'
CRS-2677: Stop of 'ora.dba12c.db' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'glsdb02'
CRS-2677: Stop of 'ora.asm' on 'glsdb02' succeeded
CRS-2676: Start of 'ora.oc4j' on 'glsdb01' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'glsdb02'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'glsdb02'
CRS-2677: Stop of 'ora.ons' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'glsdb02'
CRS-2677: Stop of 'ora.net1.network' on 'glsdb02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'glsdb02' has completed
CRS-2677: Stop of 'ora.crsd' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'glsdb02'
CRS-2673: Attempting to stop 'ora.evmd' on 'glsdb02'
CRS-2673: Attempting to stop 'ora.storage' on 'glsdb02'
CRS-2677: Stop of 'ora.storage' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'glsdb02'
CRS-2677: Stop of 'ora.ctssd' on 'glsdb02' succeeded
CRS-2677: Stop of 'ora.evmd' on 'glsdb02' succeeded
CRS-2677: Stop of 'ora.asm' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'glsdb02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'glsdb02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'glsdb02'
CRS-2677: Stop of 'ora.cssd' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'glsdb02'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'glsdb02'
CRS-2676: Start of 'ora.evmd' on 'glsdb02' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'glsdb02'
CRS-2672: Attempting to start 'ora.diskmon' on 'glsdb02'
CRS-2676: Start of 'ora.diskmon' on 'glsdb02' succeeded
CRS-2676: Start of 'ora.cssd' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'glsdb02'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'glsdb02'
CRS-2676: Start of 'ora.ctssd' on 'glsdb02' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'glsdb02'
CRS-2676: Start of 'ora.asm' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'glsdb02'
CRS-2676: Start of 'ora.storage' on 'glsdb02' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'glsdb02'
CRS-2676: Start of 'ora.crsd' on 'glsdb02' succeeded
Cluster bounced in node glsdb02

real    7m3.449s
user    0m1.587s
sys     0m0.435s
You have new mail in /var/spool/mail/oracle