You don’t have to manually remove directories. There is a deinstall script in the grid home directory. The deinstall checks and prompt questions to confirm before it performs a clean deinstallation (see below).
cd /apps/grid/11.2.0/grid/deinstall
LADB1 - oracle: ./deinstall
Checking for required files and bootstrapping ...
Please wait ...
Location of logs /tmp/deinstall2012-05-23_02-25-20PM/logs/
############ ORACLE DEINSTALL & DECONFIG TOOL START ############
######################### CHECK OPERATION START #########################
## [START] Install check configuration ##
Checking for existence of the Oracle home location /apps/11.2.0/grid
Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Cluster
Oracle Base selected for deinstall is: /apps/oracle
Checking for existence of central inventory location /apps/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /apps/11.2.0/grid
The following nodes are part of this cluster: dbsrvl100,dbsrvl101
Checking for sufficient temp space availability on node(s) : 'dbsrvl100,dbsrvl101'
## [END] Install check configuration ##
Traces log file: /tmp/deinstall2012-05-23_02-25-20PM/logs//crsdc.log
Network Configuration check config START
Network de-configuration trace file location: /tmp/deinstall2012-05-23_02-25-20PM/logs/netdc_check2012-05-23_02-25-48-PM.log
Specify all RAC listeners (do not include SCAN listener) that are to be de-configured [LISTENER]:
Network Configuration check config END
Asm Check Configuration START
ASM de-configuration trace file location: /tmp/deinstall2012-05-23_02-25-20PM/logs/asmcadc_check2012-05-23_02-37-14-PM.log
Automatic Storage Management (ASM) instance is detected in this Oracle home /apps/11.2.0/grid.
ASM Diagnostic Destination : /apps/oracle
ASM Diskgroups : +GI
ASM diskstring : /dev/oracleasm/disks
Diskgroups will be dropped
De-configuring ASM will drop all the diskgroups and it's contents at cleanup time. This will affect all of the databases and ACFS that use this ASM instance(s).
If you want to retain the existing diskgroups or if any of the information detected is incorrect, you can modify by entering 'y'. Do you want to modify above information (y|n) [n]: y
Specify the ASM Diagnostic Destination [/apps/oracle]:
Specify the diskstring [/dev/oracleasm/disks]:
Specify the diskgroups that are managed by this ASM instance [+GI]:
De-configuring ASM will drop the diskgroups at cleanup time. Do you want deconfig tool to drop the diskgroups y|n [y]: y
######################### CHECK OPERATION END #########################
####################### CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /apps/11.2.0/grid
The cluster node(s) on which the Oracle home deinstallation will be performed are:dbsrvl100,dbsrvl101
Oracle Home selected for deinstall is: /apps/11.2.0/grid
Inventory Location where the Oracle home registered is: /apps/oraInventory
Following RAC listener(s) will be de-configured: LISTENER
ASM instance will be de-configured from this Oracle home
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/tmp/deinstall2012-05-23_02-25-20PM/logs/deinstall_deconfig2012-05-23_02-25-25-PM.out'
Any error messages from this session will be written to: '/tmp/deinstall2012-05-23_02-25-20PM/logs/deinstall_deconfig2012-05-23_02-25-25-PM.err'
######################## CLEAN OPERATION START ########################
ASM de-configuration trace file location: /tmp/deinstall2012-05-23_02-25-20PM/logs/asmcadc_clean2012-05-23_02-38-32-PM.log
ASM Clean Configuration START
ASM Clean Configuration END
Network Configuration clean config START
Network de-configuration trace file location: /tmp/deinstall2012-05-23_02-25-20PM/logs/netdc_clean2012-05-23_02-40-15-PM.log
De-configuring RAC listener(s): LISTENER
De-configuring listener: LISTENER
Stopping listener: LISTENER
Listener stopped successfully.
Unregistering listener: LISTENER
Listener unregistered successfully.
Listener de-configured successfully.
De-configuring Naming Methods configuration file on all nodes...
Naming Methods configuration file de-configured successfully.
De-configuring Local Net Service Names configuration file on all nodes...
Local Net Service Names configuration file de-configured successfully.
De-configuring Directory Usage configuration file on all nodes...
Directory Usage configuration file de-configured successfully.
De-configuring backup files on all nodes...
Backup files de-configured successfully.
The network configuration has been cleaned up successfully.
Network Configuration clean config END
---------------------------------------->
The deconfig command below can be executed in parallel on all the remote nodes. Execute the command on the local node after the execution completes on all the remote nodes.
Run the following command as the root user or the administrator on node "dbsrvl101".
/tmp/deinstall2012-05-23_02-25-20PM/perl/bin/perl -I/tmp/deinstall2012-05-23_02-25-20PM/perl/lib -I/tmp/deinstall2012-05-23_02-25-20PM/crs/install /tmp/deinstall2012-05-23_02-25-20PM/crs/install/rootcrs.pl -force -deconfig -paramfile "/tmp/deinstall2012-05-23_02-25-20PM/response/deinstall_Ora11g_gridinfrahome1.rsp"
Run the following command as the root user or the administrator on node "dbsrvl100".
/tmp/deinstall2012-05-23_02-25-20PM/perl/bin/perl -I/tmp/deinstall2012-05-23_02-25-20PM/perl/lib -I/tmp/deinstall2012-05-23_02-25-20PM/crs/install /tmp/deinstall2012-05-23_02-25-20PM/crs/install/rootcrs.pl -force -deconfig -paramfile "/tmp/deinstall2012-05-23_02-25-20PM/response/deinstall_Ora11g_gridinfrahome1.rsp" -lastnode
Press Enter after you finish running the above commands
<----------------------------------------
Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START
Detach Oracle home '/apps/11.2.0/grid' from the central inventory on the local node : Done
Delete directory '/apps/11.2.0/grid' on the local node : Done
Delete directory '/apps/oraInventory' on the local node : Done
The Oracle Base directory '/apps/oracle' will not be removed on local node. The directory is not empty.
Detach Oracle home '/apps/11.2.0/grid' from the central inventory on the remote nodes 'dbsrvl101' : Done
Delete directory '/apps/11.2.0/grid' on the remote nodes 'dbsrvl101' : Done
Delete directory '/apps/oraInventory' on the remote nodes 'dbsrvl101' : Done
The Oracle Base directory '/apps/oracle' will not be removed on node 'dbsrvl101'. The directory is not empty.
Oracle Universal Installer cleanup was successful.
Oracle Universal Installer clean END
## [START] Oracle install clean ##
Clean install operation removing temporary directory '/tmp/deinstall2012-05-23_02-25-20PM' on node 'dbsrvl100'
Clean install operation removing temporary directory '/tmp/deinstall2012-05-23_02-25-20PM' on node 'dbsrvl101'
## [END] Oracle install clean ##
######################### CLEAN OPERATION END #########################
####################### CLEAN OPERATION SUMMARY #######################
ASM instance was de-configured successfully from the Oracle home
Following RAC listener(s) were de-configured successfully: LISTENER
Oracle Clusterware is stopped and successfully de-configured on node "dbsrvl101"
Oracle Clusterware is stopped and successfully de-configured on node "dbsrvl100"
Oracle Clusterware is stopped and de-configured successfully.
Successfully detached Oracle home '/apps/11.2.0/grid' from the central inventory on the local node.
Successfully deleted directory '/apps/11.2.0/grid' on the local node.
Successfully deleted directory '/apps/oraInventory' on the local node.
Successfully detached Oracle home '/apps/11.2.0/grid' from the central inventory on the remote nodes 'dbsrvl101'.
Successfully deleted directory '/apps/11.2.0/grid' on the remote nodes 'dbsrvl101'.
Successfully deleted directory '/apps/oraInventory' on the remote nodes 'dbsrvl101'.
Oracle Universal Installer cleanup was successful.
Run 'rm -rf /etc/oraInst.loc' as root on node(s) 'dbsrvl100,dbsrvl101' at the end of the session.
Run 'rm -rf /opt/ORCLfmap' as root on node(s) 'dbsrvl100,dbsrvl101' at the end of the session.
Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################
############# ORACLE DEINSTALL & DECONFIG TOOL END #############
strace - trace system calls and signals
strace - trace system calls and signals. It's a useful tool for diagnosis, debugging processes on Linux.
To install:
- root: yum install strace
Loaded plugins: rhnplugin, security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package strace.x86_64 0:4.5.18-11.el5_8 set to be updated
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
strace x86_64 4.5.18-11.el5_8 rhel-x86_64-server-5 177 k
Transaction Summary
================================================================================
Install 1 Package(s)
Upgrade 0 Package(s)
Total download size: 177 k
Is this ok [y/N]: y
Downloading Packages:
strace-4.5.18-11.el5_8.x86_64.rpm | 177 kB 00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : strace
Installed:
strace.x86_64 0:4.5.18-11.el5_8
Complete!
Some useful parameters:
-tt If given twice, the time printed will include the microseconds.
-T Show the time spent in system calls. This records the time difference between the beginning and the end of each system call.
-o filename : Write the trace output to the file filename rather than to screen (stderr).
-p PID Attach to the process with the process ID pid and begin tracing. The trace may be terminated at any time by a keyboard interrupt signal (hit CTRL-C). strace will respond by detaching itself from the traced process(es) leaving it (them) to continue running. Multiple -p options can be used to attach to up to 32 processes in addition to command (which is optional if at least one -p option is given).
-s SIZE Specify the maximum string size to print (the default is 32).
Examples:
To trace an oracle process and see what it's doing..
strace -p 28446 -s 100 -o /tmp/orammnl.txt
to trace the open and read system calls
strace -e trace=open,read -p 28446 -o /tmp/orammnl.debug
to trace the process spends time on system calls
strace -ttT -p 15878 -o /tmp/tracetiming.out
14:54:26.232457 open("/proc/15914/stat", O_RDONLY) = 26 <0.000028>
14:54:26.232584 read(26, "15914 (oracle) S 1 15914 15914 0"..., 999) = 243 <0.000049>
14:54:26.232884 close(26) = 0 <0.000019>
14:54:26.233002 open("/proc/15916/stat", O_RDONLY) = 26 <0.000027>
14:54:26.233127 read(26, "15916 (oracle) S 1 15916 15916 0"..., 999) = 240 <0.000046>
14:54:26.233269 close(26) = 0 <0.000019>
...
At 14:54:26.233127, it took 0.46 milisecond to read 240 bytes.
To install:
- root: yum install strace
Loaded plugins: rhnplugin, security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package strace.x86_64 0:4.5.18-11.el5_8 set to be updated
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
strace x86_64 4.5.18-11.el5_8 rhel-x86_64-server-5 177 k
Transaction Summary
================================================================================
Install 1 Package(s)
Upgrade 0 Package(s)
Total download size: 177 k
Is this ok [y/N]: y
Downloading Packages:
strace-4.5.18-11.el5_8.x86_64.rpm | 177 kB 00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : strace
Installed:
strace.x86_64 0:4.5.18-11.el5_8
Complete!
Some useful parameters:
-tt If given twice, the time printed will include the microseconds.
-T Show the time spent in system calls. This records the time difference between the beginning and the end of each system call.
-o filename : Write the trace output to the file filename rather than to screen (stderr).
-p PID Attach to the process with the process ID pid and begin tracing. The trace may be terminated at any time by a keyboard interrupt signal (hit CTRL-C). strace will respond by detaching itself from the traced process(es) leaving it (them) to continue running. Multiple -p options can be used to attach to up to 32 processes in addition to command (which is optional if at least one -p option is given).
-s SIZE Specify the maximum string size to print (the default is 32).
Examples:
To trace an oracle process and see what it's doing..
strace -p 28446 -s 100 -o /tmp/orammnl.txt
to trace the open and read system calls
strace -e trace=open,read -p 28446 -o /tmp/orammnl.debug
to trace the process spends time on system calls
strace -ttT -p 15878 -o /tmp/tracetiming.out
14:54:26.232457 open("/proc/15914/stat", O_RDONLY) = 26 <0.000028>
14:54:26.232584 read(26, "15914 (oracle) S 1 15914 15914 0"..., 999) = 243 <0.000049>
14:54:26.232884 close(26) = 0 <0.000019>
14:54:26.233002 open("/proc/15916/stat", O_RDONLY) = 26 <0.000027>
14:54:26.233127 read(26, "15916 (oracle) S 1 15916 15916 0"..., 999) = 240 <0.000046>
14:54:26.233269 close(26) = 0 <0.000019>
...
At 14:54:26.233127, it took 0.46 milisecond to read 240 bytes.
RDA TLsecure tool
RDA TLsecure tool can help identify potential database security risk. See the list below for checking..
ovmrac01:/apps/oracle/product/11.2.0.3/RACDB/rda
LADB01 - oracle: ./rda.pl -vT secure
Testing ...
Identification of Potential Security Risks ...
Identification of Potential Security Risks
1 Users with a default password (11g)
2 Users with a well known password
3 Users not visible in dba_users
4 Operating system authenticated user names with a password
5 Users with SYSTEM as default tablespace
6 Users and privileges from gv$pwfile_users
7 Privileges not granted by their owner
8 Listeners without a password
9 Listeners with local operating system authentication (10g and later)
10 Listeners modifiable at runtime
11 AUTHENTICATION_SERVICES value in sqlnet.ora files
12 Oracle executables owned by different users of a same group (UNIX only)
* Run all checks
Enter a menu item or . to end
> *
ovmrac01:/apps/oracle/product/11.2.0.3/RACDB/rda
LADB01 - oracle: ./rda.pl -vT secure
Testing ...
Identification of Potential Security Risks ...
Identification of Potential Security Risks
1 Users with a default password (11g)
2 Users with a well known password
3 Users not visible in dba_users
4 Operating system authenticated user names with a password
5 Users with SYSTEM as default tablespace
6 Users and privileges from gv$pwfile_users
7 Privileges not granted by their owner
8 Listeners without a password
9 Listeners with local operating system authentication (10g and later)
10 Listeners modifiable at runtime
11 AUTHENTICATION_SERVICES value in sqlnet.ora files
12 Oracle executables owned by different users of a same group (UNIX only)
* Run all checks
Enter a menu item or . to end
> *
SQL Monitor , /*+ MONITOR */, DBMS_SQLTUNE.REPORT_SQL_MONITOR
The SQL monitor is introduced in Oracle 11g database is a very effective way to monitor long running SQL statements. By default, SQL monitoring automatically starts when a SQL statement runs parallel, or when it has consumed at least 5 seconds of CPU or I/O time in a single execution. The /*+MONITOR*/ hint enables monitoring for statements and the DBMS_SQLTUNE.REPORT_SQL_MONITOR package creates a monitoring report for a specific SQL statement. It's interactive with the Enterprise Manager and statistics performance metrics are capture (elapsed time, CPU time, I/O time, reads/writes related, etc.
Below are few examples:
Example1 /*+ MONITOR */ hint
------------------------------------------------
SQLPLUS >
select /*+ MONITOR */ emp_nbr from DBALAB.EMPLOYEE where emp_nbr is not null and rownum <=200;
set long 1000000
set longchunksize 1000000
set linesize 600
set pagesize 0
set trim on
set trimspool on
select dbms_sqltune.report_sql_monitor (session_id=>sys_context('userenv','sid'), report_level=>'ALL') as report from dual;
SQL Monitoring Report
SQL Text
------------------------------
select /*+ MONITOR */ emp_nbr from DBALAB.EMPLOYEE where emp_nbr is not null and rownum <=200
Global Information
------------------------------
Status : DONE (ALL ROWS)
Instance ID : 1
Session : SYS (689:10593)
SQL ID : 9hhhwr7ns2y9y
SQL Execution ID : 16777216
Execution Started : 05/04/2012 15:02:55
First Refresh Time : 05/04/2012 15:02:55
Last Refresh Time : 05/04/2012 15:02:55
Duration : .425631s
Module/Action : sqlplus@vmsrv01.la.com (TNS V1-V3)/-
Service : SYS$USERS
Program : sqlplus@vmsrv01.la.com (TNS V1-V3)
Fetch Calls : 15
Global Stats
======================================================================================
| Elapsed | Cpu | IO | Cluster | Other | Fetch | Buffer | Read | Read |
| Time(s) | Time(s) | Waits(s) | Waits(s) | Waits(s) | Calls | Gets | Reqs | Bytes |
======================================================================================
| 0.43 | 0.00 | 0.41 | 0.00 | 0.01 | 15 | 23 | 3 | 112KB |
======================================================================================
SQL Plan Monitoring Details (Plan Hash Value=238925293)
==========================================================================================================================================================
| Id | Operation | Name | Rows | Cost | Time | Start | Execs | Rows | Read | Read | Activity | Activity Detail |
| | | | (Estim) | | Active(s) | Active | | (Actual) | Reqs | Bytes | (%) | (# samples) |
==========================================================================================================================================================
| 0 | SELECT STATEMENT | | | | 1 | +0 | 1 | 200 | | | | |
| 1 | COUNT STOPKEY | | | | 1 | +0 | 1 | 200 | | | | |
| 2 | TABLE ACCESS FULL | DBALAB.EMPLOYEE | 200 | 4 | 2 | +0 | 1 | 200 | 3 | 112KB | 100.00 | db file scattered read (1) |
==========================================================================================================================================================
Example2: DBMS_SQLTUNE.REPORT_SQL_MONITOR
-----------------------------------------------------------------
select sid, sql_id
from v$session
where status = 'ACTIVE';
SID SQL_ID
-------- -------------
232 7fnqw0qww759d
343
set long 1000000
set longchunksize 1000000
set linesize 600
set pagesize 0
set trim on
set trimspool on
select dbms_sqltune.report_sql_monitor(sql_id =>'7fnqw0qww759d', type=>'TEXT') as report from dual;
SQL Monitoring Report
SET LINESIZE 300
COLUMN sql_text FORMAT A100
SELECT sql_id, status, sql_text FROM v$sql_monitor;
acc988uzvjmmt DONE delete from WRH$_SYSMETRIC_HISTORY tab where (:beg_snap <= tab.snap_id and tab.snap_id <= :e
nd_snap and dbid = :dbid) and not exists (select 1 from WRM$_BASELINE b
where (tab.dbid = b.dbid) and (tab.snap_id >= b.start_snap_id) and
(tab.snap_id <= b.end_snap_id))
7fnqw0qww759d DONE begin dbms_feature_usage_internal.exec_db_usage_sampling(:bind1); end;
SET LONG 1000000
SET FEEDBACK OFF
SELECT DBMS_SQLTUNE.report_sql_monitor(sql_id =>'7fnqw0qww759d',type=> 'TEXT')
2 AS report FROM dual;
SQL Monitoring Report
SQL Text
------------------------------
begin dbms_feature_usage_internal.exec_db_usage_sampling(:bind1); end;
Global Information
------------------------------
Status : DONE
Instance ID : 1
Session : (1600:2927)
SQL ID : 05s9358mm6vrr
SQL Execution ID : 16777218
Execution Started : 04/30/2012 21:37:39
First Refresh Time : 04/30/2012 21:37:43
Last Refresh Time : 04/30/2012 21:37:44
Duration : 5s
Module/Action : MMON_SLAVE/0000010 FINISHED190
Service : SYS$BACKGROUND
Program : oracle@vmsrv01.la.com (M000)
Global Stats
==========================================================================================================================
| Elapsed | Cpu | IO | Application | Concurrency | Cluster | Other | Buffer | Read | Read | Write | Write |
| Time(s) | Time(s) | Waits(s) | Waits(s) | Waits(s) | Waits(s) | Waits(s) | Gets | Reqs | Bytes | Reqs | Bytes |
==========================================================================================================================
| 4.22 | 2.16 | 1.19 | 0.01 | 0.03 | 0.07 | 0.77 | 204K | 1682 | 24MB | 4 | 32768 |
==========================================================================================================================
SQL>
Below are few examples:
Example1 /*+ MONITOR */ hint
------------------------------------------------
SQLPLUS >
select /*+ MONITOR */ emp_nbr from DBALAB.EMPLOYEE where emp_nbr is not null and rownum <=200;
set long 1000000
set longchunksize 1000000
set linesize 600
set pagesize 0
set trim on
set trimspool on
select dbms_sqltune.report_sql_monitor (session_id=>sys_context('userenv','sid'), report_level=>'ALL') as report from dual;
SQL Monitoring Report
SQL Text
------------------------------
select /*+ MONITOR */ emp_nbr from DBALAB.EMPLOYEE where emp_nbr is not null and rownum <=200
Global Information
------------------------------
Status : DONE (ALL ROWS)
Instance ID : 1
Session : SYS (689:10593)
SQL ID : 9hhhwr7ns2y9y
SQL Execution ID : 16777216
Execution Started : 05/04/2012 15:02:55
First Refresh Time : 05/04/2012 15:02:55
Last Refresh Time : 05/04/2012 15:02:55
Duration : .425631s
Module/Action : sqlplus@vmsrv01.la.com (TNS V1-V3)/-
Service : SYS$USERS
Program : sqlplus@vmsrv01.la.com (TNS V1-V3)
Fetch Calls : 15
Global Stats
======================================================================================
| Elapsed | Cpu | IO | Cluster | Other | Fetch | Buffer | Read | Read |
| Time(s) | Time(s) | Waits(s) | Waits(s) | Waits(s) | Calls | Gets | Reqs | Bytes |
======================================================================================
| 0.43 | 0.00 | 0.41 | 0.00 | 0.01 | 15 | 23 | 3 | 112KB |
======================================================================================
SQL Plan Monitoring Details (Plan Hash Value=238925293)
==========================================================================================================================================================
| Id | Operation | Name | Rows | Cost | Time | Start | Execs | Rows | Read | Read | Activity | Activity Detail |
| | | | (Estim) | | Active(s) | Active | | (Actual) | Reqs | Bytes | (%) | (# samples) |
==========================================================================================================================================================
| 0 | SELECT STATEMENT | | | | 1 | +0 | 1 | 200 | | | | |
| 1 | COUNT STOPKEY | | | | 1 | +0 | 1 | 200 | | | | |
| 2 | TABLE ACCESS FULL | DBALAB.EMPLOYEE | 200 | 4 | 2 | +0 | 1 | 200 | 3 | 112KB | 100.00 | db file scattered read (1) |
==========================================================================================================================================================
Example2: DBMS_SQLTUNE.REPORT_SQL_MONITOR
-----------------------------------------------------------------
select sid, sql_id
from v$session
where status = 'ACTIVE';
SID SQL_ID
-------- -------------
232 7fnqw0qww759d
343
set long 1000000
set longchunksize 1000000
set linesize 600
set pagesize 0
set trim on
set trimspool on
select dbms_sqltune.report_sql_monitor(sql_id =>'7fnqw0qww759d', type=>'TEXT') as report from dual;
SQL Monitoring Report
SET LINESIZE 300
COLUMN sql_text FORMAT A100
SELECT sql_id, status, sql_text FROM v$sql_monitor;
acc988uzvjmmt DONE delete from WRH$_SYSMETRIC_HISTORY tab where (:beg_snap <= tab.snap_id and tab.snap_id <= :e
nd_snap and dbid = :dbid) and not exists (select 1 from WRM$_BASELINE b
where (tab.dbid = b.dbid) and (tab.snap_id >= b.start_snap_id) and
(tab.snap_id <= b.end_snap_id))
7fnqw0qww759d DONE begin dbms_feature_usage_internal.exec_db_usage_sampling(:bind1); end;
SET LONG 1000000
SET FEEDBACK OFF
SELECT DBMS_SQLTUNE.report_sql_monitor(sql_id =>'7fnqw0qww759d',type=> 'TEXT')
2 AS report FROM dual;
SQL Monitoring Report
SQL Text
------------------------------
begin dbms_feature_usage_internal.exec_db_usage_sampling(:bind1); end;
Global Information
------------------------------
Status : DONE
Instance ID : 1
Session : (1600:2927)
SQL ID : 05s9358mm6vrr
SQL Execution ID : 16777218
Execution Started : 04/30/2012 21:37:39
First Refresh Time : 04/30/2012 21:37:43
Last Refresh Time : 04/30/2012 21:37:44
Duration : 5s
Module/Action : MMON_SLAVE/0000010 FINISHED190
Service : SYS$BACKGROUND
Program : oracle@vmsrv01.la.com (M000)
Global Stats
==========================================================================================================================
| Elapsed | Cpu | IO | Application | Concurrency | Cluster | Other | Buffer | Read | Read | Write | Write |
| Time(s) | Time(s) | Waits(s) | Waits(s) | Waits(s) | Waits(s) | Waits(s) | Gets | Reqs | Bytes | Reqs | Bytes |
==========================================================================================================================
| 4.22 | 2.16 | 1.19 | 0.01 | 0.03 | 0.07 | 0.77 | 204K | 1682 | 24MB | 4 | 32768 |
==========================================================================================================================
SQL>
ORA-00600: internal error code, arguments: [ktbesc_plugged],
Recently we migrated RAC databases from 10.2.0.4 AIX to 11.2.0.3 PSU1 on Redhat using transport tablespaces. As part of our tests, we setup flashback to a gurantee restore point. After several cycle of testing, we flashed back the database to the original save point, and it completed sucessfully. However, when users inserted or deleted on tables data belonged to the tablespace that we transported, application showed error ORA-00600: internal error code, arguments: [ktbesc_plugged], [], [], [], [], [], [], [], [], [], [],
It was confirmed that we hit bug 1291956 . There are 3 solutions for this error.
1) Temporary workaround of moving the problem table to another tablespace.
alter table <table name> move tablespace <tablespace name>;
2) There is an additional workaround of setting "_fastpin_enable"=0 which is discussed in Document 12919564.8.
3) Long term resolution would be to apply one-off Patch:12919564.
We choose #3 and apply this one-off Patch. This patch overlays the PSU patch 13343438 . After applying the patch to the database home, it fixed insert/delete issues.
opatch lsinv | grep ^Patch
Patch 12919564 : applied on Mon Apr 16 13:32:53 CDT 2012
Patch 13343438 : applied on Wed Feb 22 15:56:35 CST 2012
Patch Description: "Database Patch Set Update : 11.2.0.3.1 (13343438)"
Patch 13348650 : applied on Wed Feb 22 15:56:14 CST 2012
Patch Description: "Grid Infrastructure Patch Set Update : 11.2.0.3.1 (13348650)"
opatch lsinv -bugs_fixed | grep -i 12919564
12919564 12919564 Mon Apr 16 13:32:53 CDT 2012 ENCOUNTERED ORA-600 [KTBESC_PLUGGED] WHILE
It was confirmed that we hit bug 1291956 . There are 3 solutions for this error.
1) Temporary workaround of moving the problem table to another tablespace.
alter table <table name> move tablespace <tablespace name>;
2) There is an additional workaround of setting "_fastpin_enable"=0 which is discussed in Document 12919564.8.
3) Long term resolution would be to apply one-off Patch:12919564.
We choose #3 and apply this one-off Patch. This patch overlays the PSU patch 13343438 . After applying the patch to the database home, it fixed insert/delete issues.
opatch lsinv | grep ^Patch
Patch 12919564 : applied on Mon Apr 16 13:32:53 CDT 2012
Patch 13343438 : applied on Wed Feb 22 15:56:35 CST 2012
Patch Description: "Database Patch Set Update : 11.2.0.3.1 (13343438)"
Patch 13348650 : applied on Wed Feb 22 15:56:14 CST 2012
Patch Description: "Grid Infrastructure Patch Set Update : 11.2.0.3.1 (13348650)"
opatch lsinv -bugs_fixed | grep -i 12919564
12919564 12919564 Mon Apr 16 13:32:53 CDT 2012 ENCOUNTERED ORA-600 [KTBESC_PLUGGED] WHILE
11.2.0.3 cluvfy comp healthcheck
Starting with Oracle Grid Infrastructure 11g release 2 (11.2.0.3) and later, you can use the CVU healthcheck command to check your Oracle Clusterware and Oracle Database installations for their compliance with mandatory requirements and best practices guidelines, and to ensure that they are functioning properly. In addition, If you want to perform best practices checks on these databases, then you must create the cvusys user on each database and grant that user the CVUSAPP role with the SELECT privileges needed to perform the best practice checks. The SQL script is cvusys.sql in the CVU_home/cv/admin/ directory to facilitate the creation of this user. Use this SQL script to create the cvusys user on all the databases that you want to verify using CVU.
-rw-r--r-- 1 oracle oinstall 2841 Apr 27 2011 cvusys.sql
-rwxr-xr-x 1 oracle oinstall 1011 Apr 27 2011 cvu_config*
SQL> @cvusys.sql
DROP USER cvusys
*
ERROR at line 1:
ORA-01918: user 'CVUSYS' does not exist
DROP ROLE cvusapp
*
ERROR at line 1:
ORA-01919: role 'CVUSAPP' does not exist
Enter password for user cvusys
'Creating user cvusys...'
User created.
Grant succeeded.
Role created.
Grant succeeded.
Grant succeeded.
Grant succeeded.
Grant succeeded.
................
................
................
................
Grant succeeded.
Usage:
cluvfy comp healthcheck [-collect {cluster|database}] [-db db_unique_name] [-bestpractice|-mandatory] [-deviations] [-html] [-save [-savedir directory_path]
cluvfy comp healthcheck
Verifying OS mandatory requirements
Verifying OS Best Practice
Verifying Clusterware mandatory requirements
Verifying Clusterware Best Practice
Verifying Database "wesbdr"
Please specify password for user "cvusys" :
******************************************************************************************
Summary of environment
******************************************************************************************
System recommendations
******************************************************************************************
******************************************************************************************
Clusterware requirements
******************************************************************************************
******************************************************************************************
Clusterware recommendations
******************************************************************************************
Database recommendation checks for "wesbdr"
******************************************************************************************
-rw-r--r-- 1 oracle oinstall 2841 Apr 27 2011 cvusys.sql
-rwxr-xr-x 1 oracle oinstall 1011 Apr 27 2011 cvu_config*
SQL> @cvusys.sql
DROP USER cvusys
*
ERROR at line 1:
ORA-01918: user 'CVUSYS' does not exist
DROP ROLE cvusapp
*
ERROR at line 1:
ORA-01919: role 'CVUSAPP' does not exist
Enter password for user cvusys
'Creating user cvusys...'
User created.
Grant succeeded.
Role created.
Grant succeeded.
Grant succeeded.
Grant succeeded.
Grant succeeded.
................
................
................
................
Grant succeeded.
Usage:
cluvfy comp healthcheck [-collect {cluster|database}] [-db db_unique_name] [-bestpractice|-mandatory] [-deviations] [-html] [-save [-savedir directory_path]
cluvfy comp healthcheck
Verifying OS mandatory requirements
Verifying OS Best Practice
Verifying Clusterware mandatory requirements
Verifying Clusterware Best Practice
Verifying Database "wesbdr"
Please specify password for user "cvusys" :
******************************************************************************************
Summary of environment
******************************************************************************************
System recommendations
******************************************************************************************
******************************************************************************************
Clusterware requirements
******************************************************************************************
******************************************************************************************
Clusterware recommendations
******************************************************************************************
Database recommendation checks for "wesbdr"
******************************************************************************************
RMAN backup statistics
v$RMAN_BACKUP_JOB_DETAILS contains information about backup durations, I/O rates. It's a good starting point from which you can begin turning the backup.
col input_bytes_display format a18
col output_bytes_display format a18
col time_taken_display format a18
set linesize 132
set pagesize 50
SELECT session_recid, input_bytes_display, output_bytes_display,time_taken_display, end_time
FROM v$rman_backup_job_details
ORDER BY end_time
SESSION_RECID INPUT_BYTES_DISPLA OUTPUT_BYTES_DISPL TIME_TAKEN_DISPLAY END_TIME
------------- ------------------ ------------------ ------------------ ---------
7049 6.99G 965.00M 00:09:38 13-FEB-11
7057 6.94G 948.13M 00:08:51 14-FEB-11
7065 10.28G 1.81G 00:11:22 15-FEB-11
7073 9.96G 1.73G 00:30:26 16-FEB-11
7081 6.85G 784.77M 00:10:24 17-FEB-11
7089 6.82G 784.52M 00:08:34 18-FEB-11
7097 6.93G 814.08M 00:09:03 19-FEB-11
7105 7.00G 830.98M 00:08:50 20-FEB-11
7113 6.97G 822.40M 00:09:00 21-FEB-11
7121 6.97G 821.77M 00:10:11 22-FEB-11
7129 6.98G 822.81M 00:10:40 23-FEB-11
Based on the result above, we should investigate what happened on 16-FEB-11 that backup took longer than usual. As basic RMAN tuning, we have two recomendations for improving RMAN backup
col input_bytes_display format a18
col output_bytes_display format a18
col time_taken_display format a18
set linesize 132
set pagesize 50
SELECT session_recid, input_bytes_display, output_bytes_display,time_taken_display, end_time
FROM v$rman_backup_job_details
ORDER BY end_time
SESSION_RECID INPUT_BYTES_DISPLA OUTPUT_BYTES_DISPL TIME_TAKEN_DISPLAY END_TIME
------------- ------------------ ------------------ ------------------ ---------
7049 6.99G 965.00M 00:09:38 13-FEB-11
7057 6.94G 948.13M 00:08:51 14-FEB-11
7065 10.28G 1.81G 00:11:22 15-FEB-11
7073 9.96G 1.73G 00:30:26 16-FEB-11
7081 6.85G 784.77M 00:10:24 17-FEB-11
7089 6.82G 784.52M 00:08:34 18-FEB-11
7097 6.93G 814.08M 00:09:03 19-FEB-11
7105 7.00G 830.98M 00:08:50 20-FEB-11
7113 6.97G 822.40M 00:09:00 21-FEB-11
7121 6.97G 821.77M 00:10:11 22-FEB-11
7129 6.98G 822.81M 00:10:40 23-FEB-11
Based on the result above, we should investigate what happened on 16-FEB-11 that backup took longer than usual. As basic RMAN tuning, we have two recomendations for improving RMAN backup
- Use incremental backup with block change tracking enabled.
- Adjust multipexing of backup sets by adjusting filesperset (limit the number of datafiles in each backupset), maxopenfiles (limit the number of files that can be opened for reads simutaneously), diskratio (read datafiles from specified number of disks)
ORA-01157, ORA-01110.
You’ve mistakenly added a datafile to a file system instead of ASM and you see ORA-01157 or ORA-01110. In 10gR2, you see ORA-01157 and ORA-01110 errors and the database instance is not opened. In Oracle 11gR2 RAC, you can open all Oracle RAC instances; however the errors ORA-01157 and ORA-01110 are still reported in alert logs, so you you still need to resolve the errors by moving it from a file system to ASM.
Total System Global Area 1073741824 bytes
Fixed Size 2089576 bytes
Variable Size 230690200 bytes
Database Buffers 838860800 bytes
Redo Buffers 2101248 bytes
Database mounted.
ORA-01157: cannot identify/lock data file 139 - see DBWR trace file
ORA-01110: data file 139: '/apps/oracle/product/10.2.0/DB04/dbs/DG_DBA_DD501'
Scenarios 1: Datafile is belonged to SYSAUX tablespace. You can bring SYSAUX tablespace offline and using RMAN to move the file to ASM.
Step 1 – check for file name and file number
select file_name, file_id, tablespace_name from dba_data_files where tablespace_name = 'SYSAUX';
FILE_NAME
--------------------------------------------------------------------------------
FILE_ID TABLESPACE_NAME
---------- ------------------------------
+DG_DATA/LAX/datafile/sysaux.279.768593301
2 SYSAUX
/tmp/DG_DATA
27 SYSAUX
Step 2
SQL> alter tablespace SYSAUX offline;
Tablespace altered.
Step 3
LAX1 > rman target /
RMAN> copy datafile 27 to '+DG_DATA';
Starting backup at 22-MAR-12
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=937 instance=LAX1 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00027 name=/tmp/DG_DATA
output file name=+DG_DATA/LAX/datafile/sysaux.305.778612925 tag=TAG20120322T172203 RECID=2 STAMP=778612925
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
Finished backup at 22-MAR-12
Step 4
RMAN> switch datafile 27 to copy;
datafile 27 switched to datafile copy "+DG_DATA/LAX/datafile/sysaux.305.778612925"
Step 5
SQL> alter tablespace sysaux online;
Tablespace altered.
Senarios 2: Database is in noarchivelog mode. The datafile is a regular data / index type files..
Follow above step 1, 3, 4
Step 6
SQL > alter database datafile 27 online
Scenarios 3: Database is in Archivelog mode. The data file is a regular data / index type files...
Follow step # 1, 3, 4, 7, 8
Step 7—Recover the datafile
RMAN> run {
allocate channel d1 device type disk;
allocate channel d2 device type disk;
recover datafile 27;
}
allocated channel: d1
channel d1: sid=1522 instance=EDOCPRD1 devtype=DISK
allocated channel: d2
channel d2: sid=1582 instance=EDOCPRD1 devtype=DISK
Step 8
SQL > alter database datafile 27 online
So Many Open File Descriptors /shm/dev
This only happens in Automatic Memory Management when memory_max_target and memory_target are set.
ls -ltr /dev/shm | wc -l
79
Why 79? checking the memory_max_target = 316M. The granule size is 4MB, therefore 316MB/4MB = 79. The number of files in /dev/shm = MEMORY_TARGET / granule size
Counting the number of ASM processes
ps -ef | grep -i +ASM1 | wc -l
37
lsof /dev/shm | wc -l
2923 ==> why so many?
There are 37 background processes were created, each background process opens 79. Therefore, 37 x 79 = 2923
ls -ltr /dev/shm | wc -l
79
Why 79? checking the memory_max_target = 316M. The granule size is 4MB, therefore 316MB/4MB = 79. The number of files in /dev/shm = MEMORY_TARGET / granule size
Counting the number of ASM processes
ps -ef | grep -i +ASM1 | wc -l
37
lsof /dev/shm | wc -l
2923 ==> why so many?
There are 37 background processes were created, each background process opens 79. Therefore, 37 x 79 = 2923
TCP and UDP parameters
The default maximum Linux TCP buffer sizes are too small.Oracle recommendations for related TCP and UDP parameters are following:
net.core.rmem_max = 4194304
net.core.rmem_default=262144
net.core.rmem_max= for 11g: 4194304 For 10g: 2097152 (With RDS use at least 4194304)
net.core.wmem_default=262144
net.core.wmem_max=1048576 (With RDS use at least 2097152)
ip_local_port_range 9000 65500
net.core.rmem_max= for 11g: 4194304 For 10g: 2097152 (With RDS use at least 4194304)
net.core.wmem_default=262144
net.core.wmem_max=1048576 (With RDS use at least 2097152)
ip_local_port_range 9000 65500
Oracle Trace Analyzer --TRCANLZR or TRCA
Oracle Trace Analyzer, TRCANLZR or TRCA, is a SQL trace profiling tool for analyzing your trace files. Here are the major sections of a TRCA report that provide you a lot of useful diagnostic information than tkprof.
· Response Time Summary
· Overall Time and Totals
· Non-Recursive Time and Totals
· Recursive Time and Totals
· Top SQL
· Non-Recursive SQL
· SQL Genealogy
· Individual SQL
· Overall Segment I/O Wait Summary
· Hot I/O Blocks
· Gaps in Trace
· ORA errors in Trace
· Transactions Summary
· Non-default Initialization Params
· Trace Header
· Tool Data Dictionary
· Tool Execution Environment
· Tool Configuration Parameters
You need to download the tool from Oracle Support. Once you’ve downloaded it, the setup is very easy by executing the /trca/install/trcreate.sql.
SETUP:
cd /software/tools/trca/install
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> @tacreate.sql
Uninstalling TRCA, please wait
TADOBJ completed.
SQL>
SQL> WHENEVER SQLERROR EXIT SQL.SQLCODE;
SQL> REM If this DROP USER command fails that means a session is connected with this user.
SQL> DROP USER trcanlzr CASCADE;
SQL> WHENEVER SQLERROR CONTINUE;
SQL>
SQL> SET ECHO OFF;
TADUSR completed.
TADROP completed.
Specify optional Connect Identifier (as per Oracle Net)
Include "@" symbol, ie. @PROD
If not applicable, enter nothing and hit the "Enter" key
Optional Connect Identifier (ie: @PROD): @LABPOC1
Define the TRCANLZR user password (hidden and case sensitive).
Specify TRCANLZR password:
Re-enter password:
Set up TRCANLZR temporary and default tablespaces
Below are the list of online tablespaces in this database.
Decide which tablespace you wish to create the TRCANLZR tables
and indexes. This will also be the TRCANLZR user default tablespace.
Specifying the SYSTEM tablespace will result in the installation
FAILING, as using SYSTEM for tools data is not supported.
Wait...
TABLESPACE_NAME FREE_SPACE_MB
------------------------------ -------------
TOOLS 55
TOOLS_OLD 98
LAB_I 830
USERS 1130
LAB_D 65632
Above is the list of online tablespaces in this database.
Decide which tablespace you wish to create the TRCANLZR tables
and indexes. This will also be the TRCANLZR user default tablespace.
Specifying the SYSTEM tablespace will result in the installation
FAILING, as using SYSTEM for tools data is not supported.
Tablespace name is case sensitive.
Default tablespace [UNKNOWN]: TOOLS_OLD
DEFAULT_TABLESPACE
------------------------------
TOOLS_OLD
Choose the TRCANLZR user temporary tablespace.
Specifying the SYSTEM tablespace will result in the installation
FAILING, as using SYSTEM for the temporary tablespace is not recommended.
Wait...
TABLESPACE_NAME
------------------------------
TEMP
Tablespace name is case sensitive.
Temporary tablespace [UNKNOWN]: TEMP
TEMPORARY_TABLESPACE
------------------------------
TEMP
Type of TRCA repository
Create TRCA repository as Temporary or Permanent objects?
Enter T for Temporary or P for Permanent.
T is recommended and default value.
Type of TRCA repository [T]:
TACUSR completed.
No errors.
SQLT$STAGE: write test file tasqdirset.txt
sqlt$_trca$_dir_set.open_write_close: ORA-29283: invalid file
operation
ORA-06512: at "SYS.UTL_FILE", line 536
ORA-29283: invalid file
operation
SQLT$STAGE: created
SQLT$STAGE: READ,WRITE access granted to SQLTXPLAIN
SQLT$STAGE: READ,WRITE access granted to TRCANLZR
SQLT$STAGE: write test file tasqdirset.txt
SQLT$STAGE: read test file tasqdirset.txt
…..
TRCA$INPUT2: read test file tasqdirset.txt
TRCA$INPUT2: get attributes for file tasqdirset.txt
TRCA$INPUT2: /apps/oracle/diag/rdbms/labpoc/LABPOC1/trace
Connected.
TAUTLTEST completed.
no rows selected
TACOBJ completed.
tool_owner: "TRCANLZR"
role_name: "PUBLIC"
Creating Package Specs TRCA$G
No errors.
Creating Grants on Packages
Tool Version
----------------
11.4.4.2
Install Date
----------------
20120320
Directories
--------------------------------------------------------------------------------------------------------------------------------
TRCA$INPUT1(VALID) /apps/oracle/diag/rdbms/labpoc/LABPOC1/trace
TRCA$INPUT2(VALID) /apps/oracle/diag/rdbms/labpoc/LABPOC1/trace
TRCA$STAGE(VALID) /apps/oracle/diag/rdbms/labpoc/LABPOC1/trace
user_dump_dest /apps/oracle/diag/rdbms/labpoc/LABPOC1/trace
background_dump_dest /apps/oracle/diag/rdbms/labpoc/LABPOC1/trace
Libraries
--------------------------------------------------------------------------------------------------------------------------------
VALID PACKAGE TRCA$I /* $Header: 224270.1 tacpkgi.pks 11.4.4.1 2012/01/02 csierra $ */
VALID PACKAGE TRCA$E /* $Header: 224270.1 tacpkge.pks 11.4.2.7 2011/04/08 csierra $ */
VALID PACKAGE TRCA$G /* $Header: 224270.1 tacpkgg.pks 11.4.2.7 2011/04/08 csierra $ */
VALID PACKAGE TRCA$P /* $Header: 224270.1 tacpkgp.pks 11.4.4.1 2012/01/02 csierra $ */
VALID PACKAGE TRCA$R /* $Header: 224270.1 tacpkgr.pks 11.4.2.7 2011/04/08 csierra $ */
VALID PACKAGE TRCA$T /* $Header: 224270.1 tacpkgt.pks 11.4.3.1 2011/06/17 csierra $ */
VALID PACKAGE TRCA$X /* $Header: 224270.1 tacpkgx.pks 11.4.2.7 2011/04/08 csierra $ */
VALID PACKAGE BODY TRCA$I /* $Header: 224270.1 tacpkgi.pkb 11.4.4.1 2012/01/02 csierra $ */
VALID PACKAGE BODY TRCA$E /* $Header: 224270.1 tacpkge.pkb 11.4.2.7 2011/04/08 csierra $ */
VALID PACKAGE BODY TRCA$G /* $Header: 224270.1 tacpkgg.pkb 11.4.4.2 2012/02/02 csierra $ */
VALID PACKAGE BODY TRCA$P /* $Header: 224270.1 tacpkgp.pkb 11.4.4.1 2012/01/02 csierra $ */
VALID PACKAGE BODY TRCA$R /* $Header: 224270.1 tacpkgr.pkb 11.4.4.1 2012/01/02 csierra $ */
VALID PACKAGE BODY TRCA$T /* $Header: 224270.1 tacpkgt.pkb 11.4.4.1 2012/01/02 csierra $ */
VALID PACKAGE BODY TRCA$X /* $Header: 224270.1 tacpkgx.pkb 11.4.2.7 2011/04/08 csierra $ */
TACPKG completed.
Taking a snapshot of some Data Dictionary objects, please wait...
17:48:23 => refresh_trca$_dict_from_this
17:48:23 -> purge_trca$_dict
17:48:23 dict_state_before_purge
17:48:23 -----------------------
17:48:23 -> print_dict_state
17:48:23 dict_refresh_days :
17:48:23 dict_refresh_date :
17:48:23 dict_database_id :
…..
17:48:45 <- print_dict_state
17:48:45 <= refresh_trca$_dict_from_this
PL/SQL procedure successfully completed.
Snapshot of some Data Dictionary objects completed.
TAUTLTEST completed.
TACREATE completed. Installation completed successfully.
SQL>
TRACING:
alter session set events ‘10046 trace name context forever, level 12’;
alter system set tracefile_identifier=’mytrace’;
<Your SQL statement here…>
alter session set events ‘10046 trace name context off’;
SQL> START trcanlzr.sql /apps/oracle/diag/rdbms/labpoc/LABPOC1/trace/LABPOC1_ora_23740_mytrace.trc
Parameter 1:
Trace Filename or control_file.txt (required)
Value passed to trcanlzr.sql:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
TRACE_FILENAME: /apps/oracle/diag/rdbms/labpoc/LABPOC1/trace/LABPOC1_ora_23740_mytrace.trc
Analyzing /apps/oracle/diag/rdbms/labpoc/LABPOC1/trace/LABPOC1_ora_23740_mytrace.trc
To monitor progress, login as TRCANLZR into another session and execute:
SQL> SELECT * FROM trca$_log_v;
... analyzing trace(s) ...
Trace Analyzer completed.
Review first trcanlzr_error.log file for possible fatal errors.
Review next trca_e43354.log for parsing messages and totals.
Copying now generated files into local directory
TKPROF: Release 11.2.0.3.0 - Development on Tue Mar 20 17:59:31 2012
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
adding: trca_e43354.html (deflated 89%)
adding: trca_e43354.log (deflated 84%)
adding: trca_e43354.tkprof (deflated 82%)
adding: trca_e43354.txt (deflated 84%)
adding: trcanlzr_error.log (deflated 81%)
test of trca_e43354.zip OK
deleting: trcanlzr_error.log
Archive: trca_e43354.zip
Length Date Time Name
-------- ---- ---- ----
107234 03-20-12 17:59 trca_e43354.html
15891 03-20-12 17:59 trca_e43354.log
8367 03-20-12 17:59 trca_e43354.tkprof
52391 03-20-12 17:59 trca_e43354.txt
-------- -------
183883 4 files
File trca_e43354.zip has been created
TRCANLZR completed.
Subscribe to:
Posts (Atom)