Oracle Database: Oracle Enterprise Linux : Page Allocation Failure : Performance/Kernel Tuning / - System Unresponsive

Oracle Database: Oracle Enterprise Linux : Page Allocation Failure : Performance/Kernel Tuning / - System Unresponsive

PT Vineet Sachdeva   July 26, 2018

We recently had issue with one of our physical servers. 
It will become unresponsive and hang in a lot of situations. 

Oracle Linux Server release 7.3
NAME="Oracle Linux Server"

VERSION="7.3"

Linux pwbdd-fe-1 4.1.12-61.1.18.el7uek.x86_64 #
256GB of RAM and 16 CPU (2socket * 8 cores)

Looked into sar logs and found out the memory usage was around 99.7%.

We found out Page allocations error  in the /var/log/messages

kworker/u66:0: page allocation failure: order:1, mode:0x204020

CPU: 8 PID: 1579 Comm: kworker/u66:0 Tainted: G           OE   4.1.12-61.1.18.el7uek.x86_64 #2

There was no entry in Oracle Database log or any trace file, so database had to do minimal with it. 

We finally fixed it with below kernel settings.

•  vm.min_free_kbytes = 1048576 
•  vm.swappiness = 60
•  vm.dirty_ratio = 20

I have to admit that I took help of an ODA with Linux 7 and similar amount of RAM as well


Referred Links and 
https://utcc.utoronto.ca/~cks/space/blog/linux/DecodingPageAllocFailures
http://tech.donghao.org/2016/03/25/avoid-page-allocation-failure-for-linux-kernel/
https://unix.stackexchange.com/questions/128539/what-would-happen-if-the-amount-of-free-memory-vm-min-free-kbytes-was-too-low
https://stackoverflow.com/questions/21374491/vm-min-free-kbytes-why-keep-minimum-reserved-memory

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-memory-tunables
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/performance_tuning_guide/sect-red_hat_enterprise_linux-performance_tuning_guide-configuration_tools-configuring_system_memory_capacity

Oracle Support Docs

ODA: Node Inaccessible Due to OS "page allocation failure: order:1" (Doc ID 1642258.1)

[Linux OS] System Hung with Large Numbers of Page Allocation Failures with "order:5" on Exadata Environments (Doc ID 1546861.1)
Linux: Out-of-Memory (OOM) Killer (Doc ID 452000.1)
Top 5 Issues That Cause Node Reboots or Evictions or Unexpected Recycle of CRS (Doc ID 1367153.1)
RAC and Oracle Clusterware Best Practices and Starter Kit (Linux) (Doc ID 811306.1)
Linux Kernel: The SLAB Allocator (Doc ID 434351.1)
[Linux OS] System Hung with 'page allocation failure' (Doc ID 2059297.1)

Oracle Database - 18c: On Premise Installation - Oracle Linux - Part 2

Oracle Database - 18c: On Premise Installation - Oracle Linux - Part 2

18c Vineet Sachdeva   July 24, 2018

The second part of the blog discusses on how to install 18c once all the pre-req are complete. 


Download the required software from 
http://www.oracle.com/technetwork/database/enterprise-edition/downloads/index.html

Make sure to accept the license agreement before downloading. 

Once you have the software then Create the directories

[As root]
mkdir /u01

chown oracle:oinstall /u01

[Connect as user - oracle]
mkdir -p /u01/app/oracle/product/183/dbhome_1

Copy the Downloaded file to /u01/app/oracle/product/183/dbhome_1

unzip -qq LINUX.X64_180000_db_home.zip

Working Directory = /u01/app/oracle/product/183/dbhome_1
Run - runInstaller and follow the steps as in the Screenshots

Choosing of Oracle Home is not present as it is a in place installation and linking of binaries

If you have created only one group as part of your install then dba will be selected for all 

OraInventory location if required can be edited here

At this point you are prompted to run root scripts, run root scripts and press okay. 
Root scripts as the name says are run as root user.

[root@orclbox18c ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@orclbox18c ~]# /u01/app/oracle/product/183/dbhome_1/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/183/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Do you want to setup Oracle Trace File Analyzer (TFA) now ? yes|[no] :
yes
Installing Oracle Trace File Analyzer (TFA).
Log File: /u01/app/oracle/product/183/dbhome_1/install/root_orclbox18c.novalocal_2018-07-24_01-33-35-863801359.log

Finished installing Oracle Trace File Analyzer (TFA)

Press OK once the root script are run 

This completes the Database Installation as well. 

Now if you check the installation then the DB is at 18.3

OPatch/opatch lspatches
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)
28090553;OCW RELEASE UPDATE 18.3.0.0.0 (28090553)
28090523;Database Release Update : 18.3.0.0.180717 (28090523)

Oracle Database - 18c: On Premise Installation - Oracle Linux - Part 1

Oracle Database - 18c: On Premise Installation - Oracle Linux - Part 1

18c Vineet Sachdeva   July 24, 2018

In this blog I discuss installation of Oracle 18c on premise. 

The part 1 covers the pre-requisites installation and part 2 covers the installation. 

Pre-requistites

The easy and fast way of covering the preq-reqs on Oracle Linux is to use Oracle validated rpm for 18c Linux. 

On a fresh installation of Oracle Linux (Linux orclbox18c.novalocal 4.1.12-112.14.15.el7uek.x86_64 GNU/Linux)


[root@orclbox18c ~]# yum install oracle-database-preinstall-18c.x86_64 -y

You can find the changes it did from /var/log/oracle-database-preinstall-18c/results
and backup of files from /var/log/oracle-database-preinstall-18c/backup

Swap Creation for both Manual and automated pre-req completion - http://o-dba.blogspot.com/2018/05/oracle-database-12c18c-swap-swapon.html

Manual Pre-reqs Install

1. Install required RPMs as per your Operating System and Kernel version & support. https://docs.oracle.com/en/database/oracle/oracle-database/18/ladbi/database-installation-guide-linux.pdf
2.  User Creation
1. groupadd -g 54321 oinstall
2. groupadd -g 54322 dba
3. useradd -g 54321 -G dba -u 54321 oracle
3. Create file - /etc/sysctl.d/99-oracle-database-preinstall-18c-sysctl.conf
2. For shmmax - 90% of memory in KB
3. For shmmall - shmmax/4096
4. For vm.nr_hugepages = 80% of memory in MB / 2
5. 
   
6. fs.file-max = 6815744
7. kernel.sem = 250 32000 100 128
8. kernel.shmmni = 4096
9. kernel.shmall = 1073741824
10. kernel.shmmax = 4398046511104
11. kernel.panic_on_oops = 1
12. net.core.rmem_default = 262144
13. net.core.rmem_max = 4194304
14. net.core.wmem_default = 262144
15. net.core.wmem_max = 1048576
16. net.ipv4.conf.all.rp_filter = 2
17. net.ipv4.conf.default.rp_filter = 2
18. fs.aio-max-nr = 1048576
19. net.ipv4.ip_local_port_range = 9000 65500
20. vm.nr_hugepages = 13100
5. Create file - /etc/security/limits.d/oracle-database-preinstall-18c.conf
2. oracle   soft   nofile    1024
3. oracle   hard   nofile    65536
4. oracle   soft   nproc    16384
5. oracle   hard   nproc    16384
6. oracle   soft   stack    10240
7. oracle   hard   stack    32768
8. ## oracle-database-preinstall-18c setting for memlock soft/hard limit is maximum of 128GB on x86_64 or 3GB on x86 OR 90% of RAM
9. oracle   hard   memlock    134217728
10. oracle   soft   memlock    134217728
6. Disable Transparent Huge Pages
7. Add NOZEROCONF parameter in network configuration in file /etc/sysconfig/network
8. Disabling Transparent Hugepages - Refer Oracle Note:1557478.1
9. Disabling defrag - Refer Oracle Note:1557478.1

Oracle Database : Standard Edition vs Standard Edition one (SE vs SE1)

Oracle Database : Standard Edition vs Standard Edition one (SE vs SE1)

RAC Vineet Sachdeva   July 20, 2018

We had an environment where Standard Edition one was required and engineers installed by mistake Standard Edition.

This caused us to stress due to licensing issue from Oracle. 


However, to be free you should know that Standard Edition and Standard Edition One are the same install (as given by Oracle), it's just 2 different options given to you during installation, though the installation is same. 

These 2 just differ in only feature (and licensing), you can have SE running on  RAC however, SE1 is not supported for RAC.
If you do, then you pay :)

As per the Oracle document -

How to Install "Standard Edition One" (Doc ID 352618.1)

which confirms for  Oracle Database - Standard Edition - Version 10.1.0.5 and later - 

“Standard Edition One is a licensing edition. It is same as Standard Edition for a machine that has 1 or 2 processor sockets. No packaging differences. It was called Standard Edition One as it was originally Standard Edition for a 1 processor socket machine but with the release of 10g, we expanded it to a 1 or 2 processor socket machine.

So basically, if you install the Standard Edition, you will have the correct software.


The only difference between "Standard Edition One" and "Standard Edition" is that Standard Edition One must NOT be used on RAC, since there will be a licensing issue. It means that to install Standard Edition One, you have to install Standard Edition software and it can NOT / must NOT be used in a clustered environment (RAC).”

Oracle Database - ORA-07217: sltln: environment variable cannot be evaluated. (After Windows to Linux Restore for Archive Logs)

Oracle Database - ORA-07217: sltln: environment variable cannot be evaluated. (After Windows to Linux Restore for Archive Logs)

Admin Vineet Sachdeva   July 18, 2018

We restored one of our databases from Windows to Linux, everything went fine since cross-platform is supported on our database version. 

However, when doing anything with archivelogs (crosscheck archivelog all) we faced issue 

ORA-07217: sltln: environment variable cannot be evaluated

Did quite a lot of research but there was no specific Document in Oracle which had the symptoms and solution. 

Then traced the rman for crosscheck command. 

export NLS_LANG=american

export NLS_DATE_FORMAT='Mon DD YYYY HH24:MI:SS'
rman target / trace="/tmp/tracefile.out" debug

Recovery Manager: Release 11.2.0.4.0 - Production on Wed Jul 18 08:48:41 2018

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

crosscheck archivelog all;
RMAN-06005: connected to target database: <DBNAME> (DBID=336051125)

RMAN>
RMAN-06009: using target database control file instead of recovery catalog
RMAN-08030: allocated channel: ORA_DISK_1
RMAN-08500: channel ORA_DISK_1: SID=47 device type=DISK
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of crosscheck command on ORA_DISK_1 channel at 07/18/2018 08:48:48

ORA-07217: sltln: environment variable cannot be evaluated.

Browsing through the trace file found issues (highlighted bold)

Excerpt from the trace file - 

DBGPLSQL:      channel ORA_DISK_1: started, command:556 [08:48:48.713] (change)
DBGPLSQL:      channel ORA_DISK_1: File \\PATH1\PATH2$\PATH3\\ARCH_D-MYDB_ID-336051125_S-4625
8_T-1_A-895598920_BBT74KKT belongs to 0 [08:48:48.713] (change)
DBGPLSQL:      channel ORA_DISK_1:  processing (file/handle=\\PATH1\PATH2$\PATH3\\ARCH_D-MYDB
_ID-336051125_S-46258_T-1_A-895598920_BBT74KKT,recid=93051, old_status=A, hdl_isdisk=1, devicetype=DISK, source_dbid=0)
 [08:48:48.713] (change)
DBGPLSQL:      channel ORA_DISK_1:  force: 0 [08:48:48.714] (change)
DBGPLSQL:      channel ORA_DISK_1: Calling ValidateArchivedLog for \\PATH1\PATH2$\PATH3\\ARCH_D
-MYDB_ID-336051125_S-46258_T-1_A-895598920_BBT74KKT [08:48:48.714] (change)
DBGRPC:        krmxrpc - channel ORA_DISK_1 kpurpc2 err=7217 db=target proc=SYS.DBMS_BACKUP_RESTORE.VALIDATEARCHIVEDLOG
 excl: 111
DBGRPC:        krmxrpc - caloing krmxtrim: with message of length 111: @@@ORA-07217: sltln: environment variable cannot be evaluated.
DBGRPC:        ORA-06512: at "SYS.DBMS_BACKUP_RESTORE", line 4075
DBGRPC:        @@@
DBGRPC:        krmxrpc - channel ORA_DISK_1, error 7217 received for rpc 6 before receiving 3123
DBGRPC:        krmxr - channel ORA_DISK_1 returned from peicnt
DBGMISC:       ENTERED krmstrim [08:48:48.714]
DBGMISC:        Trimming message: ORA-07217: sltln: environment variable cannot be evaluated. [08:48:48.714] (krmstrim)
DBGMISC:        ORA-06512: at "SYS.DBMS_BACKUP_RESTORE", line 4075 (krmstrim

We found out a $ was present in the path (this is because of entries of archive log in the control file which were windows path).
It was impossible to expired / delete the archives. 
So we cleared the archivelog entry section from the controlfile and recataloged the archivelogs 
Which then was the solution for us. 

From sqlplus plus prompt ran - 

 exec SYS.DBMS_BACKUP_RESTORE.resetCfileSection(11);
This clears the 

Then cataloged the archivelogs again which were present. 

Removing entries in v$archived_log referencing a particluar DEST_ID (Doc ID 845361.1)

Also you can find details on deleting other controlfile section in one of the blogs from Vijay

Copied from the above blog - 

DBMS_BACKUP_RESTORE.resetCfileSection() package:


-- Note that you can execute the following PL/SQL to remove all entries from V$RMAN_STATUS:
-- The preceding function removes all job-related entries. No rows will be visible until new backup jobs are shown in V$RMAN_BACKUP_JOB_DETAILS.

sqlplus / as sysdba
exec SYS.DBMS_BACKUP_RESTORE.resetCfileSection(28);
commit;


-- Removing entries in v$DATABASE_BLOCK_CORRUPTION view
-- Be aware that the corrupted blocks will remain corrupted until we fix the block corruption issue.
-- Error: Bug 13386807 : RMAN BACKUP FAILED ORA-03114 ORA-07445: [KRBINSERTBCR()+665]

exec SYS.DBMS_BACKUP_RESTORE.resetCfileSection(35);


-- Issue: V$DATABASE_BLOCK_CORRUPTION has rows referring to a non-existing datafile.

-- Removing entries in v$backup_corruption view
exec SYS.DBMS_BACKUP_RESTORE.resetCfileSection(17);
-- Removing entries in v$copy_corruption view
exec SYS.DBMS_BACKUP_RESTORE.resetCfileSection(18);


-- Removing entries in v$deleted_object view on RMAN catalog
-- Issue: RMAN backup is suddenly running very slowly after having deleted a large number of backupsets from catalog.

exec SYS.DBMS_BACKUP_RESTORE.resetCfileSection(19);


-- Removing entries in v$archived_log view
-- Issue: Removing entries in v$archived_log referencing a particluar DEST_ID

exec SYS.DBMS_BACKUP_RESTORE.resetCfileSection(11);


-- Removing entries in v$backup_spfile on RMAN catalog
-- RMAN Error: ORA-01400: cannot insert NULL into ("RMAN"."BSF"."MODIFICATION_TIME")


exec SYS.DBMS_BACKUP_RESTORE.resetCfileSection(21); 
-------------------------------------------------
Just to let you know my archivelogs entries looked like this on Linux

93837   1    46651   A 12-JUL-18
        Name: \\PATH1\PATH2$\PATH3\<DBNAME>\ARCH_D-MYDB_ID-336051125_S-46651_T-1_A-895598920_OCT7PIAB

93839   1    46652   A 12-JUL-18
        Name: \\PATH1\PATH2$\PATH3\<DBNAME>\ARCH_D-MYDB_ID-336051125_S-46652_T-1_A-895598920_ODT7PK2J

93841   1    46653   A 12-JUL-18
        Name: \\PATH1\PATH2$\PATH3\<DBNAME>\ARCH_D-MYDB_ID-336051125_S-46653_T-1_A-895598920_OET7PLQS

93843   1    46654   A 12-JUL-18

        Name: \\PATH1\PATH2$\PATH3\<DBNAME>\ARCH_D-MYDB_ID-336051125_S-46654_T-1_A-895598920_OFT7PNJ3

OEM 13cR3 - De-Installation - 13.3.0.0

OEM 13cR3 - De-Installation - 13.3.0.0

OEM Vineet Sachdeva   July 11, 2018

In this blog I cover how to de-install OEM 13cR3 installation. 

Login using oracle account or OMS owner account.
The proces is to copy  the uninstall utility to tmp directory and run using perl. 
You will be asked for credentials of Database sys user, sysman and Admin (weblogic) so keep them handy
 
export ORACLE_HOME=/u01/app/oracle/em13cr3/middleware 
cd /tmp
cp $ORACLE_HOME/sysman/install/EMDeinstall.pl /tmp
$ORACLE_HOME/perl/bin/perl /tmp/EMDeinstall.pl -mwHome $ORACLE_HOME -stageLoc /tmp
 
Refer to /tmp/deinstall_2018-07-11_02-29-42.log for deinstall log
Substring valuye is /u01/app/oracle/em13cr3/gc_inst
This is a First OMS install. So, this deinstalls the OMS , Repository and Agent. Confirm (y/n)y
User confirmed for deinstallation.

Enter the SYS Password :
Enter the sysman Password :
Enter the Admin Server password 
.
.
.

Deleting the em home
The deinstallation of OMS is successful.

The location of the file is : /etc/oragchomelist



Finally if the database is not required, it can be dropped

OEM 13cR3 - Post OMS Installation - 13.3.0.0

OEM 13cR3 - Post OMS Installation - 13.3.0.0

OEM Vineet Sachdeva   July 11, 2018

If you have followed my install series, then you have successfully installed OEM 13cR3

Installation details can be found on server path - $ORACLE_HOME/install/setupinfo.txt

If you have not changed the default port during installation then <Hostname>:7803/em is the URL to access. 

IN my case https://oem13cr3:7803/em

1. Login using sysman credentials provided during installation. 

 2. Click Save and continue after selecting Accessibility Preference

3. Accept the license agreement

4. Select the Home Page - I generally prefer the Summary Page set as my home page (this can be changed anytime later)

5. You should see summary screen and your installation is complete

 

OEM 13cR3 - OEM Installation (OMS Setup) - 13.3.0.0

OEM 13cR3 - OEM Installation (OMS Setup) - 13.3.0.0

OEM Vineet Sachdeva   July 10, 2018

In this blog I am going to discuss on installation of OEM 13cR3

You must do below changes before you can continue

On the Database
alter system set "_allow_insert_with_update_check" = TRUE ;
alter system set session_cached_cursors=500 scope=spfile;
alter system set shared_pool_size=600M;



Bounce the DB
Ensure Redo log size - 300M or more is configured



On OS Side you must configure in .bash_profile and .bashrc
export EM_THREAD_STACK_SIZE=8192

Start the Installation
chmod u+x em13300_linux64.bin





Note - Log file sare present in the scratch directory which is /u01 and oraInstall with datetime example  - /u01/OraInstall2018-07-10_04-39-01AM/
Repo Creation logs can be found in - /u01/app/oracle/em13cr3/middleware/sysman/log/schemamanager. 
   

[oracle@oem13cr3 install]$ ./em13300_linux64.bin
** The directory "/tmp" does not have enough space. At least 12289 MB of free space are required.
Please input another directory or [Exit]: /u01
0%...........................................................................100%
Launcher log file is /u01/OraInstall2018-07-10_04-21-17AM/launcher2018-07-10_04-21-17AM.log.
Starting Oracle Universal Installer

Checking if CPU speed is above 300 MHz.   Actual 2195.135 MHz    Passed
Checking monitor: must be configured to display at least 256 colors.   Actual 16777216    Passed
Checking swap space: must be greater than 512 MB.   Actual 16383 MB    Passed
Checking if this platform requires a 64-bit JVM.   Actual 64    Passed (64-bit not required)
 

 

Above Step can take almost 2 hours (or more on small end systems)

(Step repository configuration takes the maximum time)

 Run allroot.sh
[root@oem13cr3 ~]# /u01/app/oracle/em13cr3/middleware/allroot.sh

Starting to execute allroot.sh .........

Starting to execute /u01/app/oracle/em13cr3/middleware/root.sh ......
/etc exist

Creating /etc/oragchomelist file...
/u01/app/oracle/em13cr3/middleware
Finished product-specific root actions.
/etc exist
Finished execution of  /u01/app/oracle/em13cr3/middleware/root.sh ......


Starting to execute /u01/app/oracle/agent/agent13cr3/agent_13.3.0.0.0/root.sh ......
Finished product-specific root actions.
/etc exist
Finished execution of  /u01/app/oracle/agent/agent13cr3/agent_13.3.0.0.0/root.sh ...... 

Installation Complete 

Onscreen Log for reference
 [oracle@oem13cr3 install]$ ./em13300_linux64.bin
** The directory "/tmp" does not have enough space. At least 12289 MB of free space are required.
Please input another directory or [Exit]: /u01
0%...........................................................................100%
Launcher log file is /u01/OraInstall2018-07-10_04-39-01AM/launcher2018-07-10_04-39-01AM.log.
Starting Oracle Universal Installer

Checking if CPU speed is above 300 MHz.   Actual 2195.135 MHz    Passed
Checking monitor: must be configured to display at least 256 colors.   Actual 16777216    Passed
Checking swap space: must be greater than 512 MB.   Actual 16383 MB    Passed
Checking if this platform requires a 64-bit JVM.   Actual 64    Passed (64-bit not required)


Preparing to launch the Oracle Universal Installer from /u01/OraInstall2018-07-10_04-39-01AM
====Prereq Config Location main===
/u01/OraInstall2018-07-10_04-39-01AM/stage/prereq
EMGCInstaller args -scratchPath
EMGCInstaller args /u01/OraInstall2018-07-10_04-39-01AM
EMGCInstaller args -sourceType
EMGCInstaller args network
EMGCInstaller args -timestamp
EMGCInstaller args 2018-07-10_04-39-01AM
EMGCInstaller args -paramFile
EMGCInstaller args /u01/sfx_isiXlD/Disk1/install/linux64/oraparam.ini
EMGCInstaller args -nocleanUpOnExit
DiskLoc inside SourceLoc/tmp/install
EMFileLoc:/u01/OraInstall2018-07-10_04-39-01AM/oui/em/
ScratchPathValue :/u01/OraInstall2018-07-10_04-39-01AM
ConfigSharedLocation : shareLocationForBIP.isSelectedtrue
ConfigSharedLocation : bipConfigText
ConfigSharedLocation :bipClusterText
hostname instantiation HOSTNAME=oem13cr3.novalocal
hostname instantiation ADMIN_SERVER_HOSTNAME=oem13cr3.novalocal
hostname instantiation MANAGED_SERVER_HOSTNAME=oem13cr3.novalocal
hostname instantiation s_hostname=oem13cr3.novalocal
Agent OracleHome :/u01/app/oracle/agent/agent13cr3/agent_13.3.0.0.0

..................................................................................................
Installation in progress (Tuesday, July 10, 2018 4:48:52 AM EDT)
                              98% Done.
Install successful

Linking in progress (Tuesday, July 10, 2018 4:48:52 AM EDT)
Link successful

Setup in progress (Tuesday, July 10, 2018 4:48:52 AM EDT)
Setup successful

Saving inventory (Tuesday, July 10, 2018 4:48:52 AM EDT)
Saving inventory complete

End of install phases.(Tuesday, July 10, 2018 4:48:54 AM EDT)

...............................................................  15% Done.
...............................................................  30% Done.
...............................................................  46% Done.
...............................................................  61% Done.
...............................................................  77% Done.
...............................................................  92% Done.
........................
Installation in progress (Tuesday, July 10, 2018 4:49:22 AM EDT)
                                         98% Done.
Install successful

Linking in progress (Tuesday, July 10, 2018 4:49:22 AM EDT)
Link successful

Setup in progress (Tuesday, July 10, 2018 4:49:23 AM EDT)
Setup successful

Saving inventory (Tuesday, July 10, 2018 4:49:23 AM EDT)
Saving inventory complete

End of install phases.(Tuesday, July 10, 2018 4:49:26 AM EDT)

..................................................................................................
Installation in progress (Tuesday, July 10, 2018 4:49:38 AM EDT)
                              98% Done.
Install successful

Linking in progress (Tuesday, July 10, 2018 4:49:39 AM EDT)
Link successful

Setup in progress (Tuesday, July 10, 2018 4:49:39 AM EDT)
Setup successful

Saving inventory (Tuesday, July 10, 2018 4:49:39 AM EDT)
Saving inventory complete

End of install phases.(Tuesday, July 10, 2018 4:49:41 AM EDT)

...............................................................  17% Done.
...............................................................  34% Done.
...............................................................  51% Done.
...............................................................  68% Done.
...............................................................  85% Done.
................................................
Installation in progress (Tuesday, July 10, 2018 4:50:35 AM EDT)
                 98% Done.
Install successful

Linking in progress (Tuesday, July 10, 2018 4:50:35 AM EDT)
Link successful

Setup in progress (Tuesday, July 10, 2018 4:50:35 AM EDT)
Setup successful

Saving inventory (Tuesday, July 10, 2018 4:50:35 AM EDT)
Saving inventory complete

End of install phases.(Tuesday, July 10, 2018 4:50:38 AM EDT)

..................................................................................................
Installation in progress (Tuesday, July 10, 2018 4:50:55 AM EDT)
                              98% Done.
Install successful

Linking in progress (Tuesday, July 10, 2018 4:50:55 AM EDT)
Link successful

Setup in progress (Tuesday, July 10, 2018 4:50:55 AM EDT)
Setup successful

Saving inventory (Tuesday, July 10, 2018 4:50:55 AM EDT)
Saving inventory complete

End of install phases.(Tuesday, July 10, 2018 4:50:58 AM EDT)

..................................................................................................
Installation in progress (Tuesday, July 10, 2018 4:51:02 AM EDT)
                              98% Done.
Install successful

Linking in progress (Tuesday, July 10, 2018 4:51:02 AM EDT)
Link successful

Setup in progress (Tuesday, July 10, 2018 4:51:03 AM EDT)
Setup successful

Saving inventory (Tuesday, July 10, 2018 4:51:03 AM EDT)
Saving inventory complete

End of install phases.(Tuesday, July 10, 2018 4:51:06 AM EDT)

.....................................................................................
Installation in progress (Tuesday, July 10, 2018 4:51:09 AM EDT)
                                           85% Done.
Install successful

Linking in progress (Tuesday, July 10, 2018 4:51:09 AM EDT)
Link successful

Setup in progress (Tuesday, July 10, 2018 4:51:09 AM EDT)
Setup successful

Saving inventory (Tuesday, July 10, 2018 4:51:09 AM EDT)
Saving inventory complete

End of install phases.(Tuesday, July 10, 2018 4:51:13 AM EDT)

..................................................................................................
Installation in progress (Tuesday, July 10, 2018 4:51:16 AM EDT)
                              98% Done.
Install successful

Linking in progress (Tuesday, July 10, 2018 4:51:16 AM EDT)
Link successful

Setup in progress (Tuesday, July 10, 2018 4:51:16 AM EDT)
Setup successful

Saving inventory (Tuesday, July 10, 2018 4:51:16 AM EDT)
Saving inventory complete

End of install phases.(Tuesday, July 10, 2018 4:51:18 AM EDT)

................................................................................................
Installation in progress (Tuesday, July 10, 2018 4:51:19 AM EDT)
                                96% Done.
Install successful

Linking in progress (Tuesday, July 10, 2018 4:51:19 AM EDT)
Link successful

Setup in progress (Tuesday, July 10, 2018 4:51:19 AM EDT)
Setup successful

Saving inventory (Tuesday, July 10, 2018 4:51:19 AM EDT)
Saving inventory complete

End of install phases.(Tuesday, July 10, 2018 4:51:21 AM EDT)

........................................................................
Installation in progress (Tuesday, July 10, 2018 4:51:22 AM EDT)
                                                        72% Done.
Install successful

Linking in progress (Tuesday, July 10, 2018 4:51:22 AM EDT)
Link successful

Setup in progress (Tuesday, July 10, 2018 4:51:22 AM EDT)
Setup successful

Saving inventory (Tuesday, July 10, 2018 4:51:22 AM EDT)
Saving inventory complete

End of install phases.(Tuesday, July 10, 2018 4:51:25 AM EDT)

............................................................................
Installation in progress (Tuesday, July 10, 2018 4:51:27 AM EDT)
                                                    76% Done.
Install successful

Linking in progress (Tuesday, July 10, 2018 4:51:27 AM EDT)
Link successful

Setup in progress (Tuesday, July 10, 2018 4:51:27 AM EDT)
Setup successful

Saving inventory (Tuesday, July 10, 2018 4:51:27 AM EDT)
Saving inventory complete

End of install phases.(Tuesday, July 10, 2018 4:51:29 AM EDT)

..................................................................................................
Installation in progress (Tuesday, July 10, 2018 4:51:30 AM EDT)
                              98% Done.
Install successful

Linking in progress (Tuesday, July 10, 2018 4:51:30 AM EDT)
Link successful

Setup in progress (Tuesday, July 10, 2018 4:51:30 AM EDT)
Setup successful

Saving inventory (Tuesday, July 10, 2018 4:51:30 AM EDT)
Saving inventory complete

End of install phases.(Tuesday, July 10, 2018 4:51:32 AM EDT)

..............................................................................
Installation in progress (Tuesday, July 10, 2018 4:51:34 AM EDT)
                                                  78% Done.
Install successful

Linking in progress (Tuesday, July 10, 2018 4:51:34 AM EDT)
Link successful

Setup in progress (Tuesday, July 10, 2018 4:51:34 AM EDT)
Setup successful

Saving inventory (Tuesday, July 10, 2018 4:51:34 AM EDT)
Saving inventory complete

End of install phases.(Tuesday, July 10, 2018 4:51:35 AM EDT)

..................................................................................
Installation in progress (Tuesday, July 10, 2018 4:51:37 AM EDT)
                                              82% Done.
Install successful

Linking in progress (Tuesday, July 10, 2018 4:51:37 AM EDT)
Link successful

Setup in progress (Tuesday, July 10, 2018 4:51:37 AM EDT)
Setup successful

Saving inventory (Tuesday, July 10, 2018 4:51:37 AM EDT)
Saving inventory complete

End of install phases.(Tuesday, July 10, 2018 4:51:38 AM EDT)

...........................................................................................
Installation in progress (Tuesday, July 10, 2018 4:51:39 AM EDT)
                                     91% Done.
Install successful

Linking in progress (Tuesday, July 10, 2018 4:51:40 AM EDT)
Link successful

Setup in progress (Tuesday, July 10, 2018 4:51:40 AM EDT)
Setup successful

Saving inventory (Tuesday, July 10, 2018 4:51:40 AM EDT)
Saving inventory complete

End of install phases.(Tuesday, July 10, 2018 4:51:41 AM EDT)

..............................................................................................
Installation in progress (Tuesday, July 10, 2018 4:51:42 AM EDT)
                                  94% Done.
Install successful

Linking in progress (Tuesday, July 10, 2018 4:51:43 AM EDT)
Link successful

Setup in progress (Tuesday, July 10, 2018 4:51:43 AM EDT)
Setup successful

Saving inventory (Tuesday, July 10, 2018 4:51:43 AM EDT)
Saving inventory complete

End of install phases.(Tuesday, July 10, 2018 4:51:44 AM EDT)
OMS OracleHome :/u01/app/oracle/em13cr3/middleware
13NGCHEKAGGREGATE  : oracle.sysman.top.agent
13NGCHEKAGGREGATE  : OuiConfigVariables
13NGCHEKAGGREGATE  : OuiConfigVariables
13NGCHEKAGGREGATE  : oracle.sysman.top.oms
13NGCHEKAGGREGATE  : encap_oms
Setting the value of EMPREREQKIT_RESULTDIR /u01/app/oracle/em13cr3/middleware/.gcinstall_temp
Processing command line ....
Repository Creation Utility - Checking Prerequisites
Checking Global Prerequisites
Repository Creation Utility - Checking Prerequisites
Checking Component Prerequisites
Repository Creation Utility - Creating Tablespaces
Validating and Creating Tablespaces
Repository Creation Utility - Create
Repository Create in progress.
TNS Test Inserted Successfully
Successfully Added TNS Query Descriptors

  ********   ORACLE_HOME is /u01/app/oracle/em13cr3/middleware
test properties path: /u01/app/oracle/em13cr3/middleware/sysman/admin/emdrep/prop/imap.properties
CreateTestType:createCompleteTest: BEGIN
CreateTestType:createTestMetadataObject: BEGIN
CreateTestType:createTestMetadataObject: END
CreateTestType:createPropertyGroups: BEGIN
CreateTestType:createPropertyGroups: END
CreateTestType:createStrayProperties: BEGIN
CreateTestType:createStrayProperties: END
CreateTestType:createDefaultPromotions: BEGIN
CreateTestType:createDefaultPromotions: END
CreateTestType:createDefaultThresholds: BEGIN
CreateTestType:createDefaultThresholds: END
CreateTestType:createMetrics4TestType: BEGIN
CreateTestType:createMetrics4TestType: END
PropertyName :imap_host
PropertyName :imap_port
PropertyName :imap_user_name
PropertyName :imap_password
PropertyName :numretries
PropertyName :retryinterval
CreateTestType:createQueryDescriptor: BEGIN
CreateTestType:createQueryDescriptor: END
Enabled test for: IMAP , generic_service , 1.0
Enabled test for: IMAP , aggregate_service , 1.0
CreateTestType:createCompleteTest: END
  ********   ORACLE_HOME is /u01/app/oracle/em13cr3/middleware
Repository Creation Utility: Create - Completion Summary
Database details:
Connect Descriptor                      : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=oem13cr3.novalocal)(PORT=1521)))(CONNECT_DATA=(SID=EMPRD)))
Connected As                    : SYS
RCU Logfile                     : /u01/app/oracle/em13cr3/middleware/sysman/log/schemamanager/m_071018_0457_AM/m_071018_0457_AM.CREATE/rcu.log
Component schemas created:
Component                       Status  Logfile
EM Repository Creation Configuration            Success /u01/app/oracle/em13cr3/middleware/sysman/log/schemamanager/m_071018_0457_AM/m_071018_0457_AM.CREATE/em_repos_config.log
EM Repository Init Configuration                Success /u01/app/oracle/em13cr3/middleware/sysman/log/schemamanager/m_071018_0457_AM/m_071018_0457_AM.CREATE/em_repos_init.log
EM Repository Common            Success /u01/app/oracle/em13cr3/middleware/sysman/log/schemamanager/m_071018_0457_AM/m_071018_0457_AM.CREATE/em_repos_common.log

Repository Creation Utility - Create : Operation Completed
-------------------This action is not required ------------------
The correct order of execution is :
-showPrereqs , -runPrerequisites , -showCorrectiveActions , -runCorrectiveActions , -runPrerequisites(optional at this point), -showPostCorrectiveActions , -runPostCorrectiveActions
Setting the value of EMPREREQKIT_RESULTDIR /u01/app/oracle/em13cr3/middleware/.gcinstall_temp
Processing command line ....
Repository Creation Utility - Checking Prerequisites
Checking Global Prerequisites
Repository Creation Utility - Checking Prerequisites
Checking Component Prerequisites
Repository Creation Utility - Creating Tablespaces
Validating and Creating Tablespaces
RCU-6108:DB TablespaceFreeMB Prerequisite failure for: MGMT_TABLESPACE
Current space is 19MB. It should be be greater than 50MB.
Repository Creation Utility - Create
Repository Create in progress.
Percent Complete: 30
Percent Complete: 65
Percent Complete: 65
Percent Complete: 70
Percent Complete: 70
Percent Complete: 75
Percent Complete: 80
Percent Complete: 80
Percent Complete: 97
Percent Complete: 100
Repository Creation Utility: Create - Completion Summary
Database details:
Connect Descriptor                      : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=oem13cr3.novalocal)(PORT=1521)))(CONNECT_DATA=(SID=EMPRD)))
Connected As                    : SYS
Prefix for (prefixable) Schema Owners : SYSMAN
RCU Logfile                     : /u01/app/oracle/em13cr3/middleware/sysman/log/schemamanager/m_071018_0457_AM/m_071018_0457_AM.CREATE/rcu.log
Component schemas created:
Component                       Status  Logfile
Metadata Services               Success /u01/app/oracle/em13cr3/middleware/sysman/log/schemamanager/m_071018_0457_AM/m_071018_0457_AM.CREATE/mds.log

Repository Creation Utility - Create : Operation Completed
Processing command line ....
Repository Creation Utility - Checking Prerequisites
Checking Global Prerequisites
Repository Creation Utility - Checking Prerequisites
Checking Component Prerequisites
Repository Creation Utility - Creating Tablespaces
Validating and Creating Tablespaces
RCU-6108:DB TablespaceFreeMB Prerequisite failure for: MGMT_TABLESPACE
Current space is 19MB. It should be be greater than 50MB.
RCU-6108:DB TablespaceFreeMB Prerequisite failure for: TEMP
Current space is 30MB. It should be be greater than 50MB.
Repository Creation Utility - Create
Repository Create in progress.
Percent Complete: 50
Percent Complete: 50
Percent Complete: 60
Percent Complete: 60
Percent Complete: 75
Percent Complete: 87
Percent Complete: 93
Percent Complete: 96
Percent Complete: 96
Percent Complete: 98
Percent Complete: 100
Repository Creation Utility: Create - Completion Summary
Database details:
Connect Descriptor                      : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=oem13cr3.novalocal)(PORT=1521)))(CONNECT_DATA=(SID=EMPRD)))
Connected As                    : SYS
Prefix for (prefixable) Schema Owners : SYSMAN
RCU Logfile                     : /u01/app/oracle/em13cr3/middleware/sysman/log/schemamanager/m_071018_0457_AM/m_071018_0457_AM.CREATE/rcu.log
Component schemas created:
Component                       Status  Logfile
Oracle Platform Security Services               Success /u01/app/oracle/em13cr3/middleware/sysman/log/schemamanager/m_071018_0457_AM/m_071018_0457_AM.CREATE/opss.log

Repository Creation Utility - Create : Operation Completed
Processing command line ....
Repository Creation Utility - Checking Prerequisites
Checking Global Prerequisites
Repository Creation Utility - Checking Prerequisites
Checking Component Prerequisites
Repository Creation Utility - Creating Tablespaces
Validating and Creating Tablespaces
RCU-6108:DB TablespaceFreeMB Prerequisite failure for: MGMT_TABLESPACE
Current space is 29MB. It should be be greater than 50MB.
Repository Creation Utility - Create
Repository Create in progress.
Percent Complete: 50
Percent Complete: 50
Percent Complete: 55
Percent Complete: 60
Percent Complete: 60
Percent Complete: 60
Percent Complete: 80
Percent Complete: 100
Repository Creation Utility: Create - Completion Summary
Database details:
Connect Descriptor                      : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=oem13cr3.novalocal)(PORT=1521)))(CONNECT_DATA=(SID=EMPRD)))
Connected As                    : SYS
Prefix for (prefixable) Schema Owners : SYSMAN
RCU Logfile                     : /u01/app/oracle/em13cr3/middleware/sysman/log/schemamanager/m_071018_0457_AM/m_071018_0457_AM.CREATE/rcu.log
Component schemas created:
Component                       Status  Logfile
Common Infrastructure Services          Success /u01/app/oracle/em13cr3/middleware/sysman/log/schemamanager/m_071018_0457_AM/m_071018_0457_AM.CREATE/stb.log

Repository Creation Utility - Create : Operation Completed
Setting the value of EMPREREQKIT_RESULTDIR /u01/app/oracle/em13cr3/middleware/.gcinstall_temp
Middleware home/u01/app/oracle/em13cr3/middleware
2018_07_10_AD_06_11_45_EDT BIPLATFORM EMSchemaManagerHelper: Info : Oracle Home is /u01/app/oracle/em13cr3/middleware
2018_07_10_AD_06_11_45_EDT BIPLATFORM EMSchemaManagerHelper: Info : Schema Oracle Home is /u01/app/oracle/em13cr3/middleware/bi
2018_07_10_AD_06_11_45_EDT BIPLATFORM EMSchemaManagerHelper: Info : Setting RCU_HOME to /u01/app/oracle/em13cr3/middleware/oracle_common
2018_07_10_AD_06_11_45_EDT BIPLATFORM EMSchemaManagerHelper: Info : Setting SCRIPT_HOME to /u01/app/oracle/em13cr3/middleware/bi/rcu/integration
2018_07_10_AD_06_11_45_EDT BIPLATFORM EMSchemaManagerHelper: Info : Setting RCU_LOG_LOCATION to /u01/app/oracle/em13cr3/middleware/cfgtoollogs/bip
2018_07_10_AD_06_11_45_EDT BIPLATFORM EMSchemaManagerHelper: Info : Setting RCU_LOG_NAME to emBIPLATFORM.log
2018_07_10_AD_06_11_45_EDT BIPLATFORM version is null
2018_07_10_AD_06_11_45_EDT has BIPLATFORM user? false
2018_07_10_AD_06_11_45_EDT BIPLATFORM EMSchemaManagerHelper: Info : Prepare to initialize RCU
2018_07_10_AD_06_11_45_EDT BIPLATFORM EMSchemaManagerHelper: Info : RCU initialized with parameters : -silent -createRepository -compInfoXMLLocation /u01/app/oracle/em13cr3/middleware/sysman/emdrep/config/schemaext/ComponentInfo.xml -storageXMLLocation /u01/app/oracle/em13cr3/middleware/sysman/emdrep/config/schemaext/Storage.xml -connectString (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=oem13cr3.novalocal)(PORT=1521)))(CONNECT_DATA=(SID=EMPRD))) -dbUser SYS -dbRole sysdba -schemaPrefix SYSMAN -component BIPLATFORM
2018_07_10_AD_06_11_45_EDT BIPLATFORM EMSchemaManagerHelper: Info : Invoking RCU
Processing command line ....
Repository Creation Utility - Checking Prerequisites
Checking Global Prerequisites
Repository Creation Utility - Checking Prerequisites
Checking Component Prerequisites
Repository Creation Utility - Creating Tablespaces
Validating and Creating Tablespaces
Repository Creation Utility - Create
Repository Create in progress.
Percent Complete: 50
Percent Complete: 50
Percent Complete: 55
Percent Complete: 65
Percent Complete: 77
Percent Complete: 88
Percent Complete: 94
Percent Complete: 97
Percent Complete: 98
Percent Complete: 99
Percent Complete: 99
Percent Complete: 99
Percent Complete: 99
Percent Complete: 99
Percent Complete: 99
Percent Complete: 99
Percent Complete: 99
Percent Complete: 99
Percent Complete: 99
Percent Complete: 99
Percent Complete: 99
Percent Complete: 100
Repository Creation Utility: Create - Completion Summary
Database details:
Connect Descriptor                      : (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=oem13cr3.novalocal)(PORT=1521)))(CONNECT_DATA=(SID=EMPRD)))
Connected As                    : SYS
Prefix for (prefixable) Schema Owners : SYSMAN
RCU Logfile                     : /u01/app/oracle/em13cr3/middleware/cfgtoollogs/bip/emBIPLATFORM.log
Component schemas created:
Component                       Status  Logfile
Business Intelligence Platform          Success /u01/app/oracle/em13cr3/middleware/cfgtoollogs/bip/biplatform.log

Repository Creation Utility - Create : Operation Completed
2018_07_10_AD_06_11_58_EDT BIPLATFORM EMSchemaManagerHelper: Info : RCU Completed : COMPLETED
2018_07_10_AD_06_11_58_EDT BIPLATFORM EMSchemaManagerHelper: Info : Asscociating profile
Unable to create logger...
2018_07_10_AD_06_11_58_EDT BIPLATFORM EMSchemaManagerHelper: Info : profile associated
Successfully created SYSMAN_BIPLATFORM schema...
The AgentFreshInstaller is starting now
Oraclehome : ../u01/app/oracle/agent/agent13cr3/agent_13.3.0.0.0
InstanceHome : /u01/app/oracle/agent/agent13cr3/agent_inst
Agent Base Directory : /u01/app/oracle/agent/agent13cr3
The oraclehome /u01/app/oracle/agent/agent13cr3/agent_13.3.0.0.0
startAgent is:true
seci is :true
startAgent is:true
seci is :true
log loction is setlog
Creating log directoyr :/u01/app/oracle/agent/agent13cr3/agent_13.3.0.0.0/cfgtoollogs/agentDeploy
Writing the following contents into /u01/app/oracle/agent/agent13cr3/agent_13.3.0.0.0/install/oragchomelist
/u01/app/oracle/agent/agent13cr3/agent_13.3.0.0.0:/u01/app/oracle/agent/agent13cr3/agent_inst
Both /etc/oragchomelist and /var/opt/oracle/oragchomelist does not exist.
Agent Home is : {0}
The value of chainInstall : true forceConfigure : false skipValidation : false
Validated the oms host and port :- oem13cr3.novalocal----4903
Logs Location is : {0}
Getting Inet Addresses for host oem13cr3.novalocal
** Agent Port Check completed successfully.**
Validated the agent port :- ----3872
Executing command: {0}
shared agent value is :false

Setting system property CUSTOM_INVENTORY to {0}
chain install is :true

Cloning of agent home completed successfully
Agent Configuration completed successfully
The following configuration scripts need to be executed as the "root" user. Root script to run : /u01/app/oracle/agent/agent13cr3/agent_13.3.0.0.0/root.sh
Prompt for the allroot.sh
Logs successfully copied to /u01/app/oraInventory/logs.