Oracle Database: Archive Log Repository

Oracle Database: Archive Log Repository

Upgrade Vineet Sachdeva   April 05, 2019

Archive log repository is one of the seldom used and known concepts in Oracle Database. 

An Oracle Database can be configured to send archive logs to remote destination without database being present in that site. 

To explain
1. Database Instance is running
2. Database Control File is present
3. The CF has to be standby CF (ensure this)
4. Database is in mount state
5. No Datafiles present 

If you configure your source database similar to a dataguard configuration, your primary will start sending archive logs to the Archive log repostiory site. 

What are the use case? 
1. Backup of archive logs on remote site 
2. Remote Site can be used for tape backup etc
3. Setup during dataguard setup - the time spent to backup and transfer the archivelogs can be used by sending the logs using the archive log repo. 
So when you have your database restored, you will have your archives already there. 

For More Information : https://docs.oracle.com/cd/E11882_01/server.112/e41134/log_transport.htm#SBYDB4745

Oracle Database 11g/18c: Installing 11gr2 on 18c (Things to Keep in Mind)

Oracle Database 11g/18c: Installing 11gr2 on 18c (Things to Keep in Mind)

Upgrade Vineet Sachdeva   March 15, 2019

In this blog I list out key things to keep in mind when doing installation of 11gR2 on 18c Cluster
This can be a useful case when you are thinking of migrating and Upgrading your infrastructure.


1. Keep Note of all the rpms and create a superset of both lists of 18c and 11g.
elfutils-libelf-devel.x86_64 is the rpm which is missing from 18c install. 

2. Prefer ASM LIB always for installation 
For RHEL - the kmod-oracleasm is available from RHN or in the Product DVD
for other libraries use - https://www.oracle.com/technetwork/server-storage/linux/asmlib/rhel7-2773795.html

3. If you want to share some FS with your previous severs, remember to create the users/groups with same UID / GID.

4. In the Grid Installation 

• Patch using -applyRU option before installation
• Apply Bug Patch for 27912127 
• OJVM patch is not valid for Grid release (check the readme carefully)
• Redhat Network configuration for NTP

5. Before DB Binaries Installation, follow below docs - 

• RAC RDBMS Installation fails with Error:"PRVF-4037 : CRS is not installed on any of the nodes" (Doc ID 2315020.1)
• Error PRVF-4037 On Install Of Oracle Database 11.2 Binaries With 12.2.0.1 Grid Infrastructure (Doc ID 2302700.1)
• PRVF-4037 : CRS is not installed on any of the nodes (Doc ID 1316815.1)
• error in invoking target 'agent nmhs' of make file ins_emagent.mk while installing Oracle 11.2.0.4 on Linux (Doc ID 2299494.1)     
• Linux:CVU NTP Prerequisite check fails with PRVF-7590, PRVG-1024 and PRVF-5415 (Doc ID 2126223.1)          

And as always refer MoS for any other erros.

Oracle Database - Windows 11.2.0.3 to Linux 11.2.0.4 Standard Edition SE Migrate (on Cloud)

Oracle Database - Windows 11.2.0.3 to Linux 11.2.0.4 Standard Edition SE Migrate (on Cloud)

Upgrade Vineet Sachdeva   June 14, 2018

I was recently tasked with a Standard Edition migration from on premise to cloud.

The migration -
1. Source - 11.2.0.3 Windows (2008 x64 Bit)
2. Destination - 11.2.0.4 Oracle Linux 7
3. Database - Standard Edition
4. Db Size - 10GB - 2TB (multiple)
5. Downtime - 6 hours
 
Challenges
 1. No Dataguard possible in Standard Edition
2. No Transportable Tablespaces possible in Standard Edition
3. Export / Import - too slow to run because of parallelism restriction in Standard Edition
4. 11.2.0.3 not supported on  Linux 7 (only 11.2.0.4)
5. Limited B.W available between sites 

Solution(s)
Direct Upgrade Small Database

1. Cold Backup 11.2.0.3
2. Transfer to Linux 
3. Restore on Linux (RMAN DUPLICATE/RESTORE/RECOVER Mixed Platform Support (Doc ID 1079563.1) 
4. Upgrade Database

• Source Db run utlu112i.sql (and other preqs as in Oracle documentation such as dictionary stats etc). Make fixes as required.
• Backup database and transfer to Linux
• restore database;
• alter database open resetlogs upgrade;
• catupgrd
• start database
• catupst
• Post migration (catupst, utlrp, utlu112s)

Sync Upgrade Big Database
 
1. Setup Intermediate VM on cloud - Linux 6.9
2. Install Linux Linux 6 and Linux 7
3. Setup Standard Edition Dataguard - using scripts to recover database continuously. (Data Guard Support for Heterogeneous Primary and Physical Standbys in Same Data Guard Configuration (Doc ID 413484.1)    
4. Do final sync of Primary and Standby 
5. Activate Physical Standby and Open it
6. Upgrade the database to 11.2.0.4
7. Migrate (or attach the volume ) to Linux 7

Sync Upgrade Big Database Direct solution

1. Setup DG between 11.2.0.3 Windows and 11.2.0.4 on Linux 7
Mixed Oracle Version support with Data Guard Redo Transport Services (Doc ID 785347.1)    
2. Activate and open reset logs upgrade
3. Follow Upgrade Steps of Small Database

 

Oracle Database - RAC/Grid Infrastructure upgrade - Flex Cluster 12.1.0.2 (.x) to 12.2.0.1 - 3 Nodes

Oracle Database - RAC/Grid Infrastructure upgrade - Flex Cluster 12.1.0.2 (.x) to 12.2.0.1 - 3 Nodes

Upgrade Anonymous   March 25, 2017

In this blog, I am going to discuss about upgrading clusterware from 12.1.0.2.x to the recently released (Feb 2017) 12cr2 clusterware 12.2.0.1.

I am having a setup of 3 node cluster with 2 nodes as hub nodes and 1 as leaf node.

[root@rac1 app]# crsctl get node role config -all
Node 'rac1' configured role is 'hub'
Node 'rac2' configured role is 'hub'
Node 'rac3' configured role is 'leaf'

I will be upgrading to R2 of 12c Clusterware from my 12.1.0.2 version
In my cluster upgrade, I always keep backup of cluster registry, cluster resources running and listeners & instances running on each node for finaly comparison (just in case)

If configuration is correct, all things will come up automatically, however few things might not - for example a listener running from DB home, not registered with Clusterware.

In general below is the advise for 12cR2 upgrade.

1. Backup OCR (Local & Global, Logical too)
2. Take Snapshot of resources (just in case you want to compare)
3. Have 33G+ free space in DG with OCR as this is mandatory requirement for upgrade else you will get an error -  INS-43100 asking for space
4. Have your memory requirements right 8GB + (Ignored in testing lab environment)
5. Have /etc/resolv.conf files matching across all nodes
6. Have the mandatory patch installed for GI Upgrade which is Patch - 21255373 
7. Have atleast 15GB free space on your installation mount point.
8. And lastly make sure you have on all servers clusterware up and running (There is an option in the installer to skip upgrade on unreachable nodes, which I doubt you want it)

9. Finally run the Upgrade

Note - in my lab setup (if you have followed installation blogs)
I was having DG with 15GB (normal redundancy), I have created an external DG with 45G and relocated my OCR and Voting Disk on the new OCR Disk Group.

On  Node 1
[As root:]
Set the environment to 12.1.0.2.x Grid Home / (+ASM1)
#ocrconfig  -export /home/oracle/org/Upgrade12cGI/OCRLogicalBackupCluster.bak

#ocrconfig  -manualbackup -backuploc /home/oracle/org/Upgrade12cGI/OCRManualBackupCluster.bak

#ocrconfig -local -export /home/oracle/org/Upgrade12cGI/OCRLogicalBackup_Local.bak

[As root:]
Set the environment to 12.1.0.2.x Grid Home / (+ASM1)
#crsctl stat res -t > /home/oracle/org/Upgrade12cGI/crsctl_stat_res_t.log

On  Node 2, 3 and others (if more) 
[As root:]
Set the environment to 12.1.0.2.x Grid Home / (+ASM2) (+ASM3)
#ocrconfig -local -export /home/oracle/org/Upgrade12cGI/OCRLogicalBackup_Local.bak

On Each Node 
#ps -ef | grep pmon > /home/oracle/org/Upgrade12cGI/pmon_snapshot.log

# ps -ef | grep tns > /home/oracle/org/Upgrade12cGI/tns_snapshot.log

Create a blackout in OEM

Create Grid Home Directory on each node 

[As root:]
#mkdir -p /u01/app/12.2.0.1/grid
# chown -R grid:oinstall /u01/app/12.2.0.1/

There is a minimum patch requirement of Patch which is done as part of pre-reqs check - 21255373. 

Make sure this patch version is installed.

Unzip Oracle Media on first node. 

cd /u01/app/12.2.0.1/grid

#unzip -qq /media/sf_RAC12c/Grid/12.2.0.1/V840012-01.zip

#chown -R grid:oinstall /u01/app/12.2.0.1/grid

[As Grid]  - Run Cluster Verification in pre - crs install stage

#cd /u01/app/12.2.0.1/grid

#./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome  /u01/app/12.1.0.2/grid -dest_crshome /u01/app/12.2.0.1/grid -dest_version 12.2.0.1 -fixup -verbose  | tee /tmp/cluvfy_upgd.out

I generally redirect output using tee command to see it incoming and also have a copy

Fix anything which is shown as failed. I will be  ignoring few pre-reqs such as memory requirement.

Finally Unset any environment variables related to Oracle
# unset ORACLE_HOME ORACLE_SID ORACLE_BASE

In this version Oracle has finally come up with a new name gridSetup which we are going to run from our unzipped directory.
Note - Oracle has given an image based installation, which is more of an existing Image of the install and the re-linking process. 
This is really good in standalone based installations, and is much faster also.

[grid@rac1 grid]$ ./gridSetup.sh 

In case SSH connectivity is not there, it can be setup using the given button in the screen. 

I prefer setting it up using sshUserSetup.sh file which is present in the installation unzipped file.

The groups which are chosen by default are the ones used in your installed configuration, however it is still good idea to verify at the time of installation to be double sure.

An highlight here is that I am not specifying Oracle home in this release during installation.
Oracle home is taken where I have unzipped my software.

I prefer having self execution of root scripts and also allows to run in parallel if many nodes are there. 

At this point in the upgrade, run the rootupgrade.sh script.

The output of rootupgrade.sh script in this release is bit different from previous release(s).

 It consists of 19 steps which include the upgrade of OCR and resources.

I will be running root upgrade on rac1, then rac2 and finally rac3 (flex -leaf node).

I have pasted output of running rootupgrade.sh at the end of the blog to make look installation process bit tidy. You can see that in the end.

Note - I have ignored few cluvfy checks, due my lab machines restriction, however all other checks are passed.

[grid@rac2 ~]$ crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [12.2.0.1.0]

This completes the upgrade of RAC cluster to 12.2.0.1

Root Upgrade execution.


******root Upgrade Node 1*****
[root@rac1 trace]# /u01/app/12.2.0.1/grid/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/12.2.0.1/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: n
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/12.2.0.1/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/rac1/crsconfig/rootcrs_rac1_2017-03-24_10-56-45PM.log
2017/03/24 22:56:48 CLSRSC-595: Executing upgrade step 1 of 19: 'UpgradeTFA'.
2017/03/24 22:56:48 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2017/03/24 22:57:50 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2017/03/24 22:57:51 CLSRSC-595: Executing upgrade step 2 of 19: 'ValidateEnv'.
2017/03/24 22:58:17 CLSRSC-595: Executing upgrade step 3 of 19: 'GenSiteGUIDs'.
2017/03/24 22:58:19 CLSRSC-595: Executing upgrade step 4 of 19: 'GetOldConfig'.
2017/03/24 22:58:19 CLSRSC-464: Starting retrieval of the cluster configuration data
2017/03/24 22:58:29 CLSRSC-515: Starting OCR manual backup.
2017/03/24 22:58:39 CLSRSC-516: OCR manual backup successful.
2017/03/24 22:58:56 CLSRSC-486: 
 At this stage of upgrade, the OCR has changed.
 Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.
2017/03/24 22:58:56 CLSRSC-541: 
 To downgrade the cluster: 
 1. All nodes that have been upgraded must be downgraded.
2017/03/24 22:58:57 CLSRSC-542: 
 2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.
2017/03/24 22:58:57 CLSRSC-615: 
 3. The last node to downgrade cannot be a Leaf node.
2017/03/24 22:59:06 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2017/03/24 22:59:06 CLSRSC-595: Executing upgrade step 5 of 19: 'UpgPrechecks'.
2017/03/24 22:59:10 CLSRSC-363: User ignored prerequisites during installation
2017/03/24 22:59:22 CLSRSC-595: Executing upgrade step 6 of 19: 'SaveParamFile'.
2017/03/24 22:59:31 CLSRSC-595: Executing upgrade step 7 of 19: 'SetupOSD'.
2017/03/24 22:59:44 CLSRSC-595: Executing upgrade step 8 of 19: 'PreUpgrade'.
2017/03/24 22:59:49 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode
2017/03/24 22:59:49 CLSRSC-482: Running command: '/u01/app/12.1.0.2/grid/bin/crsctl start rollingupgrade 12.2.0.1.0'
CRS-1131: The cluster was successfully set to rolling upgrade mode.
2017/03/24 23:00:06 CLSRSC-482: Running command: '/u01/app/12.2.0.1/grid/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /u01/app/12.1.0.2/grid -oldCRSVersion 12.1.0.2.0 -firstNode true -startRolling false '

ASM configuration upgraded in local node successfully.

2017/03/24 23:00:09 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode
2017/03/24 23:00:24 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2017/03/24 23:00:54 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
2017/03/24 23:00:57 CLSRSC-595: Executing upgrade step 9 of 19: 'CheckCRSConfig'.
2017/03/24 23:00:57 CLSRSC-595: Executing upgrade step 10 of 19: 'UpgradeOLR'.
2017/03/24 23:01:07 CLSRSC-595: Executing upgrade step 11 of 19: 'ConfigCHMOS'.
2017/03/24 23:01:07 CLSRSC-595: Executing upgrade step 12 of 19: 'InstallAFD'.
2017/03/24 23:01:14 CLSRSC-595: Executing upgrade step 13 of 19: 'createOHASD'.
2017/03/24 23:01:21 CLSRSC-595: Executing upgrade step 14 of 19: 'ConfigOHASD'.
2017/03/24 23:01:37 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2017/03/24 23:02:00 CLSRSC-595: Executing upgrade step 15 of 19: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/03/24 23:02:33 CLSRSC-595: Executing upgrade step 16 of 19: 'InstallKA'.
2017/03/24 23:02:40 CLSRSC-595: Executing upgrade step 17 of 19: 'UpgradeCluster'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac1'-
CRS-2676: Start of 'ora.storage' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rac1'
CRS-2676: Start of 'ora.crf' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
CRS-6017: Processing resource auto-start for servers: rac1
CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac2'
CRS-2672: Attempting to start 'ora.ons' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac2'
CRS-2677: Stop of 'ora.rac1.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.rac1.vip' on 'rac1'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac1'
CRS-2676: Start of 'ora.rac1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'rac1'
CRS-2676: Start of 'ora.scan1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
CRS-2676: Start of 'ora.ons' on 'rac1' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.DATA2.dg' on 'rac1'
CRS-2672: Attempting to start 'ora.OCR.dg' on 'rac1'
CRS-2676: Start of 'ora.DATA2.dg' on 'rac1' succeeded
CRS-2676: Start of 'ora.OCR.dg' on 'rac1' succeeded
CRS-6016: Resource auto-start has completed for server rac1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2017/03/24 23:06:35 CLSRSC-343: Successfully started Oracle Clusterware stack
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 1.
Successfully taken the backup of node specific configuration in OCR. 
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2017/03/24 23:08:32 CLSRSC-595: Executing upgrade step 18 of 19: 'UpgradeNode'.
2017/03/24 23:09:04 CLSRSC-474: Initiating upgrade of resource types
2017/03/24 23:13:46 CLSRSC-482: Running command: 'srvctl upgrade model -s 12.1.0.2.0 -d 12.2.0.1.0 -p first'
2017/03/24 23:13:48 CLSRSC-475: Upgrade of resource types successfully initiated.
2017/03/24 23:15:22 CLSRSC-595: Executing upgrade step 19 of 19: 'PostUpgrade'.
2017/03/24 23:15:45 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded



****Root Upgrade Node 2******
[root@rac2 trace]/u01/app/12.2.0.1/grid/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/12.2.0.1/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/12.2.0.1/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/rac2/crsconfig/rootcrs_rac2_2017-03-24_11-22-35PM.log
2017/03/24 23:22:43 CLSRSC-595: Executing upgrade step 1 of 19: 'UpgradeTFA'.
2017/03/24 23:22:43 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2017/03/24 23:26:46 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2017/03/24 23:26:47 CLSRSC-595: Executing upgrade step 2 of 19: 'ValidateEnv'.
2017/03/24 23:27:10 CLSRSC-595: Executing upgrade step 3 of 19: 'GenSiteGUIDs'.
2017/03/24 23:27:10 CLSRSC-595: Executing upgrade step 4 of 19: 'GetOldConfig'.
2017/03/24 23:27:10 CLSRSC-464: Starting retrieval of the cluster configuration data
2017/03/24 23:27:46 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2017/03/24 23:27:47 CLSRSC-595: Executing upgrade step 5 of 19: 'UpgPrechecks'.
2017/03/24 23:27:51 CLSRSC-363: User ignored prerequisites during installation
2017/03/24 23:27:56 CLSRSC-595: Executing upgrade step 6 of 19: 'SaveParamFile'.
2017/03/24 23:28:10 CLSRSC-595: Executing upgrade step 7 of 19: 'SetupOSD'.
2017/03/24 23:28:16 CLSRSC-595: Executing upgrade step 8 of 19: 'PreUpgrade'.

ASM configuration upgraded in local node successfully.

2017/03/24 23:28:35 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2017/03/24 23:29:21 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
2017/03/24 23:29:26 CLSRSC-595: Executing upgrade step 9 of 19: 'CheckCRSConfig'.
2017/03/24 23:29:27 CLSRSC-595: Executing upgrade step 10 of 19: 'UpgradeOLR'.
2017/03/24 23:29:34 CLSRSC-595: Executing upgrade step 11 of 19: 'ConfigCHMOS'.
2017/03/24 23:29:35 CLSRSC-595: Executing upgrade step 12 of 19: 'InstallAFD'.
2017/03/24 23:29:36 CLSRSC-595: Executing upgrade step 13 of 19: 'createOHASD'.
2017/03/24 23:29:38 CLSRSC-595: Executing upgrade step 14 of 19: 'ConfigOHASD'.
2017/03/24 23:29:54 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2017/03/24 23:30:26 CLSRSC-595: Executing upgrade step 15 of 19: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/03/24 23:30:55 CLSRSC-595: Executing upgrade step 16 of 19: 'InstallKA'.
2017/03/24 23:30:57 CLSRSC-595: Executing upgrade step 17 of 19: 'UpgradeCluster'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
CRS-2672: Attempting to start 'ora.evmd' on 'rac2'
CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac2'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac2'
CRS-2676: Start of 'ora.asm' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac2'
CRS-2676: Start of 'ora.storage' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rac2'
CRS-2676: Start of 'ora.crf' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac2'
CRS-2676: Start of 'ora.crsd' on 'rac2' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rac2
CRS-2673: Attempting to stop 'ora.rac2.vip' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1'
CRS-2672: Attempting to start 'ora.ons' on 'rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1'
CRS-2677: Stop of 'ora.rac2.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.rac2.vip' on 'rac2'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac2'
CRS-2676: Start of 'ora.rac2.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'rac2'
CRS-2676: Start of 'ora.scan1.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac2'
CRS-2676: Start of 'ora.ons' on 'rac2' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'rac2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeeded
CRS-6016: Resource auto-start has completed for server rac2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2017/03/24 23:32:29 CLSRSC-343: Successfully started Oracle Clusterware stack
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 2.
Successfully taken the backup of node specific configuration in OCR. 
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2017/03/24 23:32:45 CLSRSC-595: Executing upgrade step 18 of 19: 'UpgradeNode'.
2017/03/24 23:33:06 CLSRSC-595: Executing upgrade step 19 of 19: 'PostUpgrade'.
2017/03/24 23:33:11 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded


****Root Upgrade Node 3********
[root@rac3 ~]# /u01/app/12.2.0.1/grid/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/12.2.0.1/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) 
[n]: 
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/12.2.0.1/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/rac3/crsconfig/rootcrs_rac3_2017-03-24_11-37-53PM.log
2017/03/24 23:38:01 CLSRSC-595: Executing upgrade step 1 of 19: 'UpgradeTFA'.
2017/03/24 23:38:01 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2017/03/24 23:40:24 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2017/03/24 23:40:24 CLSRSC-595: Executing upgrade step 2 of 19: 'ValidateEnv'.
2017/03/24 23:40:38 CLSRSC-595: Executing upgrade step 3 of 19: 'GenSiteGUIDs'.
2017/03/24 23:40:38 CLSRSC-595: Executing upgrade step 4 of 19: 'GetOldConfig'.
2017/03/24 23:40:39 CLSRSC-464: Starting retrieval of the cluster configuration data
2017/03/24 23:44:15 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2017/03/24 23:44:16 CLSRSC-595: Executing upgrade step 5 of 19: 'UpgPrechecks'.
2017/03/24 23:44:20 CLSRSC-363: User ignored prerequisites during installation
2017/03/24 23:44:24 CLSRSC-595: Executing upgrade step 6 of 19: 'SaveParamFile'.
2017/03/24 23:44:37 CLSRSC-595: Executing upgrade step 7 of 19: 'SetupOSD'.
2017/03/24 23:44:50 CLSRSC-595: Executing upgrade step 8 of 19: 'PreUpgrade'.
2017/03/24 23:45:00 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2017/03/24 23:45:10 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
2017/03/24 23:45:48 CLSRSC-595: Executing upgrade step 9 of 19: 'CheckCRSConfig'.
2017/03/24 23:45:50 CLSRSC-595: Executing upgrade step 10 of 19: 'UpgradeOLR'.
2017/03/24 23:45:58 CLSRSC-595: Executing upgrade step 11 of 19: 'ConfigCHMOS'.
2017/03/24 23:45:59 CLSRSC-595: Executing upgrade step 12 of 19: 'InstallAFD'.
2017/03/24 23:46:01 CLSRSC-595: Executing upgrade step 13 of 19: 'createOHASD'.
2017/03/24 23:46:04 CLSRSC-595: Executing upgrade step 14 of 19: 'ConfigOHASD'.
2017/03/24 23:46:20 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2017/03/24 23:46:44 CLSRSC-595: Executing upgrade step 15 of 19: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac3'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac3' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/03/24 23:47:21 CLSRSC-595: Executing upgrade step 16 of 19: 'InstallKA'.
2017/03/24 23:47:23 CLSRSC-595: Executing upgrade step 17 of 19: 'UpgradeCluster'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac3'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac3' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac3'
CRS-2672: Attempting to start 'ora.evmd' on 'rac3'
CRS-2676: Start of 'ora.evmd' on 'rac3' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rac3' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac3'
CRS-2676: Start of 'ora.gpnpd' on 'rac3' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac3'
CRS-2676: Start of 'ora.gipcd' on 'rac3' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac3'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac3' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac3'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac3'
CRS-2676: Start of 'ora.diskmon' on 'rac3' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac3' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac3'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac3'
CRS-2676: Start of 'ora.ctssd' on 'rac3' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac3'
CRS-2676: Start of 'ora.storage' on 'rac3' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rac3'
CRS-2676: Start of 'ora.crf' on 'rac3' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac3'
CRS-2676: Start of 'ora.crsd' on 'rac3' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac3' succeeded
CRS-6017: Processing resource auto-start for servers: rac3
CRS-6016: Resource auto-start has completed for server rac3
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2017/03/24 23:48:07 CLSRSC-343: Successfully started Oracle Clusterware stack
Successfully taken the backup of node specific configuration in OCR. 
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2017/03/24 23:48:28 CLSRSC-595: Executing upgrade step 18 of 19: 'UpgradeNode'.
Start upgrade invoked..
2017/03/24 23:48:38 CLSRSC-478: Setting Oracle Clusterware active version on the last node to be upgraded
2017/03/24 23:48:38 CLSRSC-482: Running command: '/u01/app/12.2.0.1/grid/bin/crsctl set crs activeversion'
Started to upgrade the active version of Oracle Clusterware. This operation may take a few minutes.
Started to upgrade CSS.
CSS was successfully upgraded.
Started to upgrade Oracle ASM.
Started to upgrade CRS.
CRS was successfully upgraded.
Successfully upgraded the active version of Oracle Clusterware.
Oracle Clusterware active version was successfully set to 12.2.0.1.0.
2017/03/24 23:49:50 CLSRSC-479: Successfully set Oracle Clusterware active version
2017/03/24 23:49:50 CLSRSC-476: Finishing upgrade of resource types
2017/03/24 23:50:33 CLSRSC-482: Running command: 'srvctl upgrade model -s 12.1.0.2.0 -d 12.2.0.1.0 -p last'
2017/03/24 23:50:34 CLSRSC-477: Successfully completed upgrade of resource types
2017/03/24 23:51:19 CLSRSC-595: Executing upgrade step 19 of 19: 'PostUpgrade'.
2017/03/24 23:51:52 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded