ODA - Patching Process and Considerations - Non-virtualized (OL6 Upgrade)

ODA or Oracle Database Appliance is one of the fastest growing Engineered system and is really popular with smaller and medium-end enterprises.

In this blog I am going to discuss  patching process and considerations for ODA.

I am discussing OL6 Upgrade, though the patch is now almost an year old, however the patching considerations are still the same and helpful in planning future upgrades as well.

Patching Considerations and things to remember when Planning

1. Key thing to remember in ODA patching is that is a 3 to 5 command patching process which will do the patching for you, for example running Opatch, installing rpms (in case of Linux Upgrade)So if you are doing patching and there is some failure you can verify that in the logs /opt/oracle/oak/log/<hostname>/client/oakcli.log
2. Make sure to run commands such as pre-checks prior to the day of upgrade, this saves serious time when you are doing an upgrade and hit issue in pre-checks
3. Generally when you are doing upgrades you will have a folder named your patch version in the hostname directory which is like - /opt/oracle/oak/log/<hostname/patch/12.1.2.6.0 for example. 
4. Black out your OEM
5. Un-mount any network and cloudfs (ACFS mounts), in case of network mount you can disable it in /etc/fstab
6.  If your machine has been up for a long time, do a rolling reboot to make sure everything comes back up. (remember to unmount the acfs again)
7. Prefer local only upgrades to identify issues on one node for HA
8. Disable your redo transport if you are having a DataGuard Solution or stop GG Processes in case of GoldenGate configuration
9. Complete your patching process
10. In case you are having an upgrade in the patching path for GI / Database then make sure to make approrpriate changes to Database homes and GI homes in OEM.
    You need to make changes to below monitoring configurations 

1. Listener (Scan + Local)
2. Database system / instance / RAC Homes
3. Scan Listeners
4. ASM Cluster target Home
5. Cluster target Home (if not done automatically)

      10. Finally remove the blackout, I generally prefer to make changes post removing blackout - though I have written it otherwise for perfectionists and start any stopped monitoring, scripts, Redo transport and GoldenGate processes

The patching process ideally is a 3 step process which comprises of 

1. Patching the Server Components ( including Storage)#oakcli update –patch 12.1.2.6.0 --infra --local
2. Patching the GI #oakcli update –patch 12.1.2.6.0 --gi --local
3. Patching RDBMS#oakcli update -patch 12.1.2.6.0 --database --local

This is the process which was available prior to 12.1.2.7.0

From 12.1.2.7.0 version, the process has been tweaked a bit by Oracle viz. 

1. Patching the Server + GI#oakcli update -patch  12.1.2.7.0 --server -local
2. Patching the Storage#oakcli update -patch  12.1.2.7.0 --storage
3. Patching RDBMS #oakcli update -patch  12.1.2.7.0 --database --local

Brief Overview of complete patching process - 

1. Download and unpack the zip files from MoS on both the nodes
2. Verify to be Patched components
3. Execute Patching Step as defined above 
4. Finally Verify Components Patched and their version

I am upgrading to 12.1.2.6.0 first and then to 12.1.2.7.0 in the next blog

Upgrade to 12.1.2.6.0 - 
The key thing with this version is that Oracle gave Linux Upgrade with this version so we will be having few additional steps with this. 

Step 1 - Download and Unpack the zip files from MoS

#/opt/oracle/oak/bin/oakcli unpack –-package /u01/stage/patch/121260/p22328442_121260_Linux-x86-64_1of2.zip
#/opt/oracle/oak/bin/oakcli unpack –-package /u01/stage/patch/121260/p22328442_121260_Linux-x86-64_2of2.zip

Step 2 - Verify to be patched components

#/opt/oracle/oak/bin/oakcli update -patch  12.1.2.6.0 --verify

Step 3 - Validate OL6 Upgrade pre-checks

#/opt/oracle/oak/bin/oakcli validate -c ol6upgrade --prechecks

Step 4 - Upgrade Locally

#/opt/oracle/oak/bin/oakcli update -patch 12.1.2.6.0 --infra --local

#/opt/oracle/oak/bin/oakcli validate -c ol6upgrade --postchecks

Run 3 - 4 for both nodes one by one, don't do in parallel

Step 5 - Upgrade GI and Database 

#/opt/oracle/oak/bin/oakcli update -patch  12.1.2.6.0 --gi

Run above  for both Nodes one after another and finally run below on both nodes one after another.

#/opt/oracle/oak/bin/oakcli update -patch  12.1.2.6.0  --database  --local

We are doing a local only upgrade so we upgrade one node and then the other node.

GI Patching can be executed in more than one way for example - Patch Node 1 infra and then patch GI or patch Node 1 and Node 2 infra and then Patch GI.

I prefer component level upgrades i.e. first upgrade Infra on Node 1, make sure everything is online and then upgrade Infra on Node 2 and then go for GI component, this ensures component level consistency across nodes in case of failures / issues.

Also, 

Essentially Oracle is applying the the Bundle patches for GI and Database in the background.

You can always go ahead and verify the patches using the famous patching utility Opatch

However, you can check using oakcli interface as well.

Step 6 - Verify

#/opt/oracle/oak/bin/oakcli show dbhomes -detail

#/opt/oracle/oak/bin/oakcli show databases -detail

#/opt/oracle/oak/bin/oakcli update -patch  12.1.2.6.0 --verify

This completes the ODA Patching process to 12.1.2.6.0.
I prefer doing local upgrades only, it has really helped me over period of time. Generally if you have not played a lot with your default ODA configuration then everything generally moves on fine.