Advertisement

Wednesday, August 29, 2018

Oracle Database 18c: RAC Upgrade 12c to 18c (12.2.0.1 to 18.3) - Part 2/2

In this blog, I pickup from my last blog where I had done all the pre-reqs successfully. 

In this blog we will be doing the real upgrade. 

Connect using user - grid (make sure 'X' is enabled and you use 'X' server - I prefer MobaXterm)

cd /u01/app/180/grid
./gridSetup.sh

Step 1 - Select Upgrade 

Step 2 - Select Node - Do not select skip unreachable nodes

Step 3 - Include EM registration if you are using EM in your environment 

Step 4 - Select your Grid Base

Step 5  - We Will run the script manually

Step 6 - If all okay - you should go to the summary screen, Save response file if you want and click "Submit"


Before the next step. If you have any patches. You can install them right away. 
Because when you install the patches before the root script execution, your cluster will have the latest patch.

Though in case of 18c we always have latest RU of the quarter :)

Step 7 - run rootupgrade.sh script. 
Refer end of blog to see the log of the execution.



Step 8 - Click OK after rootupgarde execution is complete on all nodes.





Step 9 - Upgrade is complete Click Close and Verify your cluster now. 




Note - if you had made any entries in /etc/oratab - Correct them to point to new homes respectively for ASM,APX and MGMT DB. 



[root@prodrac01 ~]# crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [18.0.0.0.0]
[grid@prodrac01 OPatch]$ pwd
/u01/app/180/grid/OPatch
[grid@prodrac01 OPatch]$ ./opatch lspatches
pw27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)
28256701;TOMCAT RELEASE UPDATE 18.3.0.0.0 (28256701)
28090564;DBWLM RELEASE UPDATE 18.3.0.0.0 (28090564)
28090557;ACFS RELEASE UPDATE 18.3.0.0.0 (28090557)
28090553;OCW RELEASE UPDATE 18.3.0.0.0 (28090553)
28090523;Database Release Update : 18.3.0.0.180717 (28090523)

Rootupgradelog
Node 1 - prodrac01

[root@prodrac01 ~]# /u01/app/180/grid/rootupgrade.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/180/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/180/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/prodrac01/crsconfig/rootcrs_prodrac01_2018-08-29_05-05-11AM.log
2018/08/29 05:05:26 CLSRSC-595: Executing upgrade step 1 of 19: 'UpgradeTFA'.
2018/08/29 05:05:26 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2018/08/29 05:06:02 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2018/08/29 05:06:02 CLSRSC-595: Executing upgrade step 2 of 19: 'ValidateEnv'.
2018/08/29 05:06:06 CLSRSC-595: Executing upgrade step 3 of 19: 'GetOldConfig'.
2018/08/29 05:06:06 CLSRSC-464: Starting retrieval of the cluster configuration data
2018/08/29 05:06:14 CLSRSC-692: Checking whether CRS entities are ready for upgrade. This operation may take a few minute
2018/08/29 05:07:48 CLSRSC-693: CRS entities validation completed successfully.
2018/08/29 05:07:53 CLSRSC-515: Starting OCR manual backup.
2018/08/29 05:08:02 CLSRSC-516: OCR manual backup successful.
2018/08/29 05:08:09 CLSRSC-486:
At this stage of upgrade, the OCR has changed.
Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.
2018/08/29 05:08:09 CLSRSC-541:
To downgrade the cluster:
1. All nodes that have been upgraded must be downgraded.
2018/08/29 05:08:09 CLSRSC-542:
2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.
2018/08/29 05:08:09 CLSRSC-615:
3. The last node to downgrade cannot be a Leaf node.
2018/08/29 05:08:15 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2018/08/29 05:08:15 CLSRSC-595: Executing upgrade step 4 of 19: 'GenSiteGUIDs'.
2018/08/29 05:08:16 CLSRSC-595: Executing upgrade step 5 of 19: 'UpgPrechecks'.
2018/08/29 05:08:29 CLSRSC-595: Executing upgrade step 6 of 19: 'SaveParamFile'.
2018/08/29 05:08:37 CLSRSC-595: Executing upgrade step 7 of 19: 'SetupOSD'.
2018/08/29 05:08:37 CLSRSC-595: Executing upgrade step 8 of 19: 'PreUpgrade'.
2018/08/29 05:10:01 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode
2018/08/29 05:10:01 CLSRSC-482: Running command: '/u01/app/12201/grid/bin/crsctl start rollingupgrade 18.0.0.0.0'
CRS-1131: The cluster was successfully set to rolling upgrade mode.
2018/08/29 05:10:06 CLSRSC-482: Running command: '/u01/app/180/grid/bin/asmca -silent -upgradeNodeASM -nonRolling false -tNode true -startRolling false '
ASM configuration upgraded in local node successfully.
2018/08/29 05:10:15 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode
2018/08/29 05:10:19 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2018/08/29 05:10:55 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
2018/08/29 05:10:58 CLSRSC-595: Executing upgrade step 9 of 19: 'CheckCRSConfig'.
2018/08/29 05:10:59 CLSRSC-595: Executing upgrade step 10 of 19: 'UpgradeOLR'.
2018/08/29 05:11:09 CLSRSC-595: Executing upgrade step 11 of 19: 'ConfigCHMOS'.
2018/08/29 05:11:09 CLSRSC-595: Executing upgrade step 12 of 19: 'UpgradeAFD'.
2018/08/29 05:11:16 CLSRSC-595: Executing upgrade step 13 of 19: 'createOHASD'.
2018/08/29 05:11:22 CLSRSC-595: Executing upgrade step 14 of 19: 'ConfigOHASD'.
2018/08/29 05:11:22 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2018/08/29 05:12:00 CLSRSC-595: Executing upgrade step 15 of 19: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'prodrac01'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'prodrac01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2018/08/29 05:12:44 CLSRSC-595: Executing upgrade step 16 of 19: 'InstallKA'.
2018/08/29 05:12:50 CLSRSC-595: Executing upgrade step 17 of 19: 'UpgradeCluster'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'prodrac01'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'prodrac01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'prodrac01'
CRS-2672: Attempting to start 'ora.evmd' on 'prodrac01'
CRS-2676: Start of 'ora.mdnsd' on 'prodrac01' succeeded
CRS-2676: Start of 'ora.evmd' on 'prodrac01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'prodrac01'
CRS-2676: Start of 'ora.gpnpd' on 'prodrac01' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'prodrac01'
CRS-2676: Start of 'ora.gipcd' on 'prodrac01' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'prodrac01'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'prodrac01'
CRS-2676: Start of 'ora.cssdmonitor' on 'prodrac01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'prodrac01'
CRS-2672: Attempting to start 'ora.diskmon' on 'prodrac01'
CRS-2676: Start of 'ora.diskmon' on 'prodrac01' succeeded
CRS-2676: Start of 'ora.crf' on 'prodrac01' succeeded
CRS-2676: Start of 'ora.cssd' on 'prodrac01' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'prodrac01'
CRS-2672: Attempting to start 'ora.ctssd' on 'prodrac01'
CRS-2676: Start of 'ora.ctssd' on 'prodrac01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'prodrac01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'prodrac01'
CRS-2676: Start of 'ora.asm' on 'prodrac01' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'prodrac01'
CRS-2676: Start of 'ora.storage' on 'prodrac01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'prodrac01'
CRS-2676: Start of 'ora.crsd' on 'prodrac01' succeeded
CRS-6017: Processing resource auto-start for servers: prodrac01
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'prodrac02'
CRS-2673: Attempting to stop 'ora.prodrac01.vip' on 'prodrac02'
CRS-2672: Attempting to start 'ora.ons' on 'prodrac01'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'prodrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'prodrac02'
CRS-2677: Stop of 'ora.prodrac01.vip' on 'prodrac02' succeeded
CRS-2672: Attempting to start 'ora.prodrac01.vip' on 'prodrac01'
CRS-2677: Stop of 'ora.scan1.vip' on 'prodrac02' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'prodrac01'
CRS-2676: Start of 'ora.prodrac01.vip' on 'prodrac01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'prodrac01'
CRS-2676: Start of 'ora.scan1.vip' on 'prodrac01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'prodrac01'
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'prodrac01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'prodrac01'
CRS-2676: Start of 'ora.ons' on 'prodrac01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'prodrac01' succeeded
CRS-2676: Start of 'ora.asm' on 'prodrac01' succeeded
CRS-2672: Attempting to start 'ora.RECO.dg' on 'prodrac01'
CRS-2676: Start of 'ora.RECO.dg' on 'prodrac01' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'prodrac01'
CRS-2676: Start of 'ora.DATA.dg' on 'prodrac01' succeeded
CRS-2672: Attempting to start 'ora.prbrm.db' on 'prodrac01'
CRS-2676: Start of 'ora.prbrm.db' on 'prodrac01' succeeded
CRS-6016: Resource auto-start has completed for server prodrac01
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2018/08/29 05:14:00 CLSRSC-343: Successfully started Oracle Clusterware stack
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 2.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2018/08/29 05:14:20 CLSRSC-595: Executing upgrade step 18 of 19: 'UpgradeNode'.
2018/08/29 05:14:23 CLSRSC-474: Initiating upgrade of resource types
2018/08/29 05:14:58 CLSRSC-475: Upgrade of resource types successfully initiated.
2018/08/29 05:15:10 CLSRSC-595: Executing upgrade step 19 of 19: 'PostUpgrade'.
2018/08/29 05:15:16 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@prodrac01 ~]#

Node 2 - prodrac02
[root@prodrac02 ~]# /u01/app/180/grid/rootupgrade.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/180/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/180/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/prodrac02/crsconfig/rootcrs_prodrac02_2018-08-29_05-21-05AM.log
2018/08/29 05:21:14 CLSRSC-595: Executing upgrade step 1 of 19: 'UpgradeTFA'.
2018/08/29 05:21:14 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2018/08/29 05:21:55 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2018/08/29 05:21:55 CLSRSC-595: Executing upgrade step 2 of 19: 'ValidateEnv'.
2018/08/29 05:21:56 CLSRSC-595: Executing upgrade step 3 of 19: 'GetOldConfig'.
2018/08/29 05:21:56 CLSRSC-464: Starting retrieval of the cluster configuration data
2018/08/29 05:22:19 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2018/08/29 05:22:19 CLSRSC-595: Executing upgrade step 4 of 19: 'GenSiteGUIDs'.
2018/08/29 05:22:19 CLSRSC-595: Executing upgrade step 5 of 19: 'UpgPrechecks'.
2018/08/29 05:22:21 CLSRSC-595: Executing upgrade step 6 of 19: 'SaveParamFile'.
2018/08/29 05:22:25 CLSRSC-595: Executing upgrade step 7 of 19: 'SetupOSD'.
2018/08/29 05:22:25 CLSRSC-595: Executing upgrade step 8 of 19: 'PreUpgrade'.
ASM configuration upgraded in local node successfully.
2018/08/29 05:22:37 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2018/08/29 05:23:08 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
2018/08/29 05:23:25 CLSRSC-595: Executing upgrade step 9 of 19: 'CheckCRSConfig'.
2018/08/29 05:23:25 CLSRSC-595: Executing upgrade step 10 of 19: 'UpgradeOLR'.
2018/08/29 05:23:30 CLSRSC-595: Executing upgrade step 11 of 19: 'ConfigCHMOS'.
2018/08/29 05:23:30 CLSRSC-595: Executing upgrade step 12 of 19: 'UpgradeAFD'.
2018/08/29 05:23:33 CLSRSC-595: Executing upgrade step 13 of 19: 'createOHASD'.
2018/08/29 05:23:34 CLSRSC-595: Executing upgrade step 14 of 19: 'ConfigOHASD'.
2018/08/29 05:23:34 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2018/08/29 05:24:05 CLSRSC-595: Executing upgrade step 15 of 19: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'prodrac02'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'prodrac02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2018/08/29 05:24:46 CLSRSC-595: Executing upgrade step 16 of 19: 'InstallKA'.
2018/08/29 05:24:48 CLSRSC-595: Executing upgrade step 17 of 19: 'UpgradeCluster'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'prodrac02'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'prodrac02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'prodrac02'
CRS-2672: Attempting to start 'ora.evmd' on 'prodrac02'
CRS-2676: Start of 'ora.mdnsd' on 'prodrac02' succeeded
CRS-2676: Start of 'ora.evmd' on 'prodrac02' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'prodrac02'
CRS-2676: Start of 'ora.gpnpd' on 'prodrac02' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'prodrac02'
CRS-2676: Start of 'ora.gipcd' on 'prodrac02' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'prodrac02'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'prodrac02'
CRS-2676: Start of 'ora.cssdmonitor' on 'prodrac02' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'prodrac02'
CRS-2672: Attempting to start 'ora.diskmon' on 'prodrac02'
CRS-2676: Start of 'ora.diskmon' on 'prodrac02' succeeded
CRS-2676: Start of 'ora.crf' on 'prodrac02' succeeded
CRS-2676: Start of 'ora.cssd' on 'prodrac02' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'prodrac02'
CRS-2672: Attempting to start 'ora.ctssd' on 'prodrac02'
CRS-2676: Start of 'ora.ctssd' on 'prodrac02' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'prodrac02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'prodrac02'
CRS-2676: Start of 'ora.asm' on 'prodrac02' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'prodrac02'
CRS-2676: Start of 'ora.storage' on 'prodrac02' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'prodrac02'
CRS-2676: Start of 'ora.crsd' on 'prodrac02' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: prodrac02
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'prodrac01'
CRS-2673: Attempting to stop 'ora.prodrac02.vip' on 'prodrac01'
CRS-2672: Attempting to start 'ora.ons' on 'prodrac02'
CRS-2677: Stop of 'ora.prodrac02.vip' on 'prodrac01' succeeded
CRS-2672: Attempting to start 'ora.prodrac02.vip' on 'prodrac02'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'prodrac01' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'prodrac01'
CRS-2677: Stop of 'ora.scan1.vip' on 'prodrac01' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'prodrac02'
CRS-2676: Start of 'ora.prodrac02.vip' on 'prodrac02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'prodrac02'
CRS-2676: Start of 'ora.scan1.vip' on 'prodrac02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'prodrac02'
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'prodrac02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'prodrac02'
CRS-2676: Start of 'ora.ons' on 'prodrac02' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'prodrac02' succeeded
CRS-2676: Start of 'ora.asm' on 'prodrac02' succeeded
CRS-2672: Attempting to start 'ora.RECO.dg' on 'prodrac02'
CRS-2676: Start of 'ora.RECO.dg' on 'prodrac02' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'prodrac02'
CRS-2676: Start of 'ora.DATA.dg' on 'prodrac02' succeeded
CRS-2672: Attempting to start 'ora.prbrm.db' on 'prodrac02'
CRS-2676: Start of 'ora.prbrm.db' on 'prodrac02' succeeded
CRS-6016: Resource auto-start has completed for server prodrac02
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2018/08/29 05:26:27 CLSRSC-343: Successfully started Oracle Clusterware stack
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 2.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2018/08/29 05:27:01 CLSRSC-595: Executing upgrade step 18 of 19: 'UpgradeNode'.
Start upgrade invoked..
2018/08/29 05:27:06 CLSRSC-478: Setting Oracle Clusterware active version on the last node to be upgraded
2018/08/29 05:27:06 CLSRSC-482: Running command: '/u01/app/180/grid/bin/crsctl set crs activeversion'
Started to upgrade the active version of Oracle Clusterware. This operation may take a few minutes.
Started to upgrade CSS.
CSS was successfully upgraded.
Started to upgrade Oracle ASM.
Started to upgrade CRS.
CRS was successfully upgraded.
Successfully upgraded the active version of Oracle Clusterware.
Oracle Clusterware active version was successfully set to 18.0.0.0.0.
2018/08/29 05:28:13 CLSRSC-479: Successfully set Oracle Clusterware active version
2018/08/29 05:28:13 CLSRSC-476: Finishing upgrade of resource types
2018/08/29 05:28:14 CLSRSC-477: Successfully completed upgrade of resource types
2018/08/29 05:29:39 CLSRSC-595: Executing upgrade step 19 of 19: 'PostUpgrade'.
2018/08/29 05:29:52 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@prodrac02 ~]#

No comments:
Write comments