Oracle Exadata: AIDE, Advanced Support Gateway and Exadata Image 19

Oracle Exadata: AIDE, Advanced Support Gateway and Exadata Image 19

Exadata Vineet Sachdeva   September 05, 2019

Oracle has released Image 19 and AIDE, however what they really missed is the platinum / advanced support gateway

the advanced support gateway is present as user orarom in directory /opt/OracleHomes

Now if you are getting daily alerts from AIDE complaining a lot of files have changed. 
Then look into the log file /var/log/aide/aide.log, you will find that the log files of agent are changed which are continously throwing the alerts 

What you need to do is add one line in the aide.conf file which is highlighted as below and then run exadataAIDE -u. 

This will ignore the directory and ensure no alerts are been sent. 


less aide.conf | grep opt
!/opt/OracleHomes

/opt/    CONTENT

Oracle Exadata: chrony Daemon (release 19.1)

Oracle Exadata: chrony Daemon (release 19.1)

Exadata Vineet Sachdeva   August 27, 2019

Release 19.1 of Exadata has taken a leap in using chrony daemon instead of age old written NTP daemon and protocol.

You will not find any NTP daemon installed once you upgrade to this release and note in this release you are at Linux 7. 

To check status of chrony daemon here are few commands which will help. 

1. Check Sources

# chronyc sources

210 Number of sources = 3
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? aa-bbb-ccc-1100.dddddddd>     3   4   377     3  -2328ms[-2328ms] +/- 8332ms
^? bb-bbb-ccc-1100.dddddddd>     3   4   377     4  -2358ms[-2358ms] +/- 8681ms

^? cc-bbb-ccc-1100.dddddddd>     3   4   377     7  -2321ms[-2321ms] +/- 8696ms

2. Check Source stats

# chronyc sourcestats

210 Number of sources = 3

Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev

==============================================================================

aa-bbb-ccc-1100.dddddddd>   6   3    81   +951.004   4830.397  -2334ms    31ms

bb-bbb-ccc-1100.dddddddd>   9   4   129   -438.641    216.568  -2372ms  4677us

cc-bbb-ccc-1100.dddddddd>  26  11   421   -381.175     20.365  -2332ms  3727us

3. Check sync

# chronyc tracking

Reference ID    : 0A901009 (cc-bbb-ccc-1100.dddddddd)

Stratum         : 4

Ref time (UTC)  : Mon Aug 26 03:19:47 2019

System time     : 0.000000304 seconds fast of NTP time

Last offset     : -0.000190595 seconds

RMS offset      : 0.012864766 seconds

Frequency       : 84.308 ppm slow

Residual freq   : -0.004 ppm

Skew            : 0.720 ppm

Root delay      : 0.140609756 seconds

Root dispersion : 1.491145134 seconds

Update interval : 16.1 seconds

Leap status     : Normal

You can find more details on chrony commands here 

https://www.thegeekdiary.com/centos-rhel-7-configuring-ntp-using-chrony/

Oracle Exadata: How to check if Flash Cache is Write Back or Write Through

Oracle Exadata: How to check if Flash Cache is Write Back or Write Through

Exadata Vineet Sachdeva   August 21, 2019

This is one of the questions which newbee's into exadata ask me quite often that how to know if flashcache is Write Back or Write Through.

Well the answer is pretty simple and this is how you check it. 

Log into cell server on which you want to know and run below 

# cellcli -e list cell attributes flashcachemode

         WriteBack

Or if you want to see for all cells you can use dcli 

# dcli -l root -g cell_group ' cellcli -e list cell attributes flashcachemode'

myexaexcel01: WriteBack

myexaexcel02: WriteBack

myexaexcel03: WriteBack

myexaexcel04: WriteBack

myexaexcel05: WriteBack

myexaexcel06: WriteBack

myexaexcel07: WriteBack

myexaexcel08: WriteBack

myexaexcel09: WriteBack

myexaexcel10: WriteBack

myexaexcel11: WriteBack

myexaexcel12: WriteBack

myexaexcel13: WriteBack

myexaexcel14: WriteBack

myexaexcel15: WriteBack

Oracle Exadata: How I removed specific Alerts (Diagnostic Package cannot be uploaded to ASR server)

Oracle Exadata: How I removed specific Alerts (Diagnostic Package cannot be uploaded to ASR server)

Exadata Vineet Sachdeva   August 20, 2019

On a weekly basis, I was getting emails (after patching to release 18) that Diagnostic Package. 

I then checked with Oracle about this and found out this is a new feature (12 release) and it is okay to ignore. But I was really bugged by the nuisance emails. So I thought of clearing it.

But I have more than 5 Exadatas which are full racks and I do not want to clear all alerts but these itself (for Diagnostic Package)

So, I came up with a small automation which will generate the right commands for me.

Here is what I did 

For Cell Servers 
# dcli -l root -g cell_group 'cellcli -e list alerthistory' |  grep 'Diagnostic packages' | awk '{print "dcli -l root -c " $1 " \x27" "cellcli -e drop alerthistory "  $2 "\x27" "   ### " $3}' | sed 's/://'

dcli -l root -c myexacel01 'cellcli -e drop alerthistory 4_1'   ### 2018-10-06T19:16:52-04:00
dcli -l root -c myexacel02 'cellcli -e drop alerthistory 4_1'   ### 2018-10-06T19:19:15-04:00
dcli -l root -c myexacel03 'cellcli -e drop alerthistory 5_1'   ### 2018-10-06T19:18:51-04:00
dcli -l root -c myexacel04 'cellcli -e drop alerthistory 4_1'   ### 2018-10-06T19:17:49-04:00
dcli -l root -c myexacel05 'cellcli -e drop alerthistory 4_1'   ### 2018-10-06T19:18:55-04:00
dcli -l root -c myexacel06 'cellcli -e drop alerthistory 5_1'   ### 2018-10-06T19:18:46-04:00
dcli -l root -c myexacel07 'cellcli -e drop alerthistory 4_1'   ### 2018-10-06T19:18:13-04:00
dcli -l root -c myexacel08 'cellcli -e drop alerthistory 4_1'   ### 2018-10-06T19:19:01-04:00
dcli -l root -c myexacel09 'cellcli -e drop alerthistory 4_1'   ### 2018-10-06T19:18:36-04:00
dcli -l root -c myexacel10 'cellcli -e drop alerthistory 4_1'   ### 2018-10-06T19:19:26-04:00
dcli -l root -c myexacel11 'cellcli -e drop alerthistory 4_1'   ### 2018-10-06T19:18:16-04:00
dcli -l root -c myexacel12 'cellcli -e drop alerthistory 5_1'   ### 2018-10-06T19:18:11-04:00
dcli -l root -c myexacel13 'cellcli -e drop alerthistory 4_1'   ### 2018-10-06T19:17:53-04:00

dcli -l root -c myexacel14 'cellcli -e drop alerthistory 4_1'   ### 2018-10-06T19:18:03-04:00

For DB Servers 


# dcli -l root -g dbs_group 'dbmcli -e list alerthistory' |  grep 'Diagnostic packages' | awk '{print "dcli -l root -c " $1 " \x27" "dbmcli -e drop alerthistory "  $2 "\x27" "   ### " $3}' | sed 's/://'

dcli -l root -c mydbnode01 'dbmcli -e drop alerthistory 7_1'   ### 2019-08-14T08:06:48-04:00

dcli -l root -c mydbnode02 'dbmcli -e drop alerthistory 7_1'   ### 2019-08-14T08:06:42-04:00

dcli -l root -c mydbnode03 'dbmcli -e drop alerthistory 7_1'   ### 2019-08-14T08:06:50-04:00

dcli -l root -c mydbnode04 'dbmcli -e drop alerthistory 7_1'   ### 2019-08-14T08:06:38-04:00

dcli -l root -c mydbnode05 'dbmcli -e drop alerthistory 7_1'   ### 2019-08-14T08:06:43-04:00

Oracle Exadata: DBM-02643: DROP ALERTHISTORY command did not include all members of the alert sequence for *_*

Oracle Exadata: DBM-02643: DROP ALERTHISTORY command did not include all members of the alert sequence for *_*

Exadata Vineet Sachdeva   August 16, 2019

In this blog I am going to discuss an error message which you might get when dropping individual alerts via dbmcli or cellcli in Exadata

if you try to drop your alert using a command like below 
drop alerthistory  2_1, 2_2, 2_3, 2_4, 2_5, 2_6, 2_7, 2_8, 2_9, 2_10, 2_11, 2_12, 2_13

and you get an error message which is something like 

DBM-02643: DROP ALERTHISTORY command did not include all members of the alert sequence for 2_1. All members of the sequence must be dropped together

That means you have not selected all the 2_* members, so, in order to proceed put all the 2_* members or the sequence you are trying to drop. 

This will clear the alert.

Oracle Exadata: How to find Serial # (to Put in Service Request)

Oracle Exadata: How to find Serial # (to Put in Service Request)

Exadata Vineet Sachdeva   August 08, 2019

In this blog I am going to talk about, how to find out the serial number of Exadata. 
it is a single command, which can be run from any node (compute or storage) using root user , the output will give you the serial number


# ipmitool sunoem cli 'show System' | grep -i Serial

        serial_number = AK00418863

You can put this serial number when raising SR for an engineering system. (Assuming the CSI is already added into your Support account)

Oracle Exadata : AIDE (Advanced Intrusion Detection System)

Oracle Exadata : AIDE (Advanced Intrusion Detection System)

Exadata Vineet Sachdeva   August 07, 2019

AIDE (Advanced Intrusion Detection System) is basically a utility which keeps the database of all file system files on the server and  alerts you via e-mail (as configured) what files and their content have changed. 
(AIDE was released with Image release 19 of Exadata)

This can help in detecting any security threats. 

It is a daily cron job which runs, scans and alerts

To know if it is enabled 
$  /opt/oracle.SupportTools/exadataAIDE -status

AIDE: daily cron is currently enabled.

If you are getting alerts and have changed the FS and are happy with the log in '/var/log/aide/aide.log'
then you can update the AIDE database which will ensure alerts are not sent to you.

$  /opt/oracle.SupportTools/exadataAIDE -update

Oracle Exadata / ODA : How to find which version of engineered system you are running

Oracle Exadata / ODA : How to find which version of engineered system you are running

ODA Vineet Sachdeva   August 06, 2019

A common question which new comers ask me is how to find out which version of  Engineered system are we running.


The answer is easy and straight forward. Just one command run as root on compute node or Storage Node (As root)

# ODA Example  $  ipmitool sunoem cli 'show System' | grep model
         model = ODA X7-2M
        component_model = ORACLE SERVER X7-2

# Exadata Example  $  ipmitool sunoem cli 'show System' | grep model
        model = Exadata X7-8
        component_model = ORACLE SERVER X7-8

Oracle Database - Exadata and Exadata Cloud - Important MoS Doc ID's

Oracle Database - Exadata and Exadata Cloud - Important MoS Doc ID's

OCI Vineet Sachdeva   December 18, 2018

In this blog  I will write on all the important links for Exadata which will be useful to you. 


This are very helpful links, which can help you keep updated with Exadata DB Machine in premise or if you are servicing one for a customer. 

So keep them handy.. :)

1. Exadata Database Machine and Exadata Storage Server Supported Versions (Doc ID 888828.1)
2. Engineered Systems Resource Center - Doc ID 1392174.1 
3. Information Center: Oracle Exadata Database Machine - Doc ID 1306791.1  
4. Information Center: Using Oracle Exadata Database Machine - Doc ID 1460198.2 
5. Information Center: Upgrading Oracle Exadata Database Machine - Doc ID 1364356.2
6. Exadata System Software Certification - Doc ID 2075007.1 
7. Exadata Storage Software Versions Supported by the Oracle Enterprise Manager Exadata Plug-in - Doc ID 1626579.1 
8. Exadata Software and Hardware Support Lifecycle- Doc ID 1570460.1 
9. Oracle Exadata Best Practices – Doc ID 757552.1 
10. Oracle Exadata Database Machine Setup/Configuration Best Practices - Doc ID 1274318.1
11. Exadata Critial Issues - Doc ID 1270094.1
12.  How To Collect Diagpack Diagnostic Package In Exadata - Doc ID 2226173.1
13. Oracle Exadata Database Machine exachk or HealthCheck – Doc ID 1070954.1
    
    Exadata Cloud
14. Information Center: Oracle Database Exadata Cloud - Doc ID 2334729.2
15. Information Center: Patching and Maintaining Oracle Database Exadata Cloud - Doc ID 2334779.2
16. Known Issues for Oracle Database Exadata Cloud Service - Doc ID 2249093.1
17. Exadata Cloud Service Software Versions - Doc ID 2333222.1
18. Known Issues for Oracle Database Exadata Cloud Machine - Doc ID 2252305.1
19. Technology Cloud Services (PaaS and IaaS) Maintenance Schedule - Doc ID 2131053.2

Exadata - ExaWatcher configuration

Exadata - ExaWatcher configuration

Exadata Anonymous   November 03, 2015

Exawatcher is the elder brother or say sibling to OSWatcher on Exadata. It has additional diagnostics options for RDS logs/diags which are specific to Exadata only.

In this blog we see how to change the destination and the total available size for Exawatcher Logs.

I can wrap it up in 3 steps
1. Change the config
2. Stop Exawatcher
3. Start Exawatcher

Pretty neat !!

Now let's do it 

You cd to opt/oracle.ExaWatcher/archive and edit the file ExaWatcher.conf

Look for the below section precisely (it is the first section generally)
.
.
.
<ResultDir> /opt/oracle.ExaWatcher/archive
<ZipOption> bzip2
<SpaceLimit> 6047
.
.
Parameters we are looking for are ResultDir and Space Limit. 

Above is the default configuration (for X5 ODA, generally is similar for all versions of Exa Hardware)

Next we edit this to our own mount point and custom size so that it looks something like 

[root@Myexadata01 oracle.ExaWatcher]# cat ExaWatcher.conf | grep -E 'Result|Space' | tail -2
<ResultDir>  /mymount/Myexadata01
<SpaceLimit> 40960

What I have done here is redirected the output to a NAS device and then increased the space available. With all defaults 40960 should be good for around 40-45 days and Exawatcher will not clean it

On a side note cleaning of the logs is done by exawatcher script only, on the logic that it wipes clean data if mount on which it is writing is 80% or more utilized.

Okay, coming back now all we need to do is stop and start. 

We are still in the /opt/oracle.ExaWatcher directory

[root@Myexadata01 oracle.ExaWatcher]# ./ExaWatcher.sh --stop

 41500 ?        SN     0:00 /bin/bash ./ExaWatcher.sh --fromconf

[INFO     ] ExaWatcher successfully terminated!

so --stop does the stop, but starting is bit different, Below is how to start

[root@Myexadata01 oracle.ExaWatcher]# /opt/oracle.cellos/validations/bin/vldrun.pl -script oswatcher

Logging started to /var/log/cellos/validations.log

Command line is /opt/oracle.cellos/validations/bin/vldrun.pl -script oswatcher

Run validation oswatcher - PASSED

Wait for a a minute or so and Exawatcher will come up, you can see check out the status using ps utility. I have taken out one line to highlight the difference.

ps -ef | grep -i exaw

sh -c /usr/bin/vmstat  5  2 >> /mymount/Myexadata01/Vmstat.ExaWatcher/2015_11_03_04_14_00_VmstatExaWatcher_Myexadata01.mydomain

Exadata - Two Seldom used attributes

Exadata - Two Seldom used attributes

Exadata Anonymous   October 07, 2015

Exadata is a one of the most powerful product from Oracle, probably the most powerful engineered system available for consumer business on the planet.

Today I talk about few features which I have seen Administrators missing or not using them a lot.

Appliance mode 

appliance.mode is a disk group attribute which improves the disk rebalancing times so redundancy is restored much faster after a disk drop operation or post failure

DMA who have upgraded the release 11.2.3.3.0 (yes it is available from this release) can use this attribute. This needs to be set at the ASM level and can be set using below SQL statement

ALTER DISKGROUP <disk_group> SET ATTRIBUTE 'appliance.mode' = ['TRUE'|'FALSE'];

As with all the Oracle Features there are restrictions when setting this 2 key ones are – compatible.asm is set to 11.2.0.4 or higher on the disk group and cell.smart_scan_capable is set to True.

Content Type

Oracle introduced with 11.2.0.3 a new parameter content.type for ASM which takes 3 different values data, recovery and system. Each type of setting modified the adjacency measure used by secondary extent placement algorithm.

Using these settings, the likelihood of double failure  (in Normal redundancy DG) causing Data loss reduced.

Having this attribute in effect would require rebalancing, so it would be good to rebalance in a quite time.

Alter diskgroup data set attribute 'content.type'='data';

Alter diskgroup data rebalance power <power>;