DBA to DA

Friday, June 26, 2020

Oracle Enterprise Manager: Silent Installation Fail (SEVERE:emctl secure agent command has failed with status=1)

OEM Vineet Sachdeva June 26, 2020

In this blog I cover what to do when the OEM agent installation fails

SEVERE:emctl secure agent command has failed with status=1

Agent configuration has failed

/u01/app/oracle/product/agent/agent_13.3.0.0.0/cfgtoollogs/agentDeploy/agentDeploy_2020-06-25_23-04-01-PM.log

Waiting for agent targets to get promoted...

Promoting agent and its related targets to Management Service failed probably due to target type meta version present on the agent is not present on the OMS.Make sure that the plug-in version deployed on agents is the same as that of the version on the OMS. Also, the agent should be on the same bundle patch version as that of the OMS.For more details, check <AGENT_INST>/sysman/log/gcagent.log on agent and emoms_pbs.log on the OMS.

Fix the cause of the error and retry the operation (or) manually run the following commands on the remote host

The fix for this pretty simple follow sequence of commands as given below -

[As root]

Run root.sh

<AGENT_BASE/AGENT_VERSION/root.sh>

/u01/app/oracle/product/agent/agent_13.3.0.0.0/root.sh

[As oracle]

emctl secure agent

emctl status agent -secure

# Ensure the status is shown as secure

emctl start agent

emctl status agent

emctl config agent listtargets

#Ensure there are no targets

emctl config agent addInternalTargets

# there can be some warning messages when above command is run

emctl config agent listtargets

# Ensure that there are 2 targets listed - agent and host

Oracle Enterprise Manager: Un-install OEM Agent (Host and OMS)

OEM Vineet Sachdeva June 26, 2020

This blog discusses on steps to un-install OEM agent from the host and remove from OMS

Step 1: Uninstall OEM Agent

$<AGENT_HOME>/perl/bin/perl <AGENT_HOME>/sysman/install/AgentDeinstall.pl -agentHome <AGENT_HOME>

Example
/u01/app/oracle/product/agent/agent_13.3.0.0.0/perl/bin/perl /u01/app/oracle/product/agent/agent_13.3.0.0.0/sysman/install/AgentDeinstall.pl -agentHome /u01/app/oracle/product/agent/agent_13.3.0.0.0

Step2: Remove OEM from OMS

emcli login -username=sysman

emcli sync

emcli delete_target -name="<target name:port>" -type="oracle_emd" -delete_monitored_targets

And that is it :)

Sunday, May 17, 2020

Oracle Database (All) : Archive Log Generation (Distribution - hourly/daily/size)

19c Vineet Sachdeva May 17, 2020

In this blog I am going to cover on how to find out

1. total number of archives generated per hour & per day

2. The size of the archives MB/ hour & MB per day

SQL Code to Run

Output Sample for redo switches / hour

Redo Switches / hour /day

Redo Size / hour /day

The complete script can be download as a GIST from here

Wednesday, May 6, 2020

DEVOPS - How to find out processes in 'D' state of uninterruptible sleep (High I/O Wait)

Performance Tuning Vineet Sachdeva May 06, 2020

IN this blog I am going to write on how to find out processes state and specially when there is high I/O wait on the system.

The key is to list down the process which are in 'D' or 'S' state (primarily 'D').

Run below command to list processes in 'D' state. Replace '^D' with '^S' to get list of processes in S State.

Once you get the process you can do action on it

for x in `seq 1 1 10`; do ps -eo state,pid,cmd | grep "^D"; echo "----"; sleep 5; done

Friday, February 14, 2020

DEVOPS : Slack : How to Upload File to Slack using Python

Slack Vineet Sachdeva February 14, 2020

When uploading files to slack using Webhook - it is not possible. You need to use Slack API.
Below is sample code which can be used to put files to slack.

The key thing here is the token. You need to request a new token from Slack which you can do from Below URL -

https://api.slack.com/legacy/custom-integrations/legacy-tokens

Sample Code -

import requests
token  = "xxxxxxx"my_file = {
  'file' : ('/tmp/temp_file.out', open('/tmp/temp_file.out', 'rb'), 'txt')
}
payload={
  "filename":"temp_file.out",
  "token":token,
  "channels":['#channel-name'],
}
r = requests.post("https://slack.com/api/files.upload", params=payload, files=my_file)

Saturday, February 8, 2020

Amazon Web Services (AWS): Boto3 SQS Operations

Boto3 Vineet Sachdeva February 08, 2020

In this blog I am going to cover on how to run multiple SQS Operations using Boto3.
Note - all the response from which are printed will give HTTP Status Code 200 which signifies that the operation which you had performed had completed successfully.
Create Queue

import boto3

QName = "MyNewFIfoQueue.fifo"Attr = {'FifoQueue': 'true'}
sqs = boto3.resource('sqs')
response = sqs.create_queue (QueueName = QName, Attributes = Attr)
print (response)import boto3

QName = "MyNewFIfoQueue.fifo"Attr = {'FifoQueue': 'true'}
sqs = boto3.resource('sqs')
response = sqs.create_queue (QueueName = QName, Attributes = Attr)
print (response)

And the response

sqs.Queue(url='https://queue.amazonaws.com/<accid>/MyNewFIfoQueue.fifo')

You can verify the same from Console or AWS CLI

$ aws sqs list-queues
{
    "QueueUrls": [
        "https://queue.amazonaws.com/<accid>/MyNewFIfoQueue.fifo"    ]
}

Send Messages to Queue

import boto3
import uuid

QName = "MyNewFIfoQueue.fifo"
sqs = boto3.client('sqs')
QUEUE_URL = sqs.get_queue_url(QueueName = QName)['QueueUrl']
response = sqs.send_message(
                           QueueUrl=QUEUE_URL,
                           MessageBody="TestMessage",
                           MessageGroupId="TestGroup",
                           MessageDeduplicationId=str(uuid.uuid4()))
print(response)


And the response
{u'MD5OfMessageBody': '08ff08d3b2981eb6c611a385ffa4f865', 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': 'b18d5e81-a666-541b-aaa4-0839850e0a6e', 'HTTPHeaders': {'x-amzn-requestid': 'b18d5e81-a666-541b-aaa4-0839850e0a6e', 'date': 'Sat, 08 Feb 2020 02:47:04 GMT', 'content-length': '431', 'content-type': 'text/xml'}}, u'SequenceNumber': '18851513359964655616', u'MessageId': '3a9b64ad-4eb3-4606-9eec-c9c5369d4304'}






Receive messages From Queue




import boto3
QName = "MyNewFIfoQueue.fifo"
sqs = boto3.client('sqs')
QUEUE_URL = sqs.get_queue_url(QueueName = QName)['QueueUrl']
response = sqs.receive_message(
                           QueueUrl=QUEUE_URL,
                           MaxNumberOfMessages=1                           )['Messages'][0]
print(response)


And the response 
{u'Body': 'TestMessage', u'ReceiptHandle': 'AQEBmnx4wSlS0Qf/kramgeBUr8lMEHrWkILeK3SIoxMjfnMjRGrXtm8w8BUXiiKJQSaFYaGYnJF6kpFrYeFPoGlrVcJgn6Ci3WpM+pVm1Ih0XT4SkHQBjH2CIxKfx21t+oyej7mYi3PwNENOHJI125BNuAVnfSAys64uBFPXgEPgRy/OFBVK2CcueJy18I8sPm6dNV5CCzxfzZE3csd/TBOQsnhtAPt3sro3MfZUUUc5d3iIrhGjVa/xNXiNNHECMu5ZifCTU8U1pX2lX1EwV3CYzrlnr2mie/R6SkJqEvPjsfc=', u'MD5OfBody': '08ff08d3b2981eb6c611a385ffa4f865', u'MessageId': '3a9b64ad-4eb3-4606-9eec-c9c5369d4304'}






Purge and Delete Queue




import boto3

QName = "MyNewFIfoQueue.fifo"
sqs = boto3.client('sqs')
QUEUE_URL = sqs.get_queue_url(QueueName = QName)['QueueUrl']
response = sqs.purge_queue(QueueUrl=QUEUE_URL)
print(response)
response = sqs.delete_queue(QueueUrl=QUEUE_URL)
print(response)

And the response

{'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': '364d9e6a-9b1b-555e-b5af-c8d2bd4abe6a', 'HTTPHeaders': {'x-amzn-requestid': '364d9e6a-9b1b-555e-b5af-c8d2bd4abe6a', 'date': 'Sat, 08 Feb 2020 02:59:29 GMT', 'content-length': '209', 'content-type': 'text/xml'}}}
{'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': '75808c73-aac5-5dec-b3a4-dacb8c9446d8', 'HTTPHeaders': {'x-amzn-requestid': '75808c73-aac5-5dec-b3a4-dacb8c9446d8', 'date': 'Sat, 08 Feb 2020 02:59:29 GMT', 'content-length': '211', 'content-type': 'text/xml'}}}

Friday, February 7, 2020

Amazon Web Services (AWS): Installing and Configuring AWSCLI

AWSCLI Vineet Sachdeva February 07, 2020

In this short blog, I am going to cover on quickly how to configure awscli.

Step 1 : Install awscli
pip3.7 install awscli --upgrade --user
(if you have different verison of pip like pip3, pip2.7) you can use that as well

Step 2: Configure API Access
The first thing is that you should know your API keys, to do that you can use the URL.

Step 3: Conifgure CLI
Add awscli to your Path.
For MAC it is generally :
export PATH=$PATH:/Users/<username>/Library/Python/3.7/bin

Then do configure , keep your key and secret access key handy.

aws configure

AWS Access Key ID []: ****************rMFO

AWS Secret Access Key [****************rE0P]: f****************rUWbzxpt

Default region name []: us-east-1

Default output format []: json

Look at the files ~/.aws/credentials and configure.

You will find what you had entered configured in these files.

So, yo u can change anytime you want from these files as well.

Thursday, January 30, 2020

Amazon Web Services (AWS): Enabling API Access

Boto3 Vineet Sachdeva January 30, 2020

The First step of enabling API access is to get your AWS Access keys properly created and defined.

Step 1 - Login to AWS console and Navigate to IAM

Step 2 - Navigate to users and select user which you want to have the access key

Step 3 - Go to Security Credentials and Click on Create Access Key.

Remember, you will get a pop-up and this is the only time you will able to see the Secret Access Key, so it is a good choice to download and save it somewhere and probably somewhere safe as it gives access to your account.
However, if you loose it, contact your admin and the admin can delete it or make it inactive

Okay, Now that you have 2 parts of Key
Access Key and Secret Access Key.

You should now configure your environment
In your home directory create a folder named .aws and create the credentials file as below.

cat ~/.aws/credentials
[default]
aws_access_key_id=xxxxxxxx
aws_secret_access_key=xxxxx

Now this will be used as your default credentials.

You should also create a config file which specifies default region
Example Below
cat ~.aws/config
[default]

region=us-east-1

The other option can be to set it as your environment variables or set it as your environment variables in the Code Editor you use.
I use pycharm, so go to Run and Edit Configurations, you will see environment variables there which you can configure.

Also, you can use boto3 quick start for your help as well.

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html

Amazon Web Services (AWS) : EC2 - Instance Metadata Get Region/Type/IP

EC2 Vineet Sachdeva January 30, 2020

In this blog I am going to demonstrate how to get the Instance Metadata information using curl.

Instance Metadata information is available at the 169.254.169.254 IP address.
If you are from networking background you will realize, this is Automatic Private IP or Self Assigned Private IP.

So, we use curl to get the information and as an example below.

$ curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | grep -e Type -e region -e Ip

"instanceType" : "m5.2xlarge",

"privateIp" : "10.114.32.79",

"region" : "us-east-1",

Wednesday, January 15, 2020

Amazon Web Services (AWS) : Customising the Sign-In Link for IAM Users

IAM Vineet Sachdeva January 15, 2020

When you start creating IAM users, the first AWS does automatically for you is to create an IAM sign in link which could be given to your users.

You can click on customize to customize it to your custom name and then provide it to your users in order to sign in directly to your account.

Amazon Web Services (AWS) : Setting up Security / IAM Best Practices at account creation

IAM Vineet Sachdeva January 15, 2020

In this blog I am going to show, how to have all the green ticks as per AWS recommendation, after setting up a new account.

Once you create a new account (most probably free tier), go to IAM and then if you are not already go to Dashboard.

You should see your status as below.
The next step is to activate an MFA on your account. MFA is multi-factor authentication which provides additional layer of security to login to your account.
The most easy (per me) downloa Duo Mobile and add your account.
You will be asked to enter 2 continous codes and scan the QR Code.

The next step is to add a password policy to the user account.

The next Step is to create a group and a user.
So, click on Add Group and name : AWSAdmins.
Attach the policy as shown below - 'Administrator Access'

Review and Create
Next Create a user and attach this policy to the user, username can be any what you want and password you can set it at your ease.

Now if you go back to the dashboard, you will see - all your boxes are 'green ticked'

Wednesday, January 8, 2020

OEM 13cR3: OMS Plugin Patching 13.3.1.0.191231 and 13.3.2.0.191231

plugin Vineet Sachdeva January 08, 2020

In this blog I am going to patch EM plugins on OMS for the given version
Note - you should always keep a backup of EM configuration and have a restore point / db backup to restore to in case there are any issues in application of the patch.

Download Patches

$ wget 'https://updates.oracle.com/Orion/Services/download/p30666063_133000_Generic.zip?aru=23270565&patch_file=p30666063_133000_Generic.zip' --http-user='<username>' --http-password=<password> --no-check-certificate --output-document=p30666063_133000_Generic.zip
$ wget 'https://updates.oracle.com/Orion/Services/download/p30666123_133000_Generic.zip?aru=23270566&patch_file=p30666123_133000_Generic.zip' --http-user='<MoS Username>' --http-password='<password>' --no-check-certificate --output-document=p30666123_133000_Generic.zip

Here are few things you must ensure -
1. Ensure that the Software Library is configured.
2. Ensure that the Oracle WebLogic Administration server that hosts the OMS is up and running.
3. Ensure that the Oracle Database, which houses the Management Repository and its listener are up and running.
4. Ensure that the Oracle Management Service (OMS) on which you are installing the patch or from which you are rolling back the patch is Oracle Management Service 13c Release 3 (13.3.0.0.0).
5. Ensure that you have the latest version of OPatch 13.9.0.0.0 and OMSPatcher 13.8.0.0.3 on all OMS instance platform homes.

6. Ensure that you set the ORACLE_HOME environment variable to OMS core home.

You must have applied latest patch and done all steps as in my last blog.

Patch apply 13.3.1.0.191231 and 13.3.2.0.19123
Unzip Patch (2 minutes)
$ unzip -qq p30666063_133000_Generic.zip
$ export ORACLE_HOME=/u01/app/oracle/product/oem13cr3

$ export PATH=$ORACLE_HOME/OMSPatcher:$ORACLE_HOME/OPatch:$ORACLE_HOME/bin:$PATH

Pre Patch (5 minutes)
$  cd 30666063
$ emctl stop oms
$ omspatcher apply -analyze

Patch (10 minutes)
$ omspatcher apply

Pre Patch (5 minutes)
$  unzip -qq p30666123_133000_Generic.zip
$  cd 30666123
$ omspatcher apply -analyze

Patch (10 minutes)
$ omspatcher apply

Start OMS
$ emctl start oms
$ emcli login -username=sysman
$ emcli sync

After Patch (13.3.1.0.191231) you can see this in output of opatch lspatches

30563546;EM FMW Plugin Bundle Patch 13.3.1.0.191231

30563523;EM Exadata Plugin Bundle Patch 13.3.1.0.191231

30563511;EM DB Plugin Bundle Patch 13.3.1.0.191231

29404834;EM SI Plugin Bundle Patch 13.3.1.0.190331

30203475;EMBP Patch Set Update 13.3.0.0.191015

After Patch (13.3.2.0.191231) you can see this in output of opatch lspatches
30563518;EM DB Plugin Bundle Patch 13.3.2.0.191231
30341229;EM SI Plugin Bundle Patch 13.3.2.0.191031
30563546;EM FMW Plugin Bundle Patch 13.3.1.0.191231
30563523;EM Exadata Plugin Bundle Patch 13.3.1.0.191231
30203475;EMBP Patch Set Update 13.3.0.0.191015

Tuesday, January 7, 2020

AWS : Lambda: Add IP to Security Group Using Boto3 - Complete Code

Boto3 Vineet Sachdeva January 07, 2020

IN this blog I am going to show how to add an IP using Lambda.

You can create a sample-SQS trigger event with an IP address in body to create an SQS event emulation for testing..

Now,


lamda_handler is the default handler for lambda
It checks if the IP is not already part of the rule
it then calls updateIP and refreshes the timestamp if yes or adds new with new timestamp if no
the revoke is to temporarily revoke and add the IP.

import boto3
from datetime import datetime

ec2 = boto3.resource('ec2')
s_group = ec2.SecurityGroup('sg-85d42ac2')
dt = datetime.now()
date_format = "%m-%d-%Y %H:%M"str_dt = dt.strftime(date_format)
ssh_port = 22code = 200max_minutes = 5

def lambda_handler(event, context):
    for record in event['Records']:
        ip = record["body"]
        if (str(ip) == 'sweep'):
            sweepIP()
        else:
            verifyAddIP(str(ip))

def verifyAddIP(strIP):
    m_strIP = strIP + '/32'
    ip_permission = s_group.ip_permissions[0]
    ip_range = ip_permission['IpRanges']

    for cidr in ip_range:
        if (cidr['CidrIp'] == m_strIP):
            updateRule(strIP + '/32', True)
        else:
            updateRule(strIP + '/32', False)


def updateRule(strIP, update_p):
    if update_p:
        response = s_group.revoke_ingress(IpProtocol="tcp", CidrIp=strIP, FromPort=ssh_port, ToPort=ssh_port)
        response = s_group.authorize_ingress(IpPermissions=[
            {'IpProtocol': 'tcp',
             'FromPort': ssh_port,
             'ToPort': ssh_port,
             'IpRanges': [{'CidrIp': strIP, 'Description': str_dt}]
             }
        ]
        )
        print ('Update IP Address Time in Ingress Rule - ' + strIP)
    else:
        response = s_group.authorize_ingress(IpPermissions=[
            {'IpProtocol': 'tcp',
             'FromPort': ssh_port,
             'ToPort': ssh_port,
             'IpRanges': [{'CidrIp': strIP, 'Description': str_dt}]
             }
        ]
        )
        print ('Added IP Address to Ingress Rule - ' + strIP)

Friday, January 3, 2020

OEM 13cR3: Patching 13.3.0.0.191015

Patch Vineet Sachdeva January 03, 2020

In this blog I am going to take up the part of Patching my OEM 13cR3 Setup.

Download Patches below
1. Patch 6880880: OPatch version 13.9.3.3.0 for EMCC 13.x customers Only (not for FMW 12c)
2 Patch 19999993: OMSPatcher patch of version 13.8.0.0.3 for Enterprise Manager Cloud Control 13.3.0.0.0

3. Patch 30203475: EMBP Patch Set Update 13.3.0.0.191015

First two are Opatch and Omspatcher and the last is the system patch

Upgrade OPatch (2 minutes)

$ export PATH=$ORACLE_HOME/OMSPatcher:$ORACLE_HOME/OPatch:$ORACLE_HOME/bin:$PATH

$ unzip -qq unzip -qq p6880880_139000_Generic.zip

$ cd 6880880

$ ${ORACLE_HOME}/oracle_common/jdk/bin/java -jar opatch_generic.jar -silent oracle_home=${ORACLE_HOME}

Upgrade OMS Patcher (1 minute)

$ cd $ORACLE_HOME

$ mv OMSPatcher OMSPatcher.03_Jan_2020

$ unzip -qq /u01/OEM/Patch/p19999993_133000_Generic.zip

$ cd OMSPatcher

$ ./omspatcher version

Analyze the Patch (5 minutes)

$ unzip -qq p30203475_133000_Generic.zip

$ cd 30203475

$ omspatcher apply -analyze

If you face issue here - then you might be hitting any one of the below. (mostly the second one)

EM 13c: Enterprise Manager 13c Cloud Control OMSPatcher Patch Analyze Error: OMSPatcher finds that it is not able to connect to OMS repository (Doc ID 2306317.1)

EM 13c: Applying a Patch to Enterprise Manager 13.3 Cloud Control OMS Fails with error code 235, OMSPatcher finds that it is not able to connect to OMS repository (Doc ID 2519247.1)

Patch the system (10 minutes)

$ emctl stop oms

$ omspatcher apply

Verify

$ opatch lspatches

Restart OMS
$ emctl start oms