Pages

Advertisement

Showing posts with label AWS. Show all posts
Showing posts with label AWS. Show all posts

Saturday, February 8, 2020

Amazon Web Services (AWS): Boto3 SQS Operations

 
In this blog I am going to cover on how to run multiple SQS Operations using Boto3. 
Note - all the response from which are printed will give HTTP Status Code 200 which signifies that the operation which you had performed had completed successfully.
Create Queue

 
import boto3

QName = "MyNewFIfoQueue.fifo"Attr = {'FifoQueue': 'true'}
sqs = boto3.resource('sqs')
response = sqs.create_queue (QueueName = QName, Attributes = Attr)
print (response)import boto3
QName = "MyNewFIfoQueue.fifo"Attr = {'FifoQueue': 'true'}
sqs = boto3.resource('sqs')
response = sqs.create_queue (QueueName = QName, Attributes = Attr)
print (response)





And the response 




sqs.Queue(url='https://queue.amazonaws.com/<accid>/MyNewFIfoQueue.fifo')








You can verify the same from Console or AWS CLI








$ aws sqs list-queues
{
    "QueueUrls": [
        "https://queue.amazonaws.com/<accid>/MyNewFIfoQueue.fifo"    ]
}










Send Messages to Queue






import boto3
import uuid

QName = "MyNewFIfoQueue.fifo"
sqs = boto3.client('sqs')
QUEUE_URL = sqs.get_queue_url(QueueName = QName)['QueueUrl']
response = sqs.send_message(
                           QueueUrl=QUEUE_URL,
                           MessageBody="TestMessage",
                           MessageGroupId="TestGroup",
                           MessageDeduplicationId=str(uuid.uuid4()))
print(response)





And the response


{u'MD5OfMessageBody': '08ff08d3b2981eb6c611a385ffa4f865', 'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': 'b18d5e81-a666-541b-aaa4-0839850e0a6e', 'HTTPHeaders': {'x-amzn-requestid': 'b18d5e81-a666-541b-aaa4-0839850e0a6e', 'date': 'Sat, 08 Feb 2020 02:47:04 GMT', 'content-length': '431', 'content-type': 'text/xml'}}, u'SequenceNumber': '18851513359964655616', u'MessageId': '3a9b64ad-4eb3-4606-9eec-c9c5369d4304'}












Receive messages From Queue









import boto3


QName = "MyNewFIfoQueue.fifo"
sqs = boto3.client('sqs')
QUEUE_URL = sqs.get_queue_url(QueueName = QName)['QueueUrl']
response = sqs.receive_message(
                           QueueUrl=QUEUE_URL,
                           MaxNumberOfMessages=1                           )['Messages'][0]
print(response)





And the response 


{u'Body': 'TestMessage', u'ReceiptHandle': 'AQEBmnx4wSlS0Qf/kramgeBUr8lMEHrWkILeK3SIoxMjfnMjRGrXtm8w8BUXiiKJQSaFYaGYnJF6kpFrYeFPoGlrVcJgn6Ci3WpM+pVm1Ih0XT4SkHQBjH2CIxKfx21t+oyej7mYi3PwNENOHJI125BNuAVnfSAys64uBFPXgEPgRy/OFBVK2CcueJy18I8sPm6dNV5CCzxfzZE3csd/TBOQsnhtAPt3sro3MfZUUUc5d3iIrhGjVa/xNXiNNHECMu5ZifCTU8U1pX2lX1EwV3CYzrlnr2mie/R6SkJqEvPjsfc=', u'MD5OfBody': '08ff08d3b2981eb6c611a385ffa4f865', u'MessageId': '3a9b64ad-4eb3-4606-9eec-c9c5369d4304'}












Purge and Delete Queue








import boto3

QName = "MyNewFIfoQueue.fifo"
sqs = boto3.client('sqs')
QUEUE_URL = sqs.get_queue_url(QueueName = QName)['QueueUrl']
response = sqs.purge_queue(QueueUrl=QUEUE_URL)
print(response)
response = sqs.delete_queue(QueueUrl=QUEUE_URL)
print(response)




And the response



{'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': '364d9e6a-9b1b-555e-b5af-c8d2bd4abe6a', 'HTTPHeaders': {'x-amzn-requestid': '364d9e6a-9b1b-555e-b5af-c8d2bd4abe6a', 'date': 'Sat, 08 Feb 2020 02:59:29 GMT', 'content-length': '209', 'content-type': 'text/xml'}}}
{'ResponseMetadata': {'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': '75808c73-aac5-5dec-b3a4-dacb8c9446d8', 'HTTPHeaders': {'x-amzn-requestid': '75808c73-aac5-5dec-b3a4-dacb8c9446d8', 'date': 'Sat, 08 Feb 2020 02:59:29 GMT', 'content-length': '211', 'content-type': 'text/xml'}}}



















          

        

          

        

Friday, February 7, 2020


          

        




















Amazon Web Services (AWS): Installing and Configuring AWSCLI













 
      







In this short blog, I am going to cover on quickly how to configure awscli.



Step 1 : Install awscli

pip3.7 install awscli --upgrade --user

(if you have different verison of pip like pip3, pip2.7) you can use that as well





Step 2: Configure API Access

The first thing is that you should know your API keys, to do that you can use the URL. 



Step 3: Conifgure CLI

Add awscli to your Path.

For MAC it is generally : 

export PATH=$PATH:/Users/<username>/Library/Python/3.7/bin



Then do configure , keep your key and secret access key handy.








aws configure 



AWS Access Key ID []: ****************rMFO





AWS Secret Access Key [****************rE0P]: f****************rUWbzxpt



Default region name []: us-east-1



Default output format []: json











Look at the files ~/.aws/credentials and configure.



You will find what you had entered configured in these files.



So, yo u can change anytime you want from these files as well. 



















          

        

          

        

Thursday, January 30, 2020


          

        




















Amazon Web Services (AWS): Enabling API Access














 
      







The First step of enabling API access is to get your AWS Access keys properly created and defined. 



Step 1 - Login to AWS console and Navigate to IAM



Step 2 - Navigate to users and select user which you want to have the access key 



Step 3 - Go to Security Credentials and Click on Create Access Key. 



Remember, you will get a pop-up and this is the only time you will able to see the Secret Access Key, so it is a good choice to download and save it somewhere and probably somewhere safe as it gives access to your account. 

However, if you loose it, contact your admin and the admin can delete it or make it inactive



Okay, Now that you have 2 parts of Key

Access Key and Secret Access Key.



You should now configure your environment

In your home directory create a folder named .aws and create the credentials file as below.



cat ~/.aws/credentials

[default]

aws_access_key_id=xxxxxxxx

aws_secret_access_key=xxxxx



Now this will be used as your default credentials.



You should also create a config file which specifies default region 

Example Below 

cat ~.aws/config

[default]



region=us-east-1



The other option can be to set it as your environment variables or set it as your environment variables in the Code Editor you use. 

I use pycharm, so go to Run and Edit Configurations, you will see environment  variables there which you can configure. 



Also, you can use boto3 quick start for your help as well.



https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html
































Amazon Web Services (AWS) : EC2 - Instance Metadata Get Region/Type/IP













 
      







In this blog I am going to demonstrate how to get the Instance Metadata information using curl.



Instance Metadata information is available at the 169.254.169.254 IP address.

If you are from networking background you will realize, this is Automatic Private IP or Self Assigned Private IP.



So, we use curl to get the information and as an example below. 

















$ curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | grep -e Type -e region -e Ip



  "instanceType" : "m5.2xlarge",



  "privateIp" : "10.114.32.79",



  "region" : "us-east-1",















          

        

          

        

Wednesday, January 15, 2020


          

        





















Amazon Web Services (AWS) : Customising the Sign-In Link for IAM Users













 
      







When you start creating IAM users, the first AWS does automatically for you is to create an IAM sign in link which could be given to your users. 











You can click on customize to customize it to your custom name and then provide it to your users in order to sign in directly to your account. 











































Amazon Web Services (AWS) : Setting up Security / IAM Best Practices at account creation













 
      







In this blog I am going to show, how to have all the green ticks as per AWS recommendation, after setting up a new account. 



Once you create a new account (most probably free tier), go to IAM and then if you are not already go to Dashboard. 













You should see your status as below.

The next step is to activate an MFA on your account. MFA is  multi-factor authentication which provides additional layer of security to login to your account.

The most easy (per me) downloa Duo Mobile and add your account. 

You will be asked to enter 2 continous codes and scan the QR Code. 







The next step is to add a password policy to the user account. 









The next Step is to create a group and a user. 

So, click on Add Group and name : AWSAdmins.

Attach the policy as shown below - 'Administrator Access'





Review and Create

Next Create a user and attach this policy to the user, username can be any what you want and password you can set it at your ease.



Now if you go back to the dashboard, you will see - all your boxes are 'green ticked'

















          

        

          

        

Tuesday, January 7, 2020


          

        




















AWS : Lambda: Add IP to Security Group Using Boto3 - Complete Code













 
      








IN this blog I am going to show how to add an IP using Lambda.


You can create a sample-SQS trigger event with an IP address in body to create an SQS event emulation for testing..






Now, 


    

  • lamda_handler is the default handler for lambda
    • 

  • It checks if the IP is not already part of the rule
    • 

  • it then calls updateIP and refreshes the timestamp if yes or adds new with new timestamp if no
    • 

  • the revoke is to temporarily revoke and add the IP.
    • 







import boto3
from datetime import datetime

ec2 = boto3.resource('ec2')
s_group = ec2.SecurityGroup('sg-85d42ac2')
dt = datetime.now()
date_format = "%m-%d-%Y %H:%M"str_dt = dt.strftime(date_format)
ssh_port = 22code = 200max_minutes = 5

def lambda_handler(event, context):
    for record in event['Records']:
        ip = record["body"]
        if (str(ip) == 'sweep'):
            sweepIP()
        else:
            verifyAddIP(str(ip))

def verifyAddIP(strIP):
    m_strIP = strIP + '/32'
    ip_permission = s_group.ip_permissions[0]
    ip_range = ip_permission['IpRanges']

    for cidr in ip_range:
        if (cidr['CidrIp'] == m_strIP):
            updateRule(strIP + '/32', True)
        else:
            updateRule(strIP + '/32', False)


def updateRule(strIP, update_p):
    if update_p:
        response = s_group.revoke_ingress(IpProtocol="tcp", CidrIp=strIP, FromPort=ssh_port, ToPort=ssh_port)
        response = s_group.authorize_ingress(IpPermissions=[
            {'IpProtocol': 'tcp',
             'FromPort': ssh_port,
             'ToPort': ssh_port,
             'IpRanges': [{'CidrIp': strIP, 'Description': str_dt}]
             }
        ]
        )
        print ('Update IP Address Time in Ingress Rule - ' + strIP)
    else:
        response = s_group.authorize_ingress(IpPermissions=[
            {'IpProtocol': 'tcp',
             'FromPort': ssh_port,
             'ToPort': ssh_port,
             'IpRanges': [{'CidrIp': strIP, 'Description': str_dt}]
             }
        ]
        )
        print ('Added IP Address to Ingress Rule - ' + strIP)















          

        

          

        

Thursday, December 5, 2019


          

        




















AWS : Lambda: Remove IP to Security Group Using Boto3














 
      







In this blog I discuss on how to remove an IP from Security Group using Boto3








import boto3


ec2 = boto3.resource('ec2')


s_group = ec2.SecurityGroup('sg-<ID>')




response = s_group.revoke_ingress(IpProtocol="tcp", CidrIp=strIP, FromPort=22, ToPort=22)


print (response)




Here strIP : IP Range - Example 10.24.25.0/24


From Port and To Port are port Ranges 








With the above you can revoke an IP address rule




































AWS: Boto3: Send Message Queue














 
      







In this simple example, I configure AWS Boto3 to send a message to known queue.






import boto3
sqs = boto3.resource('sqs')
queue = sqs.get_queue_by_name(QueueName='Your_Queue_Name')
response = queue.send_message(MessageBody='BodyText')
print(response.get('MD5OfMessageBody'))








Note - you must configure your Access Credentials, the way they can be done is given in the URL Below 








I generally prefer to set it as my shell variable, but it is totally up to you.



The user/object with which you are accessing should have SQS Policy Attached so that it can write to the Queue.




































AWS : Lambda: Add IP to Security Group Using Boto3















 
      







In this blog I am going to show example on adding an IP address to AWS security group using Boto3.



The way is simple, just create your own Lambda and add the below Code. 

You can have trigger of SQS and put an example IP in the Body . 





import json

import boto3



ec2 = boto3.resource('ec2')

s_group = ec2.SecurityGroup('sg-0308cd0e895d42ac2')

# This is your Security group unique ID





def lambda_handler(event, context):

    failed = False;

    

    try:

      print ("The value IS " + s_group.group_id)

      for record in event['Records']:

        ip = record["body"]

        print (str(ip))

        response = s_group.authorize_ingress(IpProtocol="tcp", CidrIp=str(ip),FromPort=80,ToPort=80)

    except Exception:

      logger.exception("Failed to Add IP")

      # Add your failure function 

    return {

        'statusCode': 200,

        'body': json.dumps('Hello from Lambda!')

    }





Sample Event to Use



{

  "Records": [

    {

      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",

      "receiptHandle": "MessageReceiptHandle",

      "body": "10.2.3.0/32",

      "attributes": {

        "ApproximateReceiveCount": "1",

        "SentTimestamp": "1523232000000",

        "SenderId": "123456789012",

        "ApproximateFirstReceiveTimestamp": "1523232000001"

      },

      "messageAttributes": {},

      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",

      "eventSource": "aws:sqs",

      "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:MyQueue",

      "awsRegion": "us-east-1"

    }

  ]

}



Reference: https://docs.aws.amazon.com/code-samples/latest/catalog/python-ec2-create_security_group.py.html



















          

        

          

        

Wednesday, September 4, 2019


          

        




















AWS/Linux: How to know if  a device has File system on it













 
      







It is a common question which comes to me from people to understand how to know if a raw disk has file system on it or not. 



There can be multiple ways but in this blog I will talk about file command



File command can help you know if the FS is there or not?

How 

use file -s '/device_name'

if you get output like data: that means no FS

If there is any other output, it means FS is present.

















          

        

          

        

Tuesday, September 3, 2019


          

        





















Amazon Web Services (AWS) : Placement Groups














 
      







In this blog, I am going to discuss an important new features which AWS has released which is called as placement groups. 



So, what are placement groups.

Placement groups are basically directives given to AWS Kernel to place your EC2 instances at specific locations. 

And what are these locations

- specific AZ

- specific RAC

- Zone based

and etc



So, As of now (Sep -2019), AWS has come up with 3 types of placement groups 




    

  1. Cluster
    2. 

  2. Spread
    3. 

  3. Groups
    4. 




Let's see what is what 



Cluster - Cluster Placement Group is basically a directive to launch EC2 instances within the same Rack.



Visualize thousands of servers placed in Amazon Data Centers and they are placed in different racks, so when you give 'Cluster' as placement group, all the instances will be launched within the same RAC (in same AZ).



Pros 




    

  1. Great in terms of networking (10Gbps between 2 instances)
    2. 

  2. Gives low latency and high n/w throughput
    3. 

  3. Jobs such as Big data benefit 
    4. 






Cons 

There is one big issue, if the RAC fails, you loose all the instances. 









Spread

Now visualize within all the data centers and you want to distribute your highly available application within multiple data centers, so spread helps you with that - with instances spanning multiple AZs and different physical RAC if in the same AZ.



Pros 




    

  1. Reduced Risk in terms of failure
    2. 

  2. All the instances are on different physical hardware 
    3. 

  3. Provides High Availability for critical applications
    4. 




Cons




    

  1. You are limited to 7 instances per AZ per placement group. 
    2. 














Partition



Within one AZ , span  multiple partitions, and what is partition, an isolation created by AWS in their data centers which is a physical boundary and ensures one partition's availability does not affects others. 







Pros




    

  1. Instances do not share racks with other instances
    2. 

  2. Safe from partition failure
    3. 

  3. EC2 instance get access to partition information as metadata
    4. 

  4. Can spawn 100s of instances
    5. 




Cons






    

  1. The limitation put by AWS (as of now) is 7 partitions per AZ
    2. 


























          

        

          

        

Wednesday, August 28, 2019


          

        




















Amazon Web Services (AWS) : EC2-User Data













 
      







EC2 - user data is one of the seldom used features, yet a powerful feature. 

It is basically a simple script which can is executed at boot time by the system, the script has sudo 



It can be used for - 

• Installing updates

• Installing software

• Downloading common files from the internet



• Anything you can think of



It is configured in Advanced section of the Instance Launch Page. 



Below is an example of apache Service install and launch using an ec2-script. 



#!/bin/bash
yum update -y

yum install -y httpd.x86_64

systemctl start httpd.service

systemctl enable httpd.service



echo "Heloo World" > /var/www/html/index.html/index.html











The greatest of the use of EC2-user data is when using load balancing, starting parallel instances and any boot time action you want to perform (manually or automatically).



Remember - the header #!/bin/bash is mandatory and without it your script will not work. 















          

        

          

        

Friday, August 23, 2019


          

        





















Amazon Web Services (AWS) : Budget













 
      







One of the best feature of learners is the AWS budget feature. 

this feature sends alerts to you when you reach a certain criteria of Budget. 

For example if you budget for say 5 dollars, and can get alerted when you reach 50% of your limit. 



This can help reduce your spend cost, due to instances which might have been spin up and you totally forgot about it. 





You can follow the below process to create budget. 















Rest is mostly intuitive, (Most of it is intuitive by the way)













        

      




































Popular Posts











































© Copyright 2019 DBA to DA. . Powered by Blogger.