boto3 – Page 9 – Tarik Billa

How to save S3 object to a file using boto3

November 27, 2022 by Tarik

There is a customization that went into Boto3 recently which helps with this (among other things). It is currently exposed on the low-level S3 client, and can be used like this: s3_client = boto3.client(‘s3’) open(‘hello.txt’).write(‘Hello, world!’) # Upload the file to S3 s3_client.upload_file(‘hello.txt’, ‘MyBucket’, ‘hello-remote.txt’) # Download the file from S3 s3_client.download_file(‘MyBucket’, ‘hello-remote.txt’, ‘hello2.txt’) print(open(‘hello2.txt’).read()) … Read more

Retrieving subfolders names in S3 bucket from boto3

November 15, 2022 by Tarik

Below piece of code returns ONLY the ‘subfolders’ in a ‘folder’ from s3 bucket. import boto3 bucket=”my-bucket” #Make sure you provide / in the end prefix = ‘prefix-name-with-slash/’ client = boto3.client(‘s3’) result = client.list_objects(Bucket=bucket, Prefix=prefix, Delimiter=”https://stackoverflow.com/”) for o in result.get(‘CommonPrefixes’): print ‘sub folder : ‘, o.get(‘Prefix’) For more details, you can refer to https://github.com/boto/boto3/issues/134

What is the difference between the AWS boto and boto3

November 14, 2022 by Tarik

The boto package is the hand-coded Python library that has been around since 2006. It is very popular and is fully supported by AWS but because it is hand-coded and there are so many services available (with more appearing all the time) it is difficult to maintain. So, boto3 is a new version of the … Read more

How to write a file or data to an S3 object using boto3

October 23, 2022 by Tarik

In boto 3, the ‘Key.set_contents_from_’ methods were replaced by Object.put() Client.put_object() For example: import boto3 some_binary_data = b’Here we have some data’ more_binary_data = b’Here we have some more data’ # Method 1: Object.put() s3 = boto3.resource(‘s3’) object = s3.Object(‘my_bucket_name’, ‘my/key/including/filename.txt’) object.put(Body=some_binary_data) # Method 2: Client.put_object() client = boto3.client(‘s3′) client.put_object(Body=more_binary_data, Bucket=”my_bucket_name”, Key=’my/key/including/anotherfilename.txt’) Alternatively, the binary … Read more

How to specify credentials when connecting to boto3 S3?

October 23, 2022 by Tarik

You can create a session: import boto3 session = boto3.Session( aws_access_key_id=settings.AWS_SERVER_PUBLIC_KEY, aws_secret_access_key=settings.AWS_SERVER_SECRET_KEY, ) Then use that session to get an S3 resource: s3 = session.resource(‘s3’)

Save Dataframe to csv directly to s3 Python

October 21, 2022 by Tarik

You can use: from io import StringIO # python3; python2: BytesIO import boto3 bucket=”my_bucket_name” # already created on S3 csv_buffer = StringIO() df.to_csv(csv_buffer) s3_resource = boto3.resource(‘s3’) s3_resource.Object(bucket, ‘df.csv’).put(Body=csv_buffer.getvalue())

Boto3 Error: botocore.exceptions.NoCredentialsError: Unable to locate credentials

October 19, 2022 by Tarik

try specifying keys manually s3 = boto3.resource(‘s3′, aws_access_key_id=ACCESS_ID, aws_secret_access_key= ACCESS_KEY) Make sure you don’t include your ACCESS_ID and ACCESS_KEY in the code directly for security concerns. Consider using environment configs and injecting them in the code as suggested by @Tiger_Mike. For Prod environments consider using rotating access keys: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_RotateAccessKey

Open S3 object as a string with Boto3

October 13, 2022 by Tarik

read will return bytes. At least for Python 3, if you want to return a string, you have to decode using the right encoding: import boto3 s3 = boto3.resource(‘s3’) obj = s3.Object(bucket, key) obj.get()[‘Body’].read().decode(‘utf-8’)

How to choose an AWS profile when using boto3 to connect to CloudFront

October 10, 2022 by Tarik

I think the docs aren’t wonderful at exposing how to do this. It has been a supported feature for some time, however, and there are some details in this pull request. So there are three different ways to do this: Option A) Create a new session with the profile dev = boto3.session.Session(profile_name=”dev”) Option B) Change … Read more

How to handle errors with boto3?

October 1, 2022 by Tarik

Use the response contained within the exception. Here is an example: import boto3 from botocore.exceptions import ClientError try: iam = boto3.client(‘iam’) user = iam.create_user(UserName=”fred”) print(“Created user: %s” % user) except ClientError as e: if e.response[‘Error’][‘Code’] == ‘EntityAlreadyExists’: print(“User already exists”) else: print(“Unexpected error: %s” % e) The response dict in the exception will contain the … Read more