Using aws s3 cp from the AWS Command-Line Interface (CLI) will require the –recursive parameter to copy multiple files. aws s3 cp s3://myBucket/dir localdir –recursive The aws s3 sync command will, by default, copy a whole directory. It will only copy new/modified files. aws s3 sync s3://mybucket/dir localdir Just experiment to get the result you … Read more
You can use: from io import StringIO # python3; python2: BytesIO import boto3 bucket=”my_bucket_name” # already created on S3 csv_buffer = StringIO() df.to_csv(csv_buffer) s3_resource = boto3.resource(‘s3’) s3_resource.Object(bucket, ‘df.csv’).put(Body=csv_buffer.getvalue())
I got the same error message, when I did following: created a bucket – it went by default to US region (used AWSCLI) realized, the bucket shall go to EU region and deleted it (used AWS console) (few minutes later) tried to create the bucket, specifying the EU region At step 3, AWS console has … Read more
UPDATE: Amazon now supports gzip compression, so this is no longer needed. Amazon Announcement Original answer: The answer is to gzip the CSS and JavaScript files. Yes, you read that right. gzip -9 production.min.css This will produce production.min.css.gz. Remove the .gz, upload to S3 (or whatever origin server you’re using) and explicitly set the Content-Encoding … Read more
To download multiple files from an aws bucket to your current directory, you can use recursive, exclude, and include flags. The order of the parameters matters. Example command: aws s3 cp s3://data/ . –recursive –exclude “*” –include “2016-08*” For more info on how to use these filters: http://docs.aws.amazon.com/cli/latest/reference/s3/#use-of-exclude-and-include-filters
Just a note to add on here: it’s now 3 years later, yet this post is top in Google when you type in “How to search an S3 Bucket.” Perhaps you’re looking for something more complex, but if you landed here trying to figure out how to simply find an object (file) by it’s title, … Read more
Go to http://awspolicygen.s3.amazonaws.com/policygen.html Fill in the details such as: In Action select “GetObject” Select “Add Statement” Then select “Generate Policy” Copy the text example: { “Id”: “Policy1397632521960”, “Statement”: [ { “Sid”: “Stmt1397633323327”, “Action”: [ “s3:GetObject” ], “Effect”: “Allow”, “Resource”: “arn:aws:s3:::bucketnm/*”, “Principal”: { “AWS”: [ “*” ] } } ] } Now go to your AWS … Read more
You have given permission to perform commands on objects inside the S3 bucket, but you have not given permission to perform any actions on the bucket itself. Slightly modifying your policy would look like this: { “Version”: “version_id”, “Statement”: [ { “Sid”: “some_id”, “Effect”: “Allow”, “Action”: [ “s3:*” ], “Resource”: [ “arn:aws:s3:::bucketname”, “arn:aws:s3:::bucketname/*” ] } … Read more
Solution aws s3 mb s3://[new-bucket] aws s3 sync s3://[old-bucket] s3://[new-bucket] aws s3 rb –force s3://[old-bucket] Explanation There’s no rename bucket functionality for S3 because there are technically no folders in S3 so we have to handle every file within the bucket. The code above will 1. create a new bucket, 2. copy files over and … Read more
Using AWS CLI aws s3 ls s3://mybucket/ –recursive | wc -l or aws cloudwatch get-metric-statistics \ –namespace AWS/S3 –metric-name NumberOfObjects \ –dimensions Name=BucketName,Value=BUCKETNAME \ Name=StorageType,Value=AllStorageTypes \ –start-time 2016-11-05T00:00 –end-time 2016-11-05T00:10 \ –period 60 –statistic Average Note: The above cloudwatch command seems to work for some while not for others. Discussed here: https://forums.aws.amazon.com/thread.jspa?threadID=217050 Using AWS Web … Read more