How to download multi-files from S3 buckets or Prefix using Python?
AWS Python SDK has file download function from S3 by default. But that one can download the files one at each time only. But sometimes we need to download all the files under particular S3 bucket or Prefix and it can’t be done with that function alone. So this enhanced download script will achieve our requirement.
1. Download and install boto3 and os library
$ pip install boto3 $ pip install os
2. Download multi-files from S3 bucket
# importing the boto3 and os library import boto3 import os # initialize the variables bucket_name = '<your-bucket-name>' prefix = 'output/' # connect to S3 using boto3 client s3_client = boto3.client(service_name='s3') # download file from S3 bucket objects = s3_client.list_objects_v2(Bucket=bucket_name, Prefix=prefix ) for object in objects['Contents'][1:]: # Skip the 1st prefix name in loop using [1:] fileName = object['Key'].split('/') src_des = prefix + str(fileName) print(src_des) s3_client.download_file(bucket_name, src_des, src_des)
That’s it. I hope this will help. ?
One Reply to “How to download multi-files from S3 buckets or Prefix using Python?”
Like!! Really appreciate you sharing this blog post.Really thank you! Keep writing.