Link to home
Start Free TrialLog in
Avatar of Tessando
TessandoFlag for United States of America

asked on

Python Script Help for Traversing S3 Buckets

I've got a Python script that traverses S3 Buckets and prints out what folders and files have public permissions. This can be handy when auditing for security issues.

Right now, the script runs fine, but times-out by the time it hits the third bucket.

Can someone please help me find a way to "hard code" this on a per-bucket basis? In other words, if I have a bucket called "art-bucket", how could I get the script to traverse JUST that bucket and provide me the results.

BTW - I've installed both boto3 & Paginator

Thanks for your help.

#This Script will use Paginator to print result for each bucket, executed in multiple threads
import boto3
import threading
import os.path

ACCESS_KEY = 'AKIAIXXXXXXXXXXX'
SECRET_ACCESS_KEY = 'XXUPJIsSXXxxXXxxXXo9Fl5TzSxXXxxXX3ly2XXlxjXXxxXX'

session = boto3.Session(aws_access_key_id = ACCESS_KEY, aws_secret_access_key = SECRET_ACCESS_KEY)

maxthreads = 5
sema = threading.Semaphore(value=maxthreads)

def list_object(bucket):
    try:
        s3 = session.client('s3')
        flag1 = objcount = 0
        paginator = s3.get_paginator('list_objects')
        page_iterator = paginator.paginate(Bucket= bucket)
        for page in page_iterator:
            if 'Contents' in page:
                for obj in page['Contents']:
                    uniobj = obj['Key'].encode('ascii', 'ignore').decode('ascii')
                    objAcl = s3.get_object_acl(Bucket=bucket, Key=obj['Key'])
                    flag2 = 0
                    for perm in objAcl['Grants']:
                        try:
                            if perm['Grantee']['Type'] == 'Group' and perm['Grantee']['URI'] == 'http://acs.amazonaws.com/groups/global/AllUsers':
                                if flag1 == 0:
                                    f = open(bucket + '_list.txt', 'w')
                                    print >> f, 'Bucket Name: ' + bucket + '\n####################'
                                    flag1 = 1
                                if flag2 == 0:
                                    print >> f, '\n' + uniobj + ':'
                                    flag2 = 1
                                permdets = perm['Permission']
                                if permdets == 'READ':
                                    print >> f, 'Public Permission for Reading Object'
                                elif permdets == 'READ_ACP':
                                    print >> f, 'Public Permission for Reading Object ACL'
                                elif permdets == 'WRITE_ACP':
                                    print >> f, 'Public Permission for Adding/Modifying Object ACL'
                                #f.close()
                        except Exception as e:
                            print (uniobj + ':')
                            print (e)
                            continue
                        objcount += 1
            else:
                pass
        if os.path.isfile((bucket + '_list.txt')):
            f.close()
    except Exception as e:
        print (bucket + ':')
        print (e)
    print("%s: DONE : Scanned %i Objects" % (bucket, objcount))
s3 = session.client('s3')
for bucket in s3.list_buckets()['Buckets']:
    try:
        thread = threading.Thread(target = list_object, args=(bucket['Name'],)).start()
    except Exception as e:
        continue

Open in new window

Avatar of Shalom Carmel
Shalom Carmel
Flag of Israel image

See line 58 in your code?
Replace it with the following to traverse ONLY some buckets:

list_of_buckets = ['art-bucket', 'maybe-another-bucket']
for bucket in list_of_buckets:

Open in new window

Avatar of Tessando

ASKER

Thanks Shalom!

I tried this today and the script doesn't generate the text file that shows the shares.

(1) Is there a way, like a switch or part of a command to turn on logging so that I can see where it failed?

(2) When I replaced line 58, this is what I did. Is this correct?

User generated image
Thanks for your help!
ASKER CERTIFIED SOLUTION
Avatar of Tessando
Tessando
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial