Link to home
Start Free TrialLog in
Avatar of sgcity
sgcityFlag for United States of America

asked on

Google Drive Python API - Resumable media (chunked upload) slow

I've posted this in almost all major sites now and no-one has responded. Hopefully someone here can assist.

I'm getting a max of around 20Mbps on any file upload to Google Drive via the Python API which is resumable=True and chunked (regardless of chunksize), however I get my full internet speed of around 180Mbps upload on the same file when not using this method. The problem is that if the file is too big to fit in memory (or even threading many uploads) the computer will run out of memory and I will have to use chunked upload which is just too slow.

Am I doing something wrong? How can I upload a large file without having to worry about running out of memory and not be limited to 20Mbps?

Shared code between both examples:
import os
from google.oauth2 import service_account
from googleapiclient.discovery import build, MediaFileUpload

def create_drive_service(user_email,SERVICE_ACCOUNT_JSON_FILE,SCOPES=None):
    if SCOPES is None: SCOPES = ['https://www.googleapis.com/auth/drive']
    credentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_JSON_FILE)
    credentials = credentials.with_scopes(SCOPES)
    credentials = credentials.with_subject(user_email)
    return build('drive', 'v3', credentials=credentials)
#
# Fill in the following
#
jsonpath = 'c:\\path\\to\\service_account.json'
service = create_drive_service('user@domain',jsonpath)
filename = 'filename.ext'
filepath = 'c:\\path\\to\\folder\\' + filename
parent_id = ''

Open in new window


Code that is slow:
file_metadata = {"name":filename,"parents":[parent_id]}
media = MediaFileUpload(filepath,chunksize=-1,resumable=True)
request = service.files().create(body=file_metadata,media_body=media,fields='id')
while response is None:
    status, response = request.next_chunk()
file = request.execute()

Open in new window


Code that works full-speed but not if the file is too big to fit in memory:
file_metadata = {"name":filename,"parents":[parent_id]}
media = MediaFileUpload(filepath,resumable=False)
request = service.files().create(body=file_metadata,media_body=media,fields='id').execute()

Open in new window


How can I upload a large file without having to worry about running out of memory and not be limited to 20Mbps?
Avatar of aikimark
aikimark
Flag of United States of America image

chunksize=-1
According to the Google documentation, this parameter needs to be a multiple of 256KB.
reference: https://developers.google.com/api-client-library/python/guide/media_upload
Avatar of sgcity

ASKER

You're correct, but no matter the chunksize set the upload speed is severely limited compare to using resumable=False which is 10x faster.
This question needs an answer!
Become an EE member today
7 DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform.
View membership options
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.