Skip to navigation
Attachments and Files

Large Attachment Upload

Stream a large file from disk to Benchling in one upload with the V3 API, or split it into parts with V2.

This page uses the beta V3 scratch file and attachment endpoints. Set early_access="true" on the client or send the EARLY-ACCESS: true header on raw HTTP requests. See the V3 Stability Guidelines.

V2 files larger than 10MB had to be split into parts and reassembled. The V3 API has no multipart upload: a file goes to a scratch file’s presigned upload URL in a single PUT, and an attachment is created from the scratch file. For a large file, the only care needed is to stream it from disk instead of reading it into memory.

Store your attachment ID

Attachments are not searchable on their own. If you do not persist the ID before linking the attachment to a parent object, it is lost and the file has to be re-uploaded.

To link the attachment once it is uploaded, see Uploading an Attachment to Benchling. For an image read out of a spreadsheet, see Image File Upload.

V3 upload

  1. Configuration

    The V3 SDK, httpx (installed with the SDK), the tenant URL, and an API key. Pass the tenant’s root URL; the SDK adds /api/v3 itself.

    The key is inline here for demonstration only. Benchling credentials should not be stored in source code. See Authentication for the alternatives.

  2. Choose the file

    The file name and location are arbitrary examples; substitute your own.

  3. Create a scratch file

    The response carries the upload_url the file goes to.

  4. Stream the file to the upload URL

    Passing the open file as content makes httpx read it in chunks and send it with a Content-Length header, so the whole file is never held in memory. The x-amz-server-side-encryption: AES256 header is required, and the Content-Type header becomes the attachment’s MIME type.

    1. Timeout

      httpx times out after five seconds by default, which a large upload can exceed. timeout=None turns the timeout off; set a limit that suits your file sizes instead if you prefer.

  5. Mark the upload finished

    Set the scratch file’s upload_status to SUCCEEDED once the PUT has returned.

  6. Create the attachment and keep its ID

    Create the attachment from the scratch file, then store its ID for linking. This example prints it to the console.

import httpx
from benchling_v3_sdk import Benchling
# Change to your API key
api_key = "***************"
# Change to your tenant
tenant = "https://example.benchling.com"
benchling = Benchling(base_url=tenant, api_key=api_key, early_access="true")
filename = "Compound_160500001_161000000.xml.gz"
path = "Downloads/" + filename
scratch_file = benchling.scratch_file.create(filename=filename, name=filename)
with open(path, "rb") as f:
    upload = httpx.put(
        scratch_file.upload_url,
        content=f,
        headers={"x-amz-server-side-encryption": "AES256", "Content-Type": "application/gzip"},
        timeout=None,
    )
upload.raise_for_status()
benchling.scratch_file.update(scratch_file.id, upload_status="SUCCEEDED")
attachment = benchling.attachment.create(scratch_file_id=scratch_file.id)
print(attachment.id)

V2 multipart upload

V3 has no multipart endpoints, so this flow uses the V2 blob endpoints. It splits the file into parts, uploads each one, and completes the upload so Benchling can reassemble it. Mixing the two API versions in one integration is supported, and both accept the same API key.

  1. Configuration

    A few built-in libraries, plus the tenant URL and API key.

    The key is inline here for demonstration only. Benchling credentials should not be stored in source code. See Authentication for the alternatives.

  2. Read the file in chunks

    A generator that reads the file a chunk at a time. Every part except the last has to be at least 5MB and no more than 10MB.

    1. Choosing a chunk size

      Both sizes are a byte or two over the round number so the chunk length is divisible by 3, which avoids base64 padding mid-file. Either works; pick what suits your implementation.

  3. Encode and hash helpers

    Two more helpers: one base64-encodes a chunk, the other takes its MD5 hash. Note that the hash is of the raw chunk, not of its base64 encoding.

  4. Start the multipart upload

    POST /api/v2/blobs:start-multipart-upload returns metadata about the upload, most importantly the id, which is the blob_id every later call needs.

  5. Upload a single part

    Each response from POST /api/v2/blobs/{id}/parts carries the part’s partNumber and an eTag UUID. Both are required by the final step, so store them for every part.

    If an upload fails partway through, abort it with POST /api/v2/blobs/{id}:abort-upload.

  6. Complete the multipart upload

    Once every part is uploaded, POST /api/v2/blobs/{id}:complete-upload finishes the job. Its body carries a parts array of the partNumber and eTag of each part. That serves two purposes: telling Benchling the upload is done, and giving Benchling a way to confirm it received every part it was expecting.

  7. Open the file

    The file name and location are arbitrary examples; substitute your own.

  8. Start the upload and keep the blob ID

    Start the upload with the filename, then store the blob_id for the part uploads that follow.

  9. Number the parts

    partNumber values do not have to be consecutive, but they must be unique and ascending.

  10. Upload each part

    For every chunk, encode it, hash it, upload it as a blob part, and append the response to part_data, which becomes the parts array in the final request.

  11. Complete the upload

    One final request once all the chunks are up.

  12. Log the result

    This example prints to the console. Substitute your own logging or post-upload work.

import sys, base64
import hashlib
import requests
# Change to your API Key
api_key = '***************'
# Change to your tenant
tenant = "https://example.benchling.com"
#chunk = 5242881 # 5MB + 1 byte to make divisible by 3 to avoid b64 padding
chunk = 10485762 # 10MB + 2 byte to make divisible by 3 to avoid b64 padding
def read_in_chunks(file_object, chunk_size=chunk):
    while True:
        data = file_object.read(chunk_size)
        if not data:
            break
        yield data
def b64_encode(data):
    return base64.b64encode(data).decode('ascii')
def md5_hash(data):
    md5_hasher = hashlib.md5()
    md5_hasher.update(data)
    return md5_hasher.hexdigest()
def start_multipart(filename):
    request_body = {
      "mimeType": "application/octet-stream",
      "name": filename,
      "type": "RAW_FILE"
    }
    response = requests.post(tenant + "/api/v2/blobs:start-multipart-upload", auth=(api_key, ""), json=request_body)
    return response.json()
def upload_part(part, md5, number, blob_id):
    request_body = {
          "data64": part,
          "md5": md5,
          "partNumber": number
    }
    response = requests.post(tenant + "/api/v2/blobs/" + blob_id + "/parts", auth=(api_key, ""), json=request_body)
    print(response)
    print(response.text)
    return response.json()
def complete_upload(blob_id, parts):
    request_body = {
          "parts" : parts
    }
    response = requests.post(tenant + "/api/v2/blobs/" + blob_id + ":complete-upload", auth=(api_key, ""), json=request_body)
    return response
filename = 'Compound_160500001_161000000.xml.gz'
f = open('Downloads/' + filename, 'rb')
start_json = start_multipart(filename)
print(start_json)
blob_id = start_json['id']
part_data = []
part_num = 0
for piece in read_in_chunks(f):
    part_num += 1
    b64 = b64_encode(piece)
    md5 = md5_hash(piece)
    json = upload_part(b64, md5, part_num, blob_id)
    part_data.append(json)
resp = complete_upload(blob_id, part_data)
print("\n\n\n" + "------------------------------------" + "\n")
print(resp)
print(resp.text)