Skip to navigation

Custom Transforms in Benchling Connect

Inject your own logic into the Benchling Connect processing pipeline with a Benchling App.

Background Knowledge

Custom Transforms are a way to inject arbitrary logic into Benchling Connect’s processing pipeline. The user will upload a file to a Run’s Automation Output Processor, invoke the transform by attempting to “process” the uploaded file, and then Benchling will wait until an integration writes back a blob.

Once the transformed file has been written to Benchling, the Automation Output Processor will continue to process the file according to the configuration in the Run’s outputFileConfigs.

Why this page uses the V2 SDK

V3 is the recommended API for new development, but it has no automation file transform endpoints. V3 exposes the automation configs read-only, at GET /api/v3/run-schema/{run_schema_id}/automation-input-file-configs/items and .../automation-output-file-configs/items, and nothing that updates a transform. Writing a transformed file back uses PATCH /api/v2/automation-file-transforms/{transform_id}, which the V2 Python SDK (benchling-sdk) wraps as benchling.lab_automation.update_transform. The transform events are V2 events too, and the write-back’s blobId must be a V2 blob ID (a V3 attachment ID is not accepted), so this whole flow stays on V2.

Custom Transform

Custom transforms use a Benchling App to do the transformation, and are configured as part of the processingSteps in the outputFileConfigs of a Benchling Connect Run schema.

Below is an excerpt of a Run Schema that only executes a Custom Transformation.

"processingSteps": [
{
"inputFileType": "CUSTOM",
"transformationProperties": {
"transformId": "app_<your_id_here>"
}
}
],

User Flow

A user will upload a file to the Automation Output Processor via inserting a Run into a Notebook entry, or via API.

A Benchling Connect run with a CSV file attached to its Automation Output Processor, ready to process
File uploaded to the Automation Output Processor

When the user clicks “Process”, the “Processing” indicator will spin until an integration writes back by updating the transform (PATCH /api/v2/automation-file-transforms/{transform_id}).

The Automation Output Processor showing a spinning Processing indicator while it waits for the transform
The Automation Output Processor has been invoked

If the integration writes back a blob, then the processor pipeline continues on.

The Automation Output Processor showing a completed run with results recorded from the transformed file
Successful transform and processing of the file

However, if the integration writes back an error, this will cancel the Automation Output Processor.

The Automation Output Processor in a cancelled state, displaying the error message returned by the transform
The Automation Output Processor has been cancelled due to an error encountered during the transform

Integration Walkthrough

Now that we have seen how we would use a Custom Transform, let’s discuss what’s going on under the hood.

A Top-Down View

The core functionality of any Custom Transform integration, built as an event-driven AWS Lambda, will look like this:

def lambda_handler(event, context):
transform = event["detail"]["transform"]
try:
input_data = fetch_blob(transform["blobId"])
processed_data = do_business_logic(input_data)
output_blob = create_blob(processed_data)
set_transform_output(transform["id"], output_blob)
except Exception as e:
set_transform_error(transform["id"], str(e))

Let’s break that down a little.

fetch_blob

This helper function retrieves the original file uploaded to the Automation Output Processor and reads it into a pandas DataFrame.

def fetch_blob(blob_id: str) -> pd.DataFrame:
return pd.read_csv(benchling.blobs.download_bytes(blob_id))

do_business_logic

Here, any programmatic transformation of the input blob is possible. You can look things up from external sources, collate metadata, report out to other systems; anything you deem necessary.

Please refer to the example below, which calculates the “Percentage of Whole” for each row’s Result value.

def do_business_logic(input_data: pd.DataFrame):
input_data["Percentage of Whole"] = round(input_data["result"]/sum(input_data["result"]) * 100)
return input_data

With an updated Result schema (to capture the additional column we just created), we would expect to see:

A Benchling results table with a Percentage of Whole column added by the custom transform
Transformed file includes the "Percentage of Whole" column

create_blob

This function takes the processed_data from do_business_logic and generates a blob in Benchling.

Assuming that our processed_data is a pandas DataFrame, the function could look like:

def create_blob(data: pd.DataFrame):
return benchling.blobs.create_from_bytes(
data.to_csv(index=False).encode("utf-8"),
name="Custom Transform Output.csv",
mime_type="text/csv",
)

create_from_bytes takes bytes, not a string, and requires a name for the blob.

set_transform_output

This function performs the action of writing the newly-created blob back to the Automation Output Processor in Benchling.

Notice that the update simply includes the blob ID and has no definition of errors.

from benchling_sdk.models import LabAutomationTransformUpdate
def set_transform_output(transform_id, blob):
return benchling.lab_automation.update_transform(
transform_id, LabAutomationTransformUpdate(blob_id=blob.id)
)

set_transform_error

If the Transform fails, the integration writes back details of the failure to the Automation Output Processor.

Notice that the update only includes errors, and does not provide a blob to the Automation Output Processor.

from benchling_sdk.models import LabAutomationBenchlingAppError, LabAutomationTransformUpdate
def set_transform_error(transform_id, error_string):
return benchling.lab_automation.update_transform(
transform_id,
LabAutomationTransformUpdate(errors=[LabAutomationBenchlingAppError(message=error_string)]),
)

Full Code Example

Here is a lambda handler that responds to a v2.automationFileTransform.updated.status.running event and writes a blob back.

import pandas as pd
from benchling_sdk.auth.api_key_auth import ApiKeyAuth
from benchling_sdk.benchling import Benchling
from benchling_sdk.models import LabAutomationBenchlingAppError, LabAutomationTransformUpdate
benchling = Benchling(url="https://yourcompany.benchling.com", auth_method=ApiKeyAuth("<your-api-key>"))
def fetch_blob(blob_id: str) -> pd.DataFrame:
return pd.read_csv(benchling.blobs.download_bytes(blob_id))
def create_blob(data: pd.DataFrame):
return benchling.blobs.create_from_bytes(
data.to_csv(index=False).encode("utf-8"),
name="Custom Transform Output.csv",
mime_type="text/csv",
)
def set_transform_output(transform_id, blob):
return benchling.lab_automation.update_transform(
transform_id, LabAutomationTransformUpdate(blob_id=blob.id)
)
def set_transform_error(transform_id, error_string):
return benchling.lab_automation.update_transform(
transform_id,
LabAutomationTransformUpdate(errors=[LabAutomationBenchlingAppError(message=error_string)]),
)
def do_business_logic(input_data: pd.DataFrame):
input_data["Percentage of Whole"] = round(input_data["result"]/sum(input_data["result"]) * 100)
return input_data
def lambda_handler(event, context):
transform = event["detail"]["transform"]
try:
input_data = fetch_blob(transform["blobId"])
processed_data = do_business_logic(input_data)
output_blob = create_blob(processed_data)
set_transform_output(transform["id"], output_blob)
except Exception as e:
set_transform_error(transform["id"], str(e))

Appendix

AWS EventBridge Rule

Here is an example EventBridgeRule to invoke the above code. The customTransformId is the ID of your Benchling App.

{
"detail-type": [
"v2.automationFileTransform.updated.status.running"
],
"detail": {
"transform": {
"customTransformId": ["app_FHD6INW2pyvPEXK1"]
}
}
}

Benchling Connect Run Schema

Here is an example of a Benchling Connect Run’s schema, written in JSON.

{
"processingSteps": [
{
"inputFileType": "CUSTOM",
"outputFileType": "FINAL_DATA_FRAME",
"transformationProperties": {
"transformId": "app_0HLHMi6WnyhQRSYq"
}
}
],
"resultSchemaInfo": {
"fields": {
"schema_field_77": {
"type": "PRIMARY_SAMPLE_ENTITY"
},
"percentage_of_whole": {
"columnName": "Percentage of Whole",
"type": "DIRECT"
},
"result": {
"columnName": "result",
"type": "DIRECT"
}
},
"schema": "assaysch_aGnTdxCz"
},
"tableType": "GRID",
"columnTypes": [
"ENTITY_NAME",
"SCHEMA_FIELD",
"SCHEMA_FIELD"
],
"delimiter": ",",
"fileExtension": "csv",
"name": "Records Results",
"volumeUnits": "uL"
}