> This page is for Developer Documentation. > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.benchling.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.benchling.com/_mcp/server. # Direct Data Share: Outbound Outbound Direct Data Share publishes your Benchling records as Apache Iceberg tables you query from your own platform: Snowflake, Databricks, or any Iceberg client. See [Direct Data Share](/docs/direct-data-share) for how it fits with inbound sharing, supported platforms, and beta availability. This page has three parts: * **Common setup** applies to every platform: getting access, what's shared, freshness, and your connection details. * [Connecting from Snowflake](#connecting-from-snowflake) and [Connecting from Databricks](#connecting-from-databricks) cover each platform's setup. * **Writing queries**, **Performance notes**, and **Troubleshooting** at the end apply to both. ## Common setup ### Getting access Outbound Data Share is in beta. To enable it, contact your Benchling representative and tell them: * **Where you'll read the data.** * **Snowflake:** send your Snowflake account identifier (`.`) if you want the data shared into your own Snowflake account as a listing. * **Databricks:** say that you'll read from Databricks. Benchling then turns on the Databricks export path for your tenant. You can use both platforms. * **Which object types to include**, if you want less than the full catalog. Most customers start with everything. ### Which objects are shared Direct Data Share covers a fixed catalog of Benchling object types. The catalog spans the notebook, registry, inventory, molecular biology, results and runs, lab automation, workflows, studies, chemistry, and platform configuration. Not every Benchling object is currently shareable. A table appears only once there is at least one record of that type to share, so object types you don't use won't show up. For the full list of shareable object types, including the ones your tenant has no data for yet, see [Direct Data Share: Data Shape](/docs/direct-data-share-outbound-model-coverage). To see what your tenant actually has, list the tables on your platform; there are examples in the Snowflake and Databricks sections below. ### Data freshness A change in Benchling is currently visible to your queries within about an hour. The tables shared with you are current-state tables: one row per object, reflecting its latest state. > **Warning** > > ### Direct Data Share is an analytics surface, not currently a real-time one. For event-driven integrations that need to react to a change in seconds, use [Events](/docs/events-getting-started) or [webhooks](/docs/getting-started-with-webhooks) instead. ### Your connection details Once provisioning finishes, a Benchling tenant admin can see your connection profile under **Profile Settings → Data Sharing Connection Details → Show Connection Details**: | Value | What it is | | ------------------------------- | ----------------------------------------------------------------------------------------- | | Account identifier and host | The Snowflake account holding your share, for example `.snowflakecomputing.com` | | Database | `BENCHLING_DATA_SHARING_`, where `` identifies your Benchling deployment | | Schema | `BENCHLING_`, your tenant's namespace, for example `BENCHLING_TEN_ABC123` | | Shared schema (Databricks only) | `BENCHLING__shared`, for example `BENCHLING_TEN_ABC123_shared` | | Read role | `BENCHLING__RO` | | Reader user | `BENCHLING__READER` | | Warehouse | `BENCHLING_DATA_SHARING_READER_WH` | | Programmatic access token (PAT) | The credential for the reader user. You use it as the password. | | Catalog URI and token endpoint | For Iceberg clients that read the catalog directly | > **Warning** > > ### The PAT expires one year after it's issued. Store it in your own secret manager as soon as you receive it, never in a notebook or repository. To rotate it, ask Benchling to re-run provisioning. That issues a fresh token and revokes the old one, so update your clients (including any Databricks connection) when you rotate. The read role can only read your tenant's schemas, and it cannot write. A leaked token cannot reach another tenant's data or modify anything, but it does grant read access to everything shared with you. See [Access control](/docs/direct-data-share#access-control). > **Info** > > **Names are case-sensitive.** Copy them exactly as shown. Tables are uppercase, for example `ENTRY`. The Databricks shared schema is mixed case: an uppercase prefix and a lowercase `_shared`. Organizations onboarded before October 2026 may have all-lowercase schema names. ## Connecting from Snowflake ### A Snowflake listing in your own account If you already run Snowflake, Benchling can publish your namespace as a private listing targeted at your account. You mount it once, then query it as a database in your own account, using your own roles, warehouses, and governance. No Benchling credential sits in your pipelines, and there is no cross-account token to rotate. Give your Benchling representative your Snowflake account identifier (`.`) to have the listing targeted at you. Once it appears under **Data Products → Private Sharing** in Snowsight, mount it and query: ```sql SELECT "id", "name", "createdAt" FROM .BENCHLING_TEN_ABC123.ENTRY WHERE NOT "archived"; ``` ### Exploring the catalog ```sql -- Every table shared with you SHOW TABLES IN SCHEMA BENCHLING_DATA_SHARING_.BENCHLING_TEN_ABC123; -- Columns for one object type SELECT column_name, data_type FROM BENCHLING_DATA_SHARING_.INFORMATION_SCHEMA.COLUMNS WHERE table_schema = 'BENCHLING_TEN_ABC123' AND table_name = 'ENTRY' ORDER BY ordinal_position; ``` If you query through a mounted listing, replace `BENCHLING_DATA_SHARING_` with your mounted database name. Everything in [Writing queries](#writing-queries) applies unchanged. ### Other Iceberg clients The tables are Iceberg, so any Iceberg REST catalog client can read them, such as PyIceberg, Spark, Trino, and Flink. Use the catalog URI, token endpoint, and PAT from your [connection details](#your-connection-details). ```python from pyiceberg.catalog.rest import RestCatalog catalog = RestCatalog( name="benchling", uri="", warehouse="", credential="", **{"oauth2-server-uri": ""}, ) table = catalog.load_table(("BENCHLING_TEN_ABC123", "ENTRY")) df = table.scan(selected_fields=("id", "name", "createdAt")).to_arrow() ``` > **Info** > > Authenticate from the Iceberg client itself, as above. Configuring the catalog as a Snowflake `CATALOG INTEGRATION` is not a supported path for this share. ## Connecting from Databricks In Databricks you read the current-state tables in your **shared schema** (`BENCHLING__shared`) through a **federated catalog**: * **How it appears.** Benchling's catalog shows up in Unity Catalog as a normal foreign catalog, governed like your other catalogs. * **Where queries run.** Queries run on your Databricks compute, which reads the table files directly from Benchling's Amazon S3 bucket using an IAM role you own. * **What Snowflake does.** Snowflake is only used to look up which tables exist. Setup takes three Unity Catalog objects: * **A storage credential:** your IAM role. * **A read-only external location** on your data's S3 prefix. * **A Snowflake connection** with a foreign catalog. There is one handshake with Benchling: you send the ARN of your IAM role, and Benchling grants it read access to your data. ### Before you start * **Connection details.** You need the host, reader user, PAT, warehouse, database, and shared schema from [your connection details](#your-connection-details). * **S3 location.** Your Benchling contact also sends the **S3 bucket, region, and prefix**, in the form `s3:///warehouse/shared//`. Profile Settings doesn't show them. S3 paths are case-sensitive. * **Permissions.** You need `CREATE STORAGE CREDENTIAL`, `CREATE EXTERNAL LOCATION`, `CREATE CONNECTION`, and `CREATE CATALOG` on the metastore (usually a metastore admin). You also need to be able to create an IAM role in your AWS account. ### Step 1. Create the IAM role and the storage credential Databricks needs the role's ARN to create the credential, and only shows the external ID once the credential exists. So you create the role with a placeholder trust policy first, and finish it in Step 2. 1. **Create the IAM role in AWS**, for example `benchling-shared-uc-credential`, with this placeholder trust policy. The principal is the Databricks Unity Catalog role for AWS commercial accounts. `0000` is a placeholder you replace in Step 2. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": {"AWS": "arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL"}, "Action": "sts:AssumeRole", "Condition": {"StringEquals": {"sts:ExternalId": "0000"}} } ] } ``` 2. **Attach this permissions policy** to the role, using your exact bucket and prefix: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:GetObject", "s3:GetObjectVersion"], "Resource": "arn:aws:s3:::/warehouse/shared//*" }, { "Effect": "Allow", "Action": ["s3:ListBucket", "s3:GetBucketLocation"], "Resource": "arn:aws:s3:::" }, { "Effect": "Allow", "Action": "sts:AssumeRole", "Resource": "arn:aws:iam:::role/" } ] } ``` 3. **Create the storage credential in Databricks.** Go to **Catalog → External Data → Credentials → Create credential**, choose **Storage credential** with type **AWS IAM role**, and enter the role ARN. Name it, for example, `benchling_shared_cred`. After you save it, Databricks shows the credential's **external ID**. If it isn't shown right away, open the credential's details. ### Step 2. Finish the trust policy and send the role ARN to Benchling 1. **Update the role's trust policy** in AWS. Replace `0000` with the external ID from Step 1, and add the role itself as a principal: Databricks requires storage-credential roles to be able to assume themselves. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL", "arn:aws:iam:::role/" ] }, "Action": "sts:AssumeRole", "Condition": {"StringEquals": {"sts:ExternalId": ""}} } ] } ``` 2. **Validate the credential** in Databricks. **Assume Role**, **Self Assume Role**, and **External ID Condition** should show Success. A failed **Read** check is expected: your access is limited to your prefix, and the Databricks check probes the bucket root. 3. **Send the role ARN** to your Benchling contact. Benchling grants it read access to your prefix only. > **Warning** > > ### Wait for Benchling to confirm the grant is in place before Step 3. ### Step 3. Create a read-only external location Go to **Catalog → External Data → External Locations → Create location**: | Field | Value | | ---------------------- | ----------------------------------------------------- | | Name | For example `benchling_shared` | | URL | `s3:///warehouse/shared//` | | Storage credential | `benchling_shared_cred` | | Limit to read-only use | **On**. Required, because the credential can't write. | | File events | **Off** | Read and File Events validation may fail for the same reason as in Step 2. Choose **Force create**. With the Databricks CLI, pass `"read_only": true` and `--skip-validation`; the SQL `CREATE EXTERNAL LOCATION` statement has no read-only option. Then let your users read through it: ```sql GRANT READ FILES ON EXTERNAL LOCATION benchling_shared TO ``; ``` ### Step 4. Create the Snowflake connection and foreign catalog Go to **Catalog → External Data → Connections → Create connection**: | Field | Value | | --------------- | --------------------------------------------------------------------------------------- | | Connection type | **Snowflake** | | Host / port | The host from your connection details / `443` | | Authentication | **Username and password**: user = the reader user, password = the PAT. Don't use OAuth. | | Warehouse | `BENCHLING_DATA_SHARING_READER_WH` | The wizard then creates the foreign catalog: | Field | Value | | -------------------- | ----------------------------------------- | | Catalog name | For example `benchling` | | Database | The database from your connection details | | **Storage location** | **Your external location from Step 3** | > **Info** > > **Setting the Storage location is what makes this work.** It switches the catalog to read Iceberg files from S3 on your compute, and stops queries from running in Snowflake. The warehouse is then only used for small metadata queries, such as listing schemas and tables, and never for scanning data. Grant your users access to the catalog as you would for any other catalog, for example: ```sql GRANT USE CATALOG, USE SCHEMA, SELECT ON CATALOG benchling TO ``; ``` ### Step 5. Query ```sql SHOW TABLES IN benchling.`BENCHLING_TEN_ABC123_shared`; SELECT * FROM benchling.`BENCHLING_TEN_ABC123_shared`.`ENTRY` LIMIT 100; ``` ```python df = spark.table("benchling.`BENCHLING_TEN_ABC123_shared`.`ENTRY`") ``` * **Query by name through the catalog.** Reading the S3 paths directly doesn't work: table folders are named `.`, and Databricks splits a path on the dot. * **Check where it ran.** Open the query profile. You should see a file scan through your external location, and no Snowflake query. ### Databricks troubleshooting \[#databricks-troubleshooting] | Symptom | Cause and fix | | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Storage credential or external location validation: Read / File Events failed | Expected. Force create, and test with a real query (Step 5). | | External ID Condition failed | The trust policy still has the `0000` placeholder, or a different external ID. Copy the one shown on the storage credential (Step 2). | | Self Assume Role failed | The role's trust policy must include the role itself (Step 2), and its permissions policy must allow `sts:AssumeRole` on its own ARN (Step 1). | | External location creation fails with a write error | Turn on **Limit to read-only use**. | | Connection test hangs, or "no active warehouse" | Set the connection's Warehouse field to `BENCHLING_DATA_SHARING_READER_WH`. If it's already set, ask Benchling to confirm your reader user can use it. | | Authentication fails | Use **Username and password**, with the reader user and the PAT as the password. If the token was rotated, update the connection's password. | | S3 403 / AccessDenied when querying | One of: Benchling hasn't applied the grant yet; the ARN you sent isn't the role Unity Catalog uses; the prefix case is wrong; or the foreign catalog's Storage location isn't set to your external location. | | Shared schema doesn't appear in the catalog | Names are case-sensitive and the shared schema is mixed case. Contact Benchling. | | Table "does not exist" | Use the exact case, in backticks. | | Shared schema is empty | Current-state tables appear within about an hour of onboarding. If it stays empty, contact Benchling. | | Queries are slow, or Snowflake shows query activity | The foreign catalog's Storage location isn't set, so queries are being sent to Snowflake. Set it to your external location (Step 4). | ## Writing queries \[#writing-queries] See [Outbound Data Shape](/docs/direct-data-share-outbound-model-coverage#outbound-data-shape) for table and column naming, schema field value tables, archived records, and timestamp formatting. Three conventions from there account for most mistakes: 1. **Quote column names, not table names.** `SELECT "createdAt" FROM ENTRY`, never `SELECT createdAt`. In Databricks, quote with backticks instead of double quotes. 2. **Filter archived records** with `WHERE NOT "archived"` unless you want them. 3. **Parse timestamps** with `TRY_TO_TIMESTAMP("createdAt")` before any date arithmetic. The examples below are Snowflake SQL and use `BENCHLING_TEN_ABC123` as the schema. Substitute your own, and set it with `USE SCHEMA` so you can write bare table names. In Databricks, use your shared schema (for example ``USE benchling.`BENCHLING_TEN_ABC123_shared` ``) and backticks for quoted identifiers. #### Simple joins Reference fields are IDs, so joining is how you get names. This lists recent notebook entries with their author, folder, project, and schema: ```sql SELECT e."displayId", e."name" AS entry_name, TRY_TO_TIMESTAMP(e."createdAt") AS created_at, u."name" AS creator_name, f."name" AS folder_name, p."name" AS project_name, s."name" AS schema_name FROM ENTRY AS e LEFT JOIN USER AS u ON u."id" = e."creator_id" LEFT JOIN FOLDER AS f ON f."id" = e."folder_id" LEFT JOIN PROJECT AS p ON p."id" = f."project_id" LEFT JOIN ENTRYSCHEMA AS s ON s."id" = e."schema_id" WHERE NOT e."archived" ORDER BY created_at DESC LIMIT 100; ``` Use `LEFT JOIN` for reference fields that are optional: an unfiled entry has no `"folder_id"`, and an entry with no schema has no `"schema_id"`. An inner join silently drops those rows. For a registry census, an inventory-with-storage-hierarchy example, and schema field joins (including pivoting fields into columns), see [Outbound Data Shape](/docs/direct-data-share-outbound-model-coverage#outbound-data-shape). ## Performance notes * **Project the columns you need.** These are columnar Iceberg tables, so `SELECT "id", "name"` reads far less than `SELECT *`. Object tables can be wide, and a few columns hold large payloads, including `DNASEQUENCE."bases"` and `ENTRY."sections"`. * **Filter before joining schema field values.** Narrow the parent table first in a CTE, then join to the value tables. The value tables hold a row per field per object and are usually the largest tables in the schema. * **Prefer** `"systemName"` **over** `"name"` **in field-definition predicates.** System names are stable; display names change when someone renames a field. * **Cache pivots you reuse.** If a pivoted view backs a dashboard, materialize it on your side on a schedule that matches the [hourly refresh](#data-freshness). Avoid re-pivoting on every query. ## Troubleshooting \[#troubleshooting] > **Info** > > **A row count looks stale**: current-state tables refresh hourly. If a count is still wrong after an hour, and you are querying through a mounted listing, a Databricks foreign catalog, or another external catalog, refresh your client's metadata. Do this before concluding the data is missing. > **Info** > > **Authentication fails**: PATs expire one year after issue, and re-provisioning revokes the previous token. Confirm your token is current, and that your client sends it as the password. Snowflake and Iceberg clients must also set the role to `BENCHLING__RO`. For Databricks-specific errors, see [Databricks troubleshooting](#databricks-troubleshooting). For identifier-casing errors, missing tables, and schema-field pivot issues, see [Troubleshooting](/docs/direct-data-share-outbound-model-coverage#troubleshooting) on the Data Shape page.