Cloud storage buckets

A cloud storage destination writes files into your own bucket, laid out for a data lake: one file per dataset per run, in date folders, with a manifest written last. It works with Amazon S3 and any S3-compatible store — Google Cloud Storage, Cloudflare R2, Wasabi, Backblaze B2, DigitalOcean Spaces, MinIO and others. From the bucket, warehouse recipes load the files into Snowflake, Redshift, Databricks or DuckDB. Bucket destinations are part of the Agency plan.

Settings

Setting	Notes
Storage provider	Pick yours — iSearchFrom fills in its endpoint and addressing. See providers.
Bucket	The bucket name.
Region	The bucket’s region, e.g. eu-central-1. Not asked for Google Cloud Storage and Cloudflare R2.
Account ID	Cloudflare R2 only — it forms your bucket’s endpoint.
Endpoint	MinIO and other stores only: the store’s https:// S3 endpoint. For every other provider the endpoint in use is shown below the form.
Folder	Optional prefix inside the bucket, e.g. isearchfrom. Everything is written under it.
Path-style addressing	MinIO and other stores only: address the bucket as endpoint/bucket instead of bucket.endpoint.
Access key ID	The key iSearchFrom signs in with. It tells you which key is in use.
Secret access key	Stored encrypted and never shown again. When editing, leave it empty to keep the saved one.
Format	Parquet, CSV, JSON Lines or SQL. See file formats.

Test connection writes a small marker object, _isearchfrom_connection_test, under your folder and then deletes it. If the delete is not allowed the test still passes and the marker stays; it is harmless.

Providers

Provider	You enter	Endpoint used	Region	Addressing
Amazon S3	Region	Amazon’s own	The bucket’s, e.g. eu-central-1	Virtual-hosted
Google Cloud Storage	—	https://storage.googleapis.com	auto	Path-style
Cloudflare R2	Account ID	https://<account_id>.r2.cloudflarestorage.com	auto	Path-style
Wasabi	Region	https://s3.<region>.wasabisys.com	The bucket’s, e.g. eu-central-1	Path-style
Backblaze B2	Region	https://s3.<region>.backblazeb2.com	The bucket’s, e.g. us-west-004	Path-style
DigitalOcean Spaces	Region	https://<region>.digitaloceanspaces.com	The datacenter, e.g. nyc3	Virtual-hosted
MinIO	Endpoint and region	Yours	us-east-1 unless you changed it	Path-style
Other S3-compatible	Endpoint and region	Yours	What the store expects	Your choice

Every endpoint must be reachable from the internet over HTTPS — endpoints on private networks are refused.

What gets written

<folder>/<dataset>/dt=<YYYY-MM-DD>/<run_id>.<extension>
<folder>/_manifests/<run_id>.json

For example, with the folder isearchfrom:

isearchfrom/position_tracker-results/dt=2026-09-30/0199a7c2-5b1e-7c3a-9f10-3d2b8e6a4c11.csv.gz
isearchfrom/alerts/dt=2026-09-30/0199a7c2-5b1e-7c3a-9f10-3d2b8e6a4c11.csv.gz
isearchfrom/_manifests/0199a7c2-5b1e-7c3a-9f10-3d2b8e6a4c11.json

• <dataset> is the dataset id with . replaced by -: position_tracker.results → position_tracker-results.
• dt= is the UTC date the run’s window ends — Hive-style, so Athena, Spark, DuckDB and friends recognize it as a partition.
• One file per dataset per run. A retried run writes the same key again, so a retry overwrites rather than duplicates.
• <extension> follows the format: parquet, csv.gz, ndjson.gz or sql.gz. CSV, JSON Lines and SQL files are gzip-compressed and also stored with Content-Encoding: gzip, so some HTTP clients decompress them transparently on download. Parquet compresses itself and is never gzipped.

The manifest

The manifest is written after every file of the run is complete. If you see a manifest, every file it lists is whole — so load by manifest, not by listing the bucket.

{
  "run_id": "0199a7c2-5b1e-7c3a-9f10-3d2b8e6a4c11",
  "destination_id": "0199a2f0-1c4d-7e55-8a21-6b7f0c9d2e33",
  "window_from": "2026-09-30T05:00:00.000Z",
  "window_to": "2026-09-30T05:55:00.000Z",
  "format": "csv",
  "files": [
    {
      "dataset": "position_tracker.results",
      "key": "isearchfrom/position_tracker-results/dt=2026-09-30/0199a7c2-5b1e-7c3a-9f10-3d2b8e6a4c11.csv.gz",
      "rows": 1240,
      "bytes": 48213,
      "schema_version": 1,
      "columns": [
        {
          "name": "row_id",
          "type": "string",
          "nullable": false,
          "description": "Stable id of this row; unique within the dataset."
        }
      ]
    }
  ]
}

format is parquet, csv, ndjson, sql_postgres or sql_mysql. columns lists the columns in the file, in file order — every column of the dataset, or only the ones you picked — with its type, whether it can be empty, and its description, the same as the dataset reference. bytes is the size of the file as stored.

Files without a manifest

A run that fails part-way may leave files that no manifest lists. Its data is delivered again by the next run, under a new run_id. Ignore unlisted files, and de-duplicate rows on row_id.

Which format for a bucket

Parquet is the best choice for a data lake: typed columns, small files, and read directly by Snowflake, Redshift, Databricks, BigQuery, DuckDB and pandas. CSV, JSON Lines and SQL work too; Excel isn’t offered for buckets. See file formats.

Amazon S3

Create an IAM user (or role credentials) for iSearchFrom with write access to the folder only. Nothing needs to be read or listed:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "IsearchfromExports",
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:AbortMultipartUpload", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::your-bucket/isearchfrom/*"
    }
  ]
}

• s3:PutObject writes the files and manifests (large files are uploaded in parts, which this permission also covers).
• s3:AbortMultipartUpload cleans up an upload that was interrupted.
• s3:DeleteObject is only used to remove the connection-test marker. Leave it out if you prefer; the test still passes.

If the bucket uses SSE-KMS encryption, the credentials also need kms:GenerateDataKey and kms:Decrypt on the key (multipart uploads need both).

Choose Amazon S3 as the storage provider and set Region to the bucket’s region.

Google Cloud Storage

Google Cloud Storage takes S3-compatible requests through its XML API, signed with an HMAC key:

1. Create a service account for iSearchFrom, and on the bucket grant it Storage Object User (roles/storage.objectUser). Storage Object Creator alone is not enough: a retried run replaces its own file, which needs permission to overwrite.
2. In the Cloud console, open Cloud Storage → Settings → Interoperability and create an HMAC key for that service account.
3. Choose Google Cloud Storage as the storage provider, and enter the HMAC key’s access ID as Access key ID and its secret as Secret access key.

The endpoint (https://storage.googleapis.com), region (auto) and path-style addressing are set for you.

Cloudflare R2

1. In the Cloudflare dashboard, create an R2 API token with Object Read & Write, limited to the bucket.
2. Choose Cloudflare R2 and enter your Account ID from the R2 dashboard. The endpoint https://<account_id>.r2.cloudflarestorage.com and region auto follow from it.
3. Use the token’s access key ID and secret access key.

Wasabi

Create an access key in the Wasabi console — ideally for a sub-user whose policy only allows writing to the bucket — choose Wasabi, and set Region to the bucket’s region, e.g. us-east-1 or eu-central-1.

Backblaze B2

Create an application key with Read and Write access, restricted to the bucket. Choose Backblaze B2, and set Region from the bucket’s S3 endpoint: for s3.us-west-004.backblazeb2.com the region is us-west-004. The key’s keyID is the Access key ID and its applicationKey the Secret access key.

DigitalOcean Spaces

Create a Spaces access key with read and write access to the bucket, choose DigitalOcean Spaces, and set Region to the datacenter the Space lives in, e.g. nyc3, ams3 or fra1.

MinIO and other S3-compatible stores

Set Endpoint to the store’s public https:// URL and Region to the one it expects (MinIO uses us-east-1 unless configured otherwise). Path-style addressing is on for MinIO; for other stores, turn it on unless the store supports bucket subdomains.

Troubleshooting

Code	Likely cause
not_found	The bucket does not exist, or the region or endpoint points somewhere else.
access_denied	Wrong access key or secret, or the policy doesn’t allow writing to the folder.
destination_rejected	The store refused the request — check the region, path-style setting and bucket policies.
destination_unreachable	The endpoint could not be reached, or it is on a private network. Temporary errors are retried.

See all error codes.

Related docs

• Exporting your data
• Dataset reference
• File formats
• Warehouse recipes