Airbyte Sync API

Overview

The Airbyte Sync API lets an agency stand up managed data pipelines for its clients without touching the Airbyte UI. You provision a single BigQuery destination for the agency, then create one connection per client and ad platform. Mythic manages the underlying Airbyte workspace, sources, and connections on your behalf and records every sync outcome.

Behind the scenes each connection wires an Airbyte source (Google Ads, Meta Ads, and others) to your agency's BigQuery destination. Sync results stream back into an immutable event log that powers the statistics and health endpoints.

Base URL

All endpoints share the same base URL.

https://mythic-analytics.gulp.workers.dev/client/v1/airbyte

Key concepts

Typical workflow

Response envelope

Responses are JSON. Most endpoints wrap their payload in a data field; write and action endpoints add a success flag; list endpoints for sync events add pagination fields.

Example error response:

{
  "error": "Missing authorization"
}

Provisioning

POST /config/provision runs in one of three modes.

Mythic's warehouse (default). Happens automatically on your first connect if you never provisioned. Or send no BigQuery fields. Mythic creates an Airbyte workspace and a destination in its own BigQuery project, with a credential Mythic holds. GET /config then reports warehouse: "mythic" and has_credentials: false. Not available together with your own airbyte_client_id (400 warehouse_requires_mythic_org).

Your own BigQuery. Send bigquery_project_id, bigquery_dataset and gcp_credentials_json, all three. Mythic creates the workspace and a destination in your project. warehouse is "agency".

Adopt. Send the ids of an Airbyte setup you already run, and Mythic uses it instead of building a parallel one:

When you adopt a destination, the BigQuery fields are not required: the destination already carries its own target and credentials. The id is verified at provision time, so a wrong id, or one belonging to another workspace, fails here rather than on your first connection.

One dataset per client

Whichever warehouse you use, every connection writes its own dataset, client_<client id with dashes replaced by underscores>, and every table is prefixed with the platform:

client_538d4ced_447c_40b9_9afa_d220f9afcc84.meta_ads_campaigns
client_538d4ced_447c_40b9_9afa_d220f9afcc84.meta_ads_ads_insights_action_type
client_538d4ced_447c_40b9_9afa_d220f9afcc84.google_ads_ad_groups

The connection row says where: bigquery_project_id, bigquery_dataset, table_prefix. Connections created before 2026-09-30 have these as null and keep writing the destination's default dataset under bare stream names. On Mythic's warehouse this is the same dataset as the client's BigQuery export, so an export reader also sees the synced ad tables.

Correlation, constraints and incrementality read a client's Meta tables from this dataset once its connection has synced at least once.

Moving between warehouses

An agency that already has a destination cannot be switched between Mythic's warehouse and its own BigQuery by accident. Provision answers 409 warehouse_change unless you send confirm_warehouse_change: true, because existing connections would write to the new place from their next sync, without their history.

Re-running provision

Provision is the natural thing to re-run when changing dataset or loading method, so it is built to repeat:

The response's destination_reused tells you which path ran, and GET /config reports destination_adopted so you always know whose destination is in play.

Staging is optional, and all-or-nothing

Standard inserts are the default: send no staging fields and the destination loads directly. To use GCS staging instead, send all three of staging_bucket_name, hmac_key_access_id and hmac_key_secret (staging_bucket_path defaults to staging). A partial set is rejected with 400 staging_incomplete naming what is missing.

The HMAC pair is a GCS interoperability credential created for a service account — gcloud storage hmac create SA_EMAIL — not the service-account JSON key. It does not have to belong to the same service account as gcp_credentials_json, but whichever account owns it needs write access to the staging bucket, so using the same one is simplest.

Failures

GET /config reports loading_method (Standard or GCS Staging), derived from the destination's real configuration — a reprovision that switches to standard inserts clears the staging bucket rather than leaving a stale one behind.

A provision that fails leaves no config row behind if there was none before, so GET /config keeps returning 404 not_configured — "never provisioned" and "provisioning failed" stay distinguishable. The failure response carries the reason itself (code: provision_failed with the upstream message, or a specific 400/409 code); you should never have to read it back off GET /config.

Supported platforms

The platform catalog is returned live by GET /platforms. At the time of writing it includes:

supportsOauth means no long-lived token, not no inputs

Every platform above is supportsOauth: true. That means the provider credential is obtained through consent and held by Airbyte — you never paste, store, or refresh an access or refresh token. It says nothing about the other fields a connector needs. Those you still send in source_config, in either flow.

manualFields on each catalog entry lists exactly what is left for you to supply — requiredFields minus everything flagged oauth: true:

Bing is the widest gap: fully OAuth-capable and still needs two inputs, because Microsoft's developer_token is issued to the application rather than granted by a user's consent, and its tenant_id selects which Microsoft tenant the user logs into. TikTok and Salesforce are the other end — consent supplies everything.

One connection, several ad accounts

Google Ads and Meta both let a single source pull from more than one ad account, and their catalog entries flag it with multiple: true. Send several however you like — a comma-separated string, a whitespace-separated one, or an array:

{ "platform": "meta_ads",   "source_config": { "account_id":  "act_111111111111111, act_222222222222222" } }
{ "platform": "google_ads", "source_config": { "customer_id": ["123-456-7890", "098-765-4321"] } }

Both reach the connector in the shape it actually requires, which differs between them and is not something you should have to know: Meta's account_ids is an array of bare numeric ids, Google's customer_id is one comma-separated string with no spaces. Formatting is normalised on the way through — Meta's act_ prefix is stripped, and dashes are removed from both, so act_123-456-789 and 123-456-7890 are accepted as typed.

A Google customer ID that is not exactly 10 digits after stripping is rejected with 400 invalid_customer_id, naming the offending value, rather than surfacing later as a schema error from inside Airbyte.

The connection's account_id records every account the source syncs, comma-separated — not just the first.

Every key in manualFields is a real property of the connector's own published spec — they are derived from it, not from what each ad platform's UI calls things, and a drift check fails if the catalog and the connector ever disagree. Two consequences worth knowing: bing_ads filters accounts by account_names (a list of { name, operator } predicates, optional) rather than by an account id, and tiktok_ads has no advertiser id in its OAuth flow at all.

Connecting with OAuth

Platforms whose catalog entry has supportsOauth: true can be connected without anyone handling a provider token. Fields the consent flow supplies are flagged oauth: true in requiredFields — never prompt for those. Everything in manualFields you must still collect and send in source_config.

A failed consent leaves nothing behind in Airbyte: if the connection cannot be created, the source created moments earlier is deleted too, so retrying does not accumulate half-built sources.

/oauth/initiate validates your request before it checks whether the agency is provisioned, so missing_account_id, return_url_not_allowed and unsupported_platform come back with their own codes even on an agency that has never provisioned. not_provisioned therefore means the request itself was fine.

mythic_error_code comes from a closed set you can branch on — state_invalid, missing_secret_id, missing_account_id, not_configured, not_provisioned, source_config_rejected, oauth_secret_expired, account_not_accessible, connection_failed. They differ in who can fix them:

A failed callback also writes a sync-event row with error_origin: oauth_callback, the code in error_type, and the underlying Airbyte message in error_message, readable at GET /sync-events?client_id=…. Because no connection exists yet, its connection_id is the sentinel oauth:<platform>:<client_id> rather than a UUID, so these rows are easy to filter. That is the only trail a failed consent leaves — no connection row is created — so surface it when a user comes back with mythic_oauth=error.

The state token is also your correlation handle: the connection id does not exist until the callback finishes, so hold state against whatever you are provisioning in your own UI.

Both ends are rate limited: 20 initiations per 10 minutes per agency, and 20 callbacks per 10 minutes per IP.

Stream sync modes

Every incremental stream is sent to Airbyte as incremental_deduped_history, which upserts on the stream's primary key. That matters for daily insights tables keyed on composites like (date_start, account_id, ad_id, region): a resync re-keys those rows rather than appending them, so spend is not double counted. Append-only incremental is not offered.

incremental and full_refresh are accepted as aliases of incremental_deduped_history and full_refresh_overwrite when you send a streams array, and older stored selections may still read back in the short form.

Stream names come from the connector, not the ad platform

A connector's stream names are its own, and they routinely differ from what the platform's UI calls the same thing — Meta's "Ad sets" is the stream ad_sets, not adsets. Airbyte rejects the first unrecognised name outright (No streams found with name [...]) and only reveals the next one after that is fixed.

So Mythic reconciles against the source's discovered catalog when a connection is created:

• You omit streams — the platform defaults are used, filtered to what the source actually offers. A default the connector has since renamed is dropped and logged rather than failing your connection.
• You supply streams — an unrecognised name is rejected with 400 unknown_streams and the response lists the names this source does offer. Nothing you explicitly asked for is ever silently dropped.

The stored connection's tables_json always reflects the streams actually configured, so read it back if you want to know what a connection ended up with.

Meta is one connection

ad_creatives_from_ads syncs incrementally (cursor updated_time, key id) on the same daily connection as the other Meta streams. activities (the account change log) syncs incrementally on the same connection, cursor event_time, key account_id, actor_id, application_id, event_type, object_id, event_time. The connector declares no key for it, so Mythic sends one. event_time is in the key so repeated edits are kept as separate rows.

Meta connections created before 2026-09-30 are two connections off one source: the daily streams, and a companion carrying only ad_creatives_from_ads. The companion is not a duplicate. Mythic never adds a stream to one of the pair that the other already syncs, and deleting one keeps the shared source for the other (the response reports source_deleted and siblings_kept).

sync_dow (0 = Sunday … 6 = Saturday, null = every run) on PUT /connections/{id} limits a connection to one pipeline run a week. It is read by the pipeline only. A connection on its own Airbyte cron ignores it.

Sync frequency

The pipeline is the default. A connection created without an explicit sync_frequency is created manual and joins the client's pipeline — created automatically, enabled and daily, with the client's first connection. The pipeline runs it daily, and an Airbyte cron on top would sync it twice. The response carries pipeline.

An explicit sync_frequency always wins and opts out of the pipeline entirely: no pipeline is created for it and nothing is enrolled.

Choosing the hour

sync_hour_utc (0–23) sets when a cadence runs, in UTC. It is accepted on POST /connections, PUT /connections/{id} and PATCH /connections/{id}/schedule, and defaults to 2 when unset.

Sub-daily cadences anchor on hour % interval, so the hour you ask for is always one of the run times and the runs stay evenly spaced:

Responses from the schedule endpoints include runs_at_utc, the concrete list above, so you can tell a client when their data lands without decoding a cron expression. Changing sync_frequency alone keeps the connection's existing hour rather than resetting it.

Sync results

Every sync outcome Mythic reports comes from Airbyte's own workspace notifications, which it POSTs to Mythic when a job finishes. Provisioning registers that automatically; POST /config/notifications does it for a workspace provisioned earlier or one you adopted, and is idempotent.

Where to read what:

success is not the same as data arriving

A sync can succeed and commit zero records. In one measured week of real notifications that was 7 out of 221 — a revoked ad-account permission, an emptied date range, a paused campaign. Every status field calls those green, and a connection can stay green for months while its data quietly goes stale.

So each sync also carries landed: success and at least one record committed. Read landed when the question is "did the data arrive", and success only when the question is "did the job run". GET /health lists the difference under no_data_connections.

Two related distinctions in the same numbers:

• records_committed vs records_emitted. Emitted is what the source produced; committed is what the destination has. Bill, alert and chart on committed. A real failure in the sample emitted 1,485 records and committed none.
• consecutive_failures. One red run is a flake; four in a row is broken since Tuesday. A single event cannot tell those apart, so the streak is counted for you and failing_connections is ordered worst-first.

Backfilling history

POST /connections/{id}/backfill imports past jobs from Airbyte's jobs API for a connection that ran before notifications were registered. Backfilled rows are thinner — that API reports one combined row count with no emitted/committed split and no error classification, so records_emitted is null rather than a copy of the committed figure.

Both paths key on (connection_id, job_id), so a backfill never duplicates a notification, re-running one is a no-op, and a re-delivered notification never double-counts. Jobs still running are skipped rather than recorded as failures; the response reports how many under skipped_in_flight.

Getting sync results pushed to you

Airbyte allows exactly one webhook URL per event type per workspace, and Mythic holds it. Everything on this page — /sync-events, /stats, /health, last_sync, landed, consecutive_failures — is built from those posts, so a workspace pointed somewhere else reports a permanently empty, permanently healthy-looking history.

So instead of taking the slot, subscribe and Mythic relays to you:

curl -X POST https://mythic-analytics.gulp.workers.dev/client/v1/airbyte/webhooks \
  -H "Authorization: Bearer ak_..." -H "Content-Type: application/json" \
  -d '{"url": "https://you.example.com/hooks/airbyte?secret=...", "label": "portal"}'

Your endpoint receives Airbyte's payload verbatim under data, plus the client_id and platform Airbyte cannot know. The URL is treated as opaque and stored encrypted — put your own authentication in it, since Airbyte's notifications are themselves unsigned. GET /webhooks reads subscriptions back with every query-string value replaced by ***, alongside last_error and consecutive_failures.

Delivery is best-effort and there is no redelivery queue: the event is already durable, so reconcile a gap from GET /sync-events rather than waiting for a retry. An endpoint that fails 20 times in a row is parked (active: false) so a dead URL stops costing a request on every sync — POST the same URL again to resume it. Posting a URL that already exists updates it rather than adding a second subscription, which would deliver everything twice.

Orchestrated syncs

By default each connection runs on its own Airbyte schedule. A pipeline runs one client's connections as a single daily run and POSTs a signed webhook once all of them have synced, so downstream work starts when the data is actually in. Setup, run states, handling held runs, the webhook payload and signature verification are on the Pipelines page.

Error handling