Airbyte Sync API

Overview

The Airbyte Sync API lets an agency stand up managed data pipelines for its clients without touching the Airbyte UI. You provision a single BigQuery destination for the agency, then create one connection per client and ad platform. Mythic manages the underlying Airbyte workspace, sources, and connections on your behalf and records every sync outcome.

Behind the scenes each connection wires an Airbyte source (Google Ads, Meta Ads, and others) to your agency's BigQuery destination. Sync results stream back into an immutable event log that powers the statistics and health endpoints.

Base URL

All endpoints share the same base URL.

https://mythic-analytics.gulp.workers.dev/client/v1/airbyte

Key concepts

Typical workflow

Response envelope

Responses are JSON. Most endpoints wrap their payload in a data field; write and action endpoints add a success flag; list endpoints for sync events add pagination fields.

Example error response:

{
  "error": "Missing authorization"
}

Provisioning

POST /config/provision runs in one of two modes.

Create. Send bigquery_project_id, bigquery_dataset and gcp_credentials_json; Mythic creates an Airbyte workspace and a BigQuery destination for you.

Adopt. Send the ids of an Airbyte setup you already run, and Mythic uses it instead of building a parallel one:

When you adopt a destination, the BigQuery fields are not required: the destination already carries its own target and credentials. The id is verified at provision time, so a wrong id, or one belonging to another workspace, fails here rather than on your first connection.

Re-running provision

Provision is the natural thing to re-run when changing dataset or loading method, so it is built to repeat:

The response's destination_reused tells you which path ran, and GET /config reports destination_adopted so you always know whose destination is in play.

Staging is optional, and all-or-nothing

Standard inserts are the default: send no staging fields and the destination loads directly. To use GCS staging instead, send all three of staging_bucket_name, hmac_key_access_id and hmac_key_secret (staging_bucket_path defaults to staging). A partial set is rejected with 400 staging_incomplete naming what is missing.

The HMAC pair is a GCS interoperability credential created for a service account — gcloud storage hmac create SA_EMAIL — not the service-account JSON key. It does not have to belong to the same service account as gcp_credentials_json, but whichever account owns it needs write access to the staging bucket, so using the same one is simplest.

Failures

GET /config reports loading_method (Standard or GCS Staging), derived from the destination's real configuration — a reprovision that switches to standard inserts clears the staging bucket rather than leaving a stale one behind.

A provision that fails leaves no config row behind if there was none before, so GET /config keeps returning 404 not_configured — "never provisioned" and "provisioning failed" stay distinguishable. The failure response carries the reason itself (code: provision_failed with the upstream message, or a specific 400/409 code); you should never have to read it back off GET /config.

Supported platforms

The platform catalog is returned live by GET /platforms. At the time of writing it includes:

supportsOauth means no long-lived token, not no inputs

Every platform above is supportsOauth: true. That means the provider credential is obtained through consent and held by Airbyte — you never paste, store, or refresh an access or refresh token. It says nothing about the other fields a connector needs. Those you still send in source_config, in either flow.

manualFields on each catalog entry lists exactly what is left for you to supply — requiredFields minus everything flagged oauth: true:

Bing is the widest gap: fully OAuth-capable and still needs two inputs, because Microsoft's developer_token is issued to the application rather than granted by a user's consent, and its tenant_id selects which Microsoft tenant the user logs into. TikTok and Salesforce are the other end — consent supplies everything.

One connection, several ad accounts

Google Ads and Meta both let a single source pull from more than one ad account, and their catalog entries flag it with multiple: true. Send several however you like — a comma-separated string, a whitespace-separated one, or an array:

{ "platform": "meta_ads",   "source_config": { "account_id":  "act_111111111111111, act_222222222222222" } }
{ "platform": "google_ads", "source_config": { "customer_id": ["123-456-7890", "098-765-4321"] } }

Both reach the connector in the shape it actually requires, which differs between them and is not something you should have to know: Meta's account_ids is an array of bare numeric ids, Google's customer_id is one comma-separated string with no spaces. Formatting is normalised on the way through — Meta's act_ prefix is stripped, and dashes are removed from both, so act_123-456-789 and 123-456-7890 are accepted as typed.

A Google customer ID that is not exactly 10 digits after stripping is rejected with 400 invalid_customer_id, naming the offending value, rather than surfacing later as a schema error from inside Airbyte.

The connection's account_id records every account the source syncs, comma-separated — not just the first.

Every key in manualFields is a real property of the connector's own published spec — they are derived from it, not from what each ad platform's UI calls things, and a drift check fails if the catalog and the connector ever disagree. Two consequences worth knowing: bing_ads filters accounts by account_names (a list of { name, operator } predicates, optional) rather than by an account id, and tiktok_ads has no advertiser id in its OAuth flow at all.

Connecting with OAuth

Platforms whose catalog entry has supportsOauth: true can be connected without anyone handling a provider token. Fields the consent flow supplies are flagged oauth: true in requiredFields — never prompt for those. Everything in manualFields you must still collect and send in source_config.

A failed consent leaves nothing behind in Airbyte: if the connection cannot be created, the source created moments earlier is deleted too, so retrying does not accumulate half-built sources.

/oauth/initiate validates your request before it checks whether the agency is provisioned, so missing_account_id, return_url_not_allowed and unsupported_platform come back with their own codes even on an agency that has never provisioned. not_provisioned therefore means the request itself was fine.

mythic_error_code comes from a closed set you can branch on — state_invalid, missing_secret_id, missing_account_id, not_configured, not_provisioned, source_config_rejected, oauth_secret_expired, account_not_accessible, connection_failed. They differ in who can fix them:

A failed callback also writes a sync-event row with error_origin: oauth_callback, the code in error_type, and the underlying Airbyte message in error_message, readable at GET /sync-events?client_id=…. Because no connection exists yet, its connection_id is the sentinel oauth:<platform>:<client_id> rather than a UUID, so these rows are easy to filter. That is the only trail a failed consent leaves — no connection row is created — so surface it when a user comes back with mythic_oauth=error.

The state token is also your correlation handle: the connection id does not exist until the callback finishes, so hold state against whatever you are provisioning in your own UI.

Both ends are rate limited: 20 initiations per 10 minutes per agency, and 20 callbacks per 10 minutes per IP.

Stream sync modes

Every incremental stream is sent to Airbyte as incremental_deduped_history, which upserts on the stream's primary key. That matters for daily insights tables keyed on composites like (date_start, account_id, ad_id, region): a resync re-keys those rows rather than appending them, so spend is not double counted. Append-only incremental is not offered.

incremental and full_refresh are accepted as aliases of incremental_deduped_history and full_refresh_overwrite when you send a streams array, and older stored selections may still read back in the short form.

Stream names come from the connector, not the ad platform

A connector's stream names are its own, and they routinely differ from what the platform's UI calls the same thing — Meta's "Ad sets" is the stream ad_sets, not adsets. Airbyte rejects the first unrecognised name outright (No streams found with name [...]) and only reveals the next one after that is fixed.

So Mythic reconciles against the source's discovered catalog when a connection is created:

• You omit streams — the platform defaults are used, filtered to what the source actually offers. A default the connector has since renamed is dropped and logged rather than failing your connection.
• You supply streams — an unrecognised name is rejected with 400 unknown_streams and the response lists the names this source does offer. Nothing you explicitly asked for is ever silently dropped.

The stored connection's tables_json always reflects the streams actually configured, so read it back if you want to know what a connection ended up with.

Some streams sync weekly, on their own connection

A platform can also declare weeklyStreams in GET /platforms. Those streams are expensive to sync and slow to change — Meta's ad_creatives_from_ads is a full refresh of every creative on every ad — so they do not belong on the same cadence as daily insights.

Airbyte's cadence is a property of the connection, not the stream, so this is expressed the only way it can be: one POST /connections for Meta creates two connections off one source.

• Primary — the defaultStreams, on whatever sync_frequency you asked for.
• Weekly companion — the weeklyStreams only, manual in Airbyte, run by the pipeline one day a week (sync_dow, Sunday by default).

They share a source, so there is one consent, one credential and one account list. The response to POST /connections (and the connection created by an OAuth callback) carries a companions array with the second connection's row, plus a pipeline object saying whether both were added to the client's pipeline scope. Both stream sets are disjoint — no stream is ever written by both connections.

{
  "id": "…", "platform": "meta_ads", "tables_json": [ /* daily streams */ ],
  "companions": [
    { "id": "…", "display_name": "meta_ads - acme (weekly)", "sync_dow": 1,
      "tables_json": [{ "name": "ad_creatives_from_ads", "syncMode": "full_refresh_overwrite" }] }
  ],
  "pipeline": { "enrolled": true, "pipeline_id": "…", "connection_ids": ["…", "…"], "pipeline_enabled": true }
}

Three things follow from the split, all of them things a portal has to handle:

• Supplying your own streams opts out. An explicit stream list is a decision, so no companion is created — put the weekly streams wherever you want them.
• A companion needs the pipeline. It is manual in Airbyte on purpose; if the client has no pipeline, nothing will ever trigger it. It appears in GET /pipelines/{clientId}/candidates like any other connection — add it, or sync it by hand with POST /connections/{id}/sync.
• Deleting one does not delete the other. The shared source is removed only with the last connection using it; the response reports source_deleted and siblings_kept.

sync_dow (0 = Sunday … 6 = Saturday, null = every run) is writable on PUT /connections/{id} if Sunday is the wrong day. It is read by the pipeline only — a connection on its own Airbyte cron ignores it, because the cron already says when it runs.

Sync frequency

manual is the default. A connection created without an explicit sync_frequency does not sync on a schedule — you trigger it with POST /connections/{id}/sync. Scheduled syncing is something you opt into.

Choosing the hour

sync_hour_utc (0–23) sets when a cadence runs, in UTC. It is accepted on POST /connections, PUT /connections/{id} and PATCH /connections/{id}/schedule, and defaults to 2 when unset.

Sub-daily cadences anchor on hour % interval, so the hour you ask for is always one of the run times and the runs stay evenly spaced:

Responses from the schedule endpoints include runs_at_utc, the concrete list above, so you can tell a client when their data lands without decoding a cron expression. Changing sync_frequency alone keeps the connection's existing hour rather than resetting it.

Sync results

Every sync outcome Mythic reports comes from Airbyte's own workspace notifications, which it POSTs to Mythic when a job finishes. Provisioning registers that automatically; POST /config/notifications does it for a workspace provisioned earlier or one you adopted, and is idempotent.

Where to read what:

success is not the same as data arriving

A sync can succeed and commit zero records. In one measured week of real notifications that was 7 out of 221 — a revoked ad-account permission, an emptied date range, a paused campaign. Every status field calls those green, and a connection can stay green for months while its data quietly goes stale.

So each sync also carries landed: success and at least one record committed. Read landed when the question is "did the data arrive", and success only when the question is "did the job run". GET /health lists the difference under no_data_connections.

Two related distinctions in the same numbers:

• records_committed vs records_emitted. Emitted is what the source produced; committed is what the destination has. Bill, alert and chart on committed. A real failure in the sample emitted 1,485 records and committed none.
• consecutive_failures. One red run is a flake; four in a row is broken since Tuesday. A single event cannot tell those apart, so the streak is counted for you and failing_connections is ordered worst-first.

Backfilling history

POST /connections/{id}/backfill imports past jobs from Airbyte's jobs API for a connection that ran before notifications were registered. Backfilled rows are thinner — that API reports one combined row count with no emitted/committed split and no error classification, so records_emitted is null rather than a copy of the committed figure.

Both paths key on (connection_id, job_id), so a backfill never duplicates a notification, re-running one is a no-op, and a re-delivered notification never double-counts. Jobs still running are skipped rather than recorded as failures; the response reports how many under skipped_in_flight.

Getting sync results pushed to you

Airbyte allows exactly one webhook URL per event type per workspace, and Mythic holds it. Everything on this page — /sync-events, /stats, /health, last_sync, landed, consecutive_failures — is built from those posts, so a workspace pointed somewhere else reports a permanently empty, permanently healthy-looking history.

So instead of taking the slot, subscribe and Mythic relays to you:

curl -X POST https://mythic-analytics.gulp.workers.dev/client/v1/airbyte/webhooks \
  -H "Authorization: Bearer ak_..." -H "Content-Type: application/json" \
  -d '{"url": "https://you.example.com/hooks/airbyte?secret=...", "label": "portal"}'

Your endpoint receives Airbyte's payload verbatim under data, plus the client_id and platform Airbyte cannot know. The URL is treated as opaque and stored encrypted — put your own authentication in it, since Airbyte's notifications are themselves unsigned. GET /webhooks reads subscriptions back with every query-string value replaced by ***, alongside last_error and consecutive_failures.

Delivery is best-effort and there is no redelivery queue: the event is already durable, so reconcile a gap from GET /sync-events rather than waiting for a retry. An endpoint that fails 20 times in a row is parked (active: false) so a dead URL stops costing a request on every sync — POST the same URL again to resume it. Posting a URL that already exists updates it rather than adding a second subscription, which would deliver everything twice.

Orchestrated syncs

By default each connection runs on its own Airbyte schedule. That is fine until something has to run after the data lands — a transform, an export, your own pipeline. Twelve connections on twelve independent crons never produce a moment that means "today's data is here", so a consumer either guesses a time and is sometimes wrong, or polls forever.

A pipeline fixes that. Its connections go to manual, Mythic triggers them, and once the whole set is done Mythic POSTs to a URL you provide.

curl -X PUT https://mythic-analytics.gulp.workers.dev/client/v1/airbyte/pipelines/acme-retail \
  -H "Authorization: Bearer ak_..." -H "Content-Type: application/json" \
  -d '{"enabled": true, "connection_ids": ["...","..."], "run_hour_utc": 8,
       "webhook_url": "https://you.example.com/hooks/mythic",
       "webhook_secret": "a-secret-you-generate"}'

What "done" means

A run holds rather than lying. The rules, in order:

1. Skipped connections never gate. A connection disabled in Airbyte can't sync, and must not block the transform every day forever.
2. A first-time backfill doesn't gate — it can legitimately run for days and nothing downstream references it yet. Unless it failed, in which case that connection has never delivered anything and someone should know. A re-sync of a connection that had data always gates, because that is exactly when a consumer would publish half-rebuilt history as if it were complete.
3. Anything still running → wait.
4. Anything failed → hold. The run stays open, retries the failed steps every five minutes, and completes on its own the moment they pass. held is a parking state, not an outcome.
5. Otherwise → done, and the webhook fires.

Failed steps are retried twice, then wait for a person. POST .../retry resets them once the cause is fixed upstream; POST .../force closes a held run anyway.

Reading the payload

The body is {type: "airbyte.pipeline.completed", delivered_at, data} where data is a PipelineRun. Three fields deserve attention:

• complete, not status. They are different questions. status: "succeeded" with two connections skipped is a real and normal outcome; complete is what tells you whether everything that could sync, did.
• connections[].reason. skipped alone collapses "disabled on purpose", "inactive", "first-time backfill" and "not due this week" into one word, and only some of those mean data you expected is missing. A weekly connection reads weekly — only runs on Sunday (UTC) on the six days it is skipped; that is a complete run, not a partial one, and it never holds the gate.
• connections[].last_success_at. "All done" next to a source that last succeeded nine days ago is a very different message, and you cannot work that out from a run that skipped it today.

Delivery is at-least-once: dedupe on run_id and don't fail closed on a duplicate. A delivery that fails is retried automatically on later ticks, up to five attempts, after which it needs POST .../redeliver. Your endpoint has 10 seconds to respond — accept and process asynchronously rather than doing work inline, or you will be timed out and retried. A delivery failure marks the run but never un-finishes it — the syncs really did complete, and a run that flipped back to held because your endpoint was briefly down would be a lie about the data. POST .../redeliver re-sends on demand.

Timing and guarantees

Runs are advanced by a five-minute tick, so completion is detected within about five minutes of the last sync finishing — not instantly. Nothing anywhere is blocked waiting on a sync, which is what lets a first historical backfill run for days without holding a connection open.

If the worker is unavailable across your run_hour_utc, the run is started as soon as it is back — in the early hours of the next UTC day if the hour itself was missed — and stamped with the date it was for, so a day is never silently lost.

Exactly one run per client per day, enforced in the database. A run already in flight rejects a manual start with 409 run_in_flight, since a second run would stack a duplicate sync on every connection the first one owns. A run that somehow wedges — nothing in flight, no progress for 36 hours — fails itself loudly rather than leaving the client unable to ever refresh again.

Error handling