Integrating ClickHouse with Google Calendar
The Google Calendar registry item copies a canonical event reader with journaled page and sync-token progress into a chkit project.
This integration is part of chkit Sync (beta), the optional TypeScript ingestion layer built on chkit Schema.
Install
Section titled “Install”bunx chkit add google-calendar --with-testsbunx chkit checkbunx chkit generate --name add_google_calendarbunx chkit migrate --applybunx chkit ingest run --tag provider:google-calendarbunx chkit ingest status --tag provider:google-calendarSet GOOGLE_CALENDAR_ACCESS_TOKEN with calendar.readonly access before ingestion, or set accessToken in src/integrations/google-calendar/config.ts. Review installation settings and chunk limits in the same file. primary is resolved through calendar metadata to detect a different account reusing the checkpoint. Configuration and the token are checked when a stream runs: schema imports do not need credentials or call Google, and invalid settings or a missing token fail the stream before any request.
| Resource | Default ClickHouse table | Records synced | API reference |
|---|---|---|---|
Events (events) | google_calendar_events_raw | Canonical events, recurring masters, exceptions, and cancellation payloads from the configured calendar | GET/calendars/{calendarId}, GET/calendars/{calendarId}/events |
Pipeline and stream
Section titled “Pipeline and stream”One installation exports one pipeline with a canonical events stream. pipeline.ts defines its destination and strategy; sources/events.ts handles acknowledged sync-token progress, and client.ts parses the configuration and handles authorized Google requests, pagination, and payload validation.
sourceId identifies the installation in raw rows and checkpoint scope. streamPrefix determines journal IDs; defaults preserve google-calendar.events. A separate installation needs both a distinct source label and stream prefix. The pipeline factory snapshots configuration, including accessToken, while using the exported destination. Edit database in config.ts before schema imports to change table placement.
Sync behavior
Section titled “Sync behavior”The first run reads canonical events with recurring masters and exceptions, using singleEvents=false and showDeleted=true. Later runs request changes through Google’s syncToken. Cancellation payloads are stored without hydrating fields from prior observations. Recurring occurrences require a separate bounded instances reader or application projection.
See destinations.
Events are the only synced resource. Attendees, organizer, recurrence, reminders, attachments, and conferenceData stay inside the native event payload. Calendar metadata resolves identity without creating another raw table. Build projections and joins with other integrations in ClickHouse; retain source and calendar keys in those joins. A sparse cancellation replaces the previous observation of the same event without copying fields from its old payload or recurring master.
Each acknowledged page stores its next-page position and retains the same input sync token. Only a loaded terminal page promotes nextSyncToken, including an empty page. Rerun after interruption to resume; failed loads leave the previous checkpoint. Requests use fixed parameters, and validation rejects changed source scope or malformed pagination. Native JSON requires ClickHouse 25.3 or later.
Google’s 403 rate-limit responses (rateLimitExceeded, userRateLimitExceeded) are retried with backoff like 429. Other 403 responses and 401 for an expired token fail the run without retrying.
The reader uses paginate() for requests and continuation validation. Each full page includes raw events and a complete checkpoint candidate in metadata; the reader explicitly yields that candidate as chunk state. A rejected token becomes an empty reset page with a fresh continuation and retained recovery debt, using the same pagination loop. The existing checkpoint format is preserved.
Google forbids timeMin, timeMax, updatedMin, and orderBy on sync-token requests. This canonical reader rejects explicit --from/--to bounds. A named backfill without bounds runs an independent canonical sync. See Google synchronization and events.list parameters.
Expiry and stored data
Section titled “Expiry and stored data”410 Gone resets provider progress and starts a new baseline without deleting raw rows. A rejected page token replays its collection once. The destination keeps the latest observed payload per [sourceId, resolvedCalendarId, event.id]; absent records are not reconciled after a reset. A current-calendar view requires completed-baseline generations or explicit reconciliation before treating absence as removal.
The checkpoint retains the recovery count until the terminal sync page is acknowledged. Pauses and successful nonterminal replay pages keep that count; a second token rejection for the unfinished sync fails across executions. Adjust maxChunks in config.ts, execution duration, or polling frequency so the sync finishes while tokens remain valid. After reviewing coverage and correcting those conditions, migrate saved state explicitly or use a new stream identity for a fresh sync.
Version 0.2.0 changes the former rolling occurrence-window semantics and row identities. Keep old tables as archives and start a new canonical dataset, or migrate identities explicitly; old expanded occurrences are not removed automatically. The installed README describes upgrade and recovery details.
Fixture verification
Section titled “Fixture verification”bun test src/integrations/google-calendar/tests/basic.test.tsFixtures cover page resumption, empty deltas, sink failures before token promotion, cancellation payloads, expired tokens, and calendar-account changes. Schedule repeat runs externally; ChKit supports overlapping runs with independent journal histories.
Changelog
Section titled “Changelog”Version 0.3.1
- Trim raw-storage notes from the README.
Version 0.3.0
- Read configuration and the access token when a stream runs, so `generate`, `migrate`, and `chkit check` never depend on them; invalid settings or a missing token fail that run before any request. Requires zod 4 and chkit 0.2.0-beta.12.
- Send requests through the ingestion runtime, which retries only network failures, timeouts, retryable HTTP statuses, and Google's 403 rate-limit responses; malformed payloads and invalid checkpoints fail at once.
- Set `accessToken` in the configuration to use another token, for example for a second installation; it defaults to `GOOGLE_CALENDAR_ACCESS_TOKEN`.
Version 0.2.1
- Retry Google's 403 rateLimitExceeded and userRateLimitExceeded responses with backoff like 429 instead of failing the run after one attempt.
- Harden local maxChunks configuration validation by requiring a positive safe integer before pipeline construction.
- Clarify native JSON normalization of explicit null paths and empty objects in stored event observations.
Version 0.2.0
- Replace the rolling occurrence window with canonical events and incremental provider sync tokens, including cancellation payloads.
- Resume acknowledged pages, promote the terminal sync token after destination acknowledgement, and bound recovery from rejected or expired tokens.
- Scope row IDs to the installation and resolved calendar; migrate legacy occurrence identities or use a fresh table and stream when upgrading.
- Separate editable installation configuration, injectable HTTP clients, and resource readers from one pipeline; preserve independent stream identities and provider recovery.
- Keep events as the sole raw resource with native nested fields and sparse cancellation observations; verify recurring-exception replacement without hydration.
- Use full paginate pages and checkpoint metadata for canonical sync-token progress and bounded token recovery in one pagination loop, including empty terminal pages and the existing saved state format.
Version 0.1.0
- Introduce raw events ingestion over a rolling occurrence window.