Dataform (Google Cloud)

Dataform is Google Cloud’s native SQL transformation framework for BigQuery, positioned as a free alternative to dbt Cloud for teams committed to BigQuery.

Reviewed by 7wData
API Available

On this page

Publisher review

Dataform is Google Cloud's native SQL transformation framework for BigQuery, positioned as a free alternative to dbt Cloud for teams committed to BigQuery. Founded in 2018 by ex-Googlers Lewis Hemens and Guillaume-Henri Huon, the London-based startup was acquired by Google in December 2020 and is now offered at no additional charge within Google Cloud. It handles the transformation layer of data pipelines through SQLX (SQL with JavaScript templating), enabling teams to version control transformations via Git, define dependencies between tables using ref() functions, and execute automated testing through assertions.

The service includes a web-based IDE, built-in scheduling, real-time lineage visualization, and as of late 2025, AI-assisted code generation via Gemini. For BigQuery-exclusive teams, the cost advantage is substantial—no per-user licensing, only BigQuery compute charges—which can save $8,000+ annually compared to dbt Cloud's $100/month-per-seat model. However, Dataform remains tightly bound to BigQuery; there is no support for Snowflake, Redshift, Databricks, or Postgres when using the managed service.

This creates a critical trade-off: teams get simplicity, cost-effectiveness, and deep BigQuery integration at the cost of warehouse portability. The open-source Dataform Core provides some flexibility for local use, but the managed service is where most adoption concentrates. Community discussion emphasizes this lock-in risk and the smaller ecosystem compared to dbt's established plugin libraries and hiring-market recognition.

Get the AI & data signal, daily.

335k+ subscribers read this every morning. One email, both newsletters. Unsubscribe anytime.

How it works

  1. SQL-based transformations with JavaScript templating

    Define data models and transformations in .sqlx files combining SQL with JavaScript, enabling dynamic feature generation without external templating languages.

  2. Git-native version control and collaboration

    Integrated Git workflows with GitHub, GitLab, Azure DevOps, and Bitbucket; all transformations stored as code in repositories with branch-based development.

  3. Dependency management and execution orchestration

    Automatic compilation of dependency graphs; ref() function establishes table-to-table lineage; Dataform determines correct execution order without manual workflow definition.

  4. Built-in data quality testing via assertions

    Define assertions (uniqueness, null checks, custom SQL conditions) that run after each transformation; failures halt downstream pipelines and alert teams.

  5. Web-based IDE with real-time validation

    Browser-based editor with live query compilation, error detection, and preview functionality; no local development environment or CLI required.

  6. Automatic metadata and lineage tracking

    Column-level data lineage captured automatically and integrated with BigQuery console and Dataplex; lineage sent to Data Lineage API after each run.

  7. Gemini AI-assisted code generation

    Generate SQLX models from natural language descriptions; auto-document models and suggest test assertions using Google's Gemini models (preview as of late 2025).

Strengths and trade-offs

Strengths

  • Zero licensing cost for BigQuery teams; only pay for BigQuery compute, saving $8k–9k annually versus dbt Cloud.
  • Native BigQuery integration with automatic lineage visualization in the BigQuery console and no separate authentication setup.
  • Built-in scheduling, Git integration, and collaborative web IDE eliminate orchestration and version-control toolchain dependencies.

Trade-offs

  • Locked to BigQuery only; no support for Snowflake, Redshift, Databricks, or other warehouses when using the managed service.
  • Smaller community and ecosystem compared to dbt; fewer published packages, integrations, and recognized industry standard for hiring.
  • Misleading 'free' messaging obscures hidden BigQuery compute costs, which scale with query complexity and can surprise teams at scale.

Pricing context

Dataform itself is offered at no charge within Google Cloud Platform. However, costs accrue from dependent services: BigQuery (primary cost, charged per query execution), Cloud Logging (required for all workflow invocations), and optional services like Cloud Scheduler, Cloud Workflows, and Managed Service for Apache Airflow. For teams comparing to dbt Cloud—which charges $100/month per developer seat—Dataform can save $6,000–9,000 annually for small-to-mid-sized teams. There are no Dataform-specific pricing tiers, per-user fees, or workspace subscriptions.

Getting started with Dataform (Google Cloud)

  1. Enable Dataform in GCP

    Open the Google Cloud Console, navigate to the Dataform API page, and enable the service for your project. Ensure BigQuery and Cloud Logging APIs are also enabled, as they are required dependencies for Dataform to execute transformations.

  2. Connect a Git repository

    In the Dataform web interface, create a new repository and link it to your Git provider (GitHub, GitLab, Azure DevOps, or Bitbucket). Authenticate using a personal access token or SSH key, then select a branch to serve as your main development branch.

  3. Define your first SQLX model

    In the repository, create a new .sqlx file. Write a SELECT statement that references an existing BigQuery table using the ref() function to establish dependencies. Save the file and commit it to your Git branch to trigger automatic compilation.

  4. Run a compilation and preview

    Click the Compile button in the Dataform IDE to generate the dependency graph and compiled SQL. Review the execution order and preview the output in BigQuery to verify the transformation logic before scheduling.

  5. Schedule a recurring execution

    In the Dataform interface, create a release configuration and set a schedule (e.g., daily at 6 AM UTC). Attach the release to a compilation result, then enable the schedule to automate transformation runs. Monitor execution logs in Cloud Logging.

Frequently Asked Questions

What is Dataform on Google Cloud?

Dataform is Google Cloud's free SQL transformation framework for BigQuery. It handles the transformation layer of data pipelines using SQLX, which combines SQL with JavaScript templating. It includes a web-based IDE, Git version control, and built-in scheduling.

How does Dataform pricing compare to dbt Cloud?

Dataform itself is free, with costs only from BigQuery compute and Cloud Logging. For a seven-person team, this can save $8,000 to $9,000 annually compared to dbt Cloud's $100 per developer seat per month model. There are no per-user fees or workspace subscriptions.

What are the main features of Dataform?

Dataform offers SQL-based transformations with JavaScript templating, Git-native version control, automatic dependency management, built-in data quality testing via assertions, a web-based IDE with real-time validation, and column-level lineage tracking. As of late 2025, it also includes Gemini AI-assisted code generation.

What is the BigQuery lock-in issue with Dataform?

Dataform's managed service only supports BigQuery, not Snowflake, Redshift, Databricks, or Postgres. This lock-in means teams sacrifice warehouse portability for deep BigQuery integration and cost savings. The open-source Dataform Core offers some flexibility for local use, but most adoption is on the managed service.

How does Dataform handle data quality testing?

Dataform allows you to define assertions for data quality, such as uniqueness checks, null checks, and custom SQL conditions. These assertions run after each transformation, and failures halt downstream pipelines and alert teams. This ensures data integrity without manual validation.

Does Dataform have AI features for code generation?

Yes, as of late 2025, Dataform includes Gemini AI-assisted code generation in preview. You can generate SQLX models from natural language descriptions, auto-document models, and suggest test assertions. This feature uses Google's Gemini models to streamline transformation development.

Alternatives

How Dataform (Google Cloud) compares

Direct head-to-head against 3 competitors. Picked by 7wData.

This tool

Dataform (Google Cloud)

Pricing
Dataform itself is offered at no charge within Google Cloud Platform. However, costs accrue from dependent services: BigQuery (primary cost, charged per query execution), Cloud Logging (required for all workflow invocations), and optional services like Cloud Scheduler, Cloud Workflows, and Managed Service for Apache Airflow. For teams comparing to dbt Cloud—which charges $100/month per developer seat—Dataform can save $6,000–9,000 annually for small-to-mid-sized teams. There are no Dataform-specific pricing tiers, per-user fees, or workspace subscriptions.
Target
Dataform is Google Cloud's native SQL transformation framework for BigQuery, positioned as a free alternative to dbt Cloud for teams committed to BigQuery.
Strength
Zero licensing cost for BigQuery teams; only pay for BigQuery compute, saving $8k–9k annually versus dbt Cloud.
Watch for
Locked to BigQuery only; no support for Snowflake, Redshift, Databricks, or other warehouses when using the managed service.

dbt Cloud

Pricing
Starts at $100/month for Developer plan; Team plan $150/user/month.
Target
Data analysts and engineers building SQL-first transformation pipelines.
Deployment
Multi-cloud SaaS
Strength
Largest community of SQL transformation practitioners and pre-built packages.
Watch for
Pricing escalates quickly with user count; complex multi-project setups require Enterprise plan.

Apache Spark

Pricing
Open source; cloud services (e.g., Dataproc) charge per cluster hour.
Target
Data engineers needing distributed processing for large-scale ETL and ML.
Deployment
Self-managed or cloud
Strength
Handles massive datasets with in-memory processing and advanced analytics.
Watch for
Steep learning curve; requires cluster management and tuning for performance.

Google BigQuery

Pricing
$5/TB processed for queries; storage $0.02/GB/month; flat-rate slots available.
Target
Analysts and engineers querying large datasets with serverless SQL.
Deployment
Serverless cloud
Strength
No infrastructure management; auto-scales to petabyte-scale queries.
Watch for
Costs can spike with unoptimized queries; limited to SQL-only transformations.

User reviews

No user reviews yet. Be the first to write one.

Sources

Reporting on this tool draws on these publicly available sources.

  1. cloud.google.com — Dataform is free; costs come from BigQuery, Cloud Logging, and optional orchestration services
  2. docs.cloud.google.com — Dataform is a service for developing, testing, versioning, and scheduling transformations in BigQuery
  3. valiotti.com — Key differences between Dataform and dbt; Dataform's BigQuery lock-in and cost advantage; smaller ecosystem and community
  4. www.thedataletter.com — Trade-offs between Dataform and dbt; hidden BigQuery compute costs; ecosystem differences
  5. medium.com — Pricing comparison ($8k annual savings for seven-person team using Dataform vs dbt Cloud); advantages and disadvantages of lock-in
  6. docs.cloud.google.com — Gemini AI assistance, strict act-as mode, BigLake/Iceberg support, and 2025–2026 feature releases
  7. techcrunch.com — Dataform acquisition by Google Cloud in December 2020; founding details and acquisition context
  8. www.ycombinator.com — Dataform founders (Lewis Hemens, Guillaume-Henri Huon), Y Combinator Winter 2018 batch, London-based startup