Google¶

Airflow has support for the Google service.

All hooks are based on airflow.providers.google.common.hooks.base_google.GoogleBaseHook. Some integration also use airflow.providers.google.common.hooks.discovery_api.

See the Google Cloud connection type documentation to configure connections to Google services.

Google Cloud¶

Airflow has extensive support for the Google Cloud.

Note

You can learn how to use Google Cloud integrations by analyzing the source code of the Google Cloud example DAGs

Services¶

These integrations allow you to perform various operations within the Google Cloud.

Transfers¶

These integrations allow you to copy data from/to Google Cloud.

Google Cloud Storage (GCS) to Amazon Simple Storage Service (S3)¶

Source product documentation: Google Cloud Storage (GCS)
Target product documentation: Amazon Simple Storage Service (S3)
Operator guide: Google Cloud Storage to Amazon S3 transfer operator
Python API: airflow.providers.amazon.aws.transfers.gcs_to_s3
Provider: apache-airflow-providers-amazon

Amazon Glacier to Google Cloud Storage (GCS)¶

Source product documentation: Amazon Glacier
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Amazon S3 Glacier to GCS
Python API: airflow.providers.amazon.aws.transfers.glacier_to_gcs
Provider: apache-airflow-providers-amazon

Presto to Google Cloud Storage (GCS)¶

Source product documentation: Presto
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Presto to Google Cloud Storage Transfer Operator
Python API: airflow.providers.google.cloud.transfers.presto_to_gcs
Provider: apache-airflow-providers-google

Trino to Google Cloud Storage (GCS)¶

Source product documentation: Trino
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Trino to Google Cloud Storage Transfer Operator
Python API: airflow.providers.google.cloud.transfers.trino_to_gcs
Provider: apache-airflow-providers-google

Common SQL to Google Cloud Storage (GCS)¶

Source product documentation: Common SQL
Target product documentation: Google Cloud Storage (GCS)
Python API: airflow.providers.google.cloud.transfers.sql_to_gcs
Provider: apache-airflow-providers-google

Google Cloud Storage (GCS) to Google Drive¶

Source product documentation: Google Cloud Storage (GCS)
Target product documentation: Google Drive
Operator guide: Google Cloud Storage to Google Drive Transfer Operators
Python API: airflow.providers.google.suite.transfers.gcs_to_gdrive
Provider: apache-airflow-providers-google

Google Drive to Google Cloud Storage (GCS)¶

Source product documentation: Google Drive
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Google Drive to Google Cloud Storage Transfer Operator
Python API: airflow.providers.google.cloud.transfers.gdrive_to_gcs
Provider: apache-airflow-providers-google

Microsoft Azure FileShare to Google Cloud Storage (GCS)¶

Source product documentation: Microsoft Azure FileShare
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Transfers data from Azure FileShare Storage to Google Cloud Storage
Python API: airflow.providers.google.cloud.transfers.azure_fileshare_to_gcs
Provider: apache-airflow-providers-google

Apache Cassandra to Google Cloud Storage (GCS)¶

Source product documentation: Apache Cassandra
Target product documentation: Google Cloud Storage (GCS)
Python API: airflow.providers.google.cloud.transfers.cassandra_to_gcs
Provider: apache-airflow-providers-google

Google Calendar to Google Cloud Storage (GCS)¶

Source product documentation: Google Calendar
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Google Calendar to Google Cloud Storage Transfer Operators
Python API: airflow.providers.google.cloud.transfers.calendar_to_gcs
Provider: apache-airflow-providers-google

Google Spreadsheet to Google Cloud Storage (GCS)¶

Source product documentation: Google Spreadsheet
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Google Sheets to Google Cloud Storage Transfer Operators
Python API: airflow.providers.google.cloud.transfers.sheets_to_gcs
Provider: apache-airflow-providers-google

Amazon Simple Storage Service (S3) to Google Cloud Storage (GCS)¶

Source product documentation: Amazon Simple Storage Service (S3)
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Transfer Data from Amazon S3 to Google Cloud Storage
Python API: airflow.providers.google.cloud.transfers.s3_to_gcs
Provider: apache-airflow-providers-google

Google Cloud Storage (GCS) to SSH File Transfer Protocol (SFTP)¶

Source product documentation: Google Cloud Storage (GCS)
Target product documentation: SSH File Transfer Protocol (SFTP)
Operator guide: Google Cloud Storage Transfer Operator to SFTP
Python API: airflow.providers.google.cloud.transfers.gcs_to_sftp
Provider: apache-airflow-providers-google

PostgreSQL to Google Cloud Storage (GCS)¶

Source product documentation: PostgreSQL
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Postgres To Google Cloud Storage Operator
Python API: airflow.providers.google.cloud.transfers.postgres_to_gcs
Provider: apache-airflow-providers-google

Google BigQuery to Common SQL¶

Source product documentation: Google BigQuery
Target product documentation: Common SQL
Python API: airflow.providers.google.cloud.transfers.bigquery_to_sql
Provider: apache-airflow-providers-google

Google BigQuery to MySQL¶

Source product documentation: Google BigQuery
Target product documentation: MySQL
Operator guide: Google Cloud BigQuery Transfer Operator to MySQL
Python API: airflow.providers.google.cloud.transfers.bigquery_to_mysql
Provider: apache-airflow-providers-google

Google BigQuery to Microsoft SQL Server (MSSQL)¶

Source product documentation: Google BigQuery
Target product documentation: Microsoft SQL Server (MSSQL)
Operator guide: Google Cloud BigQuery Transfer Operator to Microsoft SQL Server
Python API: airflow.providers.google.cloud.transfers.bigquery_to_mssql
Provider: apache-airflow-providers-google

Google BigQuery to PostgreSQL¶

Source product documentation: Google BigQuery
Target product documentation: PostgreSQL
Operator guide: Google Cloud BigQuery Transfer Operator to Postgres
Python API: airflow.providers.google.cloud.transfers.bigquery_to_postgres
Provider: apache-airflow-providers-google

Google Cloud Storage (GCS) to Google BigQuery¶

Source product documentation: Google Cloud Storage (GCS)
Target product documentation: Google BigQuery
Operator guide: Google Cloud Storage Transfer Operator to BigQuery
Python API: airflow.providers.google.cloud.transfers.gcs_to_bigquery
Provider: apache-airflow-providers-google

Google Cloud Storage (GCS) to Google Cloud Storage (GCS)¶

Source product documentation: Google Cloud Storage (GCS)
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Transfer data in Google Cloud Storage
Python API: airflow.providers.google.cloud.transfers.gcs_to_gcs
Provider: apache-airflow-providers-google

Facebook Ads to Google Cloud Storage (GCS)¶

Source product documentation: Facebook Ads
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Facebook Ads To GCS Operators
Python API: airflow.providers.google.cloud.transfers.facebook_ads_to_gcs
Provider: apache-airflow-providers-google

SSH File Transfer Protocol (SFTP) to Google Cloud Storage (GCS)¶

Source product documentation: SSH File Transfer Protocol (SFTP)
Target product documentation: Google Cloud Storage (GCS)
Operator guide: SFTP to Google Cloud Storage Transfer Operator
Python API: airflow.providers.google.cloud.transfers.sftp_to_gcs
Provider: apache-airflow-providers-google

Microsoft Azure Data Lake Storage to Google Cloud Storage (GCS)¶

Source product documentation: Microsoft Azure Data Lake Storage
Target product documentation: Google Cloud Storage (GCS)
Python API: airflow.providers.google.cloud.transfers.adls_to_gcs
Provider: apache-airflow-providers-google

Google BigQuery to Google BigQuery¶

Source product documentation: Google BigQuery
Target product documentation: Google BigQuery
Operator guide: Google Cloud BigQuery Transfer Operator to BigQuery
Python API: airflow.providers.google.cloud.transfers.bigquery_to_bigquery
Provider: apache-airflow-providers-google

MySQL to Google Cloud Storage (GCS)¶

Source product documentation: MySQL
Target product documentation: Google Cloud Storage (GCS)
Operator guide: MySQL To Google Cloud Storage Operator
Python API: airflow.providers.google.cloud.transfers.mysql_to_gcs
Provider: apache-airflow-providers-google

Oracle to Google Cloud Storage (GCS)¶

Source product documentation: Oracle
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Oracle To Google Cloud Storage Operator
Python API: airflow.providers.google.cloud.transfers.oracle_to_gcs
Provider: apache-airflow-providers-google

Google Cloud Storage (GCS) to Google Spreadsheet¶

Source product documentation: Google Cloud Storage (GCS)
Target product documentation: Google Spreadsheet
Operator guide: Google Cloud Storage to Google Sheets Transfer Operators
Python API: airflow.providers.google.suite.transfers.gcs_to_sheets
Provider: apache-airflow-providers-google

Local to Google Cloud Storage (GCS)¶

Target product documentation: Google Cloud Storage (GCS)
Operator guide: Upload data from Local Filesystem to Google Cloud Storage
Python API: airflow.providers.google.cloud.transfers.local_to_gcs
Provider: apache-airflow-providers-google

Google BigQuery to Google Cloud Storage (GCS)¶

Source product documentation: Google BigQuery
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Google Cloud BigQuery Transfer Operator to Google Cloud Storage
Python API: airflow.providers.google.cloud.transfers.bigquery_to_gcs
Provider: apache-airflow-providers-google

Google Cloud Storage (GCS) to Local¶

Source product documentation: Google Cloud Storage (GCS)
Operator guide: Downloads data from Google Cloud Storage to Local Filesystem
Python API: airflow.providers.google.cloud.transfers.gcs_to_local
Provider: apache-airflow-providers-google

Salesforce to Google Cloud Storage (GCS)¶

Source product documentation: Salesforce
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Salesforce To GCS Operators
Python API: airflow.providers.google.cloud.transfers.salesforce_to_gcs
Provider: apache-airflow-providers-google

Google Ads to Google Cloud Storage (GCS)¶

Source product documentation: Google Ads
Target product documentation: Google Cloud Storage (GCS)
Python API: airflow.providers.google.ads.transfers.ads_to_gcs
Provider: apache-airflow-providers-google

Microsoft SQL Server (MSSQL) to Google Cloud Storage (GCS)¶

Source product documentation: Microsoft SQL Server (MSSQL)
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Microsoft SQL Server To Google Cloud Storage Operator
Python API: airflow.providers.google.cloud.transfers.mssql_to_gcs
Provider: apache-airflow-providers-google

Microsoft Azure Blob Storage to Google Cloud Storage (GCS)¶

Source product documentation: Microsoft Azure Blob Storage
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Azure Blob Storage to Google Cloud Storage (GCS) Transfer Operator
Python API: airflow.providers.google.cloud.transfers.azure_blob_to_gcs
Provider: apache-airflow-providers-google

Microsoft Azure Blob Storage to Google Cloud Storage (GCS)¶

Source product documentation: Microsoft Azure Blob Storage
Target product documentation: Google Cloud Storage (GCS)
Operator guide: Upload data from Azure Blob Storage to Google Cloud Storage (Moved to Google Providers)
Python API: airflow.providers.microsoft.azure.transfers.azure_blob_to_gcs
Provider: apache-airflow-providers-microsoft-azure

Google Cloud Storage (GCS) to Presto¶

Source product documentation: Google Cloud Storage (GCS)
Target product documentation: Presto
Operator guide: Google Cloud Storage to Presto Transfer Operator
Python API: airflow.providers.presto.transfers.gcs_to_presto
Provider: apache-airflow-providers-presto

Google Cloud Storage (GCS) to Samba¶

Source product documentation: Google Cloud Storage (GCS)
Target product documentation: Samba
Operator guide: Google Cloud Storage Transfer Operator to Samba
Python API: airflow.providers.samba.transfers.gcs_to_samba
Provider: apache-airflow-providers-samba

Google Cloud Storage (GCS) to Snowflake¶

Source product documentation: Google Cloud Storage (GCS)
Target product documentation: Snowflake
Python API: airflow.providers.snowflake.transfers.copy_into_snowflake
Provider: apache-airflow-providers-snowflake

Google Cloud Storage (GCS) to Trino¶

Source product documentation: Google Cloud Storage (GCS)
Target product documentation: Trino
Operator guide: Google Cloud Storage to Trino Transfer Operator
Python API: airflow.providers.trino.transfers.gcs_to_trino
Provider: apache-airflow-providers-trino

Google Marketing Platform¶

Note

You can learn how to use Google Marketing Platform integrations by analyzing the source code of the example DAGs.

Google Ads¶

Operators: airflow.providers.google.ads.operators.ads.
Hooks: airflow.providers.google.ads.hooks.ads.
Guides: Google Ads Operators.
Provider: apache-airflow-providers-google
Product documentation: Google Ads

Google Analytics360¶

Operators: airflow.providers.google.marketing_platform.operators.analytics.
Hooks: airflow.providers.google.marketing_platform.hooks.analytics.
Guides: Google Analytics 360 Operators.
Provider: apache-airflow-providers-google
Product documentation: Google Analytics360

Google Display&Video 360¶

Operators: airflow.providers.google.marketing_platform.operators.display_video.
Hooks: airflow.providers.google.marketing_platform.hooks.display_video.
Sensors: airflow.providers.google.marketing_platform.sensors.display_video.
Guides: Google Display & Video 360 Operators.
Provider: apache-airflow-providers-google
Product documentation: Google Display&Video 360

Google Search Ads 360¶

Operators: airflow.providers.google.marketing_platform.operators.search_ads.
Hooks: airflow.providers.google.marketing_platform.hooks.search_ads.
Sensors: airflow.providers.google.marketing_platform.sensors.search_ads.
Guides: Google Search Ads Operators.
Provider: apache-airflow-providers-google
Product documentation: Google Search Ads 360

Other Google operators and hooks¶

Google¶

Hooks: airflow.providers.google.common.hooks.base_google.
Provider: apache-airflow-providers-google
Product documentation: Google

Google Calendar¶

Hooks: airflow.providers.google.suite.hooks.calendar.
Provider: apache-airflow-providers-google
Product documentation: Google Calendar

Google Cloud Batch¶

Operators: airflow.providers.google.cloud.operators.cloud_batch.
Hooks: airflow.providers.google.cloud.hooks.cloud_batch.
Guides: Google Cloud Batch Operators.
Provider: apache-airflow-providers-google
Product documentation: Google Cloud Batch

Google Cloud Composer¶

Operators: airflow.providers.google.cloud.operators.cloud_composer.
Hooks: airflow.providers.google.cloud.hooks.cloud_composer.
Sensors: airflow.providers.google.cloud.sensors.cloud_composer.
Guides: Google Cloud Composer Operators.
Provider: apache-airflow-providers-google
Product documentation: Google Cloud Composer

Google Cloud Dataform¶

Operators: airflow.providers.google.cloud.operators.dataform.
Hooks: airflow.providers.google.cloud.hooks.dataform.
Sensors: airflow.providers.google.cloud.sensors.dataform.
Guides: Google Dataform Operators.
Provider: apache-airflow-providers-google
Product documentation: Google Cloud Dataform

Google Cloud Run¶

Operators: airflow.providers.google.cloud.operators.cloud_run.
Hooks: airflow.providers.google.cloud.hooks.cloud_run.
Guides: Google Cloud Run Operators.
Provider: apache-airflow-providers-google
Product documentation: Google Cloud Run

Google Discovery API¶

Hooks: airflow.providers.google.common.hooks.discovery_api.
Provider: apache-airflow-providers-google
Product documentation: Google Discovery API

Google Drive¶

Hooks: airflow.providers.google.suite.hooks.drive.
Sensors: airflow.providers.google.suite.sensors.drive.
Provider: apache-airflow-providers-google
Product documentation: Google Drive

Google LevelDB¶

Operators: airflow.providers.google.leveldb.operators.leveldb.
Hooks: airflow.providers.google.leveldb.hooks.leveldb.
Guides: Google LevelDB Operator.
Provider: apache-airflow-providers-google
Product documentation: Google LevelDB

Google Spreadsheet¶

Operators: airflow.providers.google.suite.operators.sheets.
Hooks: airflow.providers.google.suite.hooks.sheets.
Guides: Google Sheets Operators.
Provider: apache-airflow-providers-google
Product documentation: Google Spreadsheet

Google¶

Google Cloud¶

Services¶

Google AutoML¶

Google BigQuery¶

Google BigQuery Data Transfer Service¶

Google Bigtable¶

Google Campaign Manager¶

Google Cloud Build¶

Google Cloud Common¶

Google Cloud Data Loss Prevention (DLP)¶

Google Cloud Firestore¶

Google Cloud Functions¶

Google Cloud Key Management Service (KMS)¶

Google Cloud Life Sciences¶

Google Cloud Memorystore¶

Google Cloud Natural Language¶

Google Cloud OS Login¶

Google Cloud Pub/Sub¶

Google Cloud Secret Manager¶

Google Cloud Spanner¶

Google Cloud Speech-to-Text¶

Google Cloud SQL¶

Google Cloud Stackdriver¶

Google Cloud Storage (GCS)¶

Google Cloud Storage Transfer Service¶

Google Cloud Tasks¶

Google Cloud Text-to-Speech¶

Google Cloud Translation¶

Google Cloud Video Intelligence¶

Google Cloud Vision¶

Google Cloud Workflows¶

Google Compute Engine¶

Google Data Catalog¶

Google Data Fusion¶

Google Data Pipelines¶

Google Dataflow¶

Google Dataplex¶

Google Dataprep¶

Google Dataproc¶

Google Dataproc Metastore¶

Google Datastore¶

Google Deployment Manager¶

Google Kubernetes Engine¶

Google Looker¶

Google Machine Learning Engine¶

Google Vertex AI¶

Transfers¶

Google Cloud Storage (GCS) to Amazon Simple Storage Service (S3)¶

Amazon Glacier to Google Cloud Storage (GCS)¶

Presto to Google Cloud Storage (GCS)¶

Trino to Google Cloud Storage (GCS)¶

Common SQL to Google Cloud Storage (GCS)¶

Google Cloud Storage (GCS) to Google Drive¶

Google Drive to Google Cloud Storage (GCS)¶

Microsoft Azure FileShare to Google Cloud Storage (GCS)¶

Apache Cassandra to Google Cloud Storage (GCS)¶

Google Calendar to Google Cloud Storage (GCS)¶

Google Spreadsheet to Google Cloud Storage (GCS)¶

Amazon Simple Storage Service (S3) to Google Cloud Storage (GCS)¶

Google Cloud Storage (GCS) to SSH File Transfer Protocol (SFTP)¶

PostgreSQL to Google Cloud Storage (GCS)¶

Google BigQuery to Common SQL¶

Google BigQuery to MySQL¶

Google BigQuery to Microsoft SQL Server (MSSQL)¶

Google BigQuery to PostgreSQL¶

Google Cloud Storage (GCS) to Google BigQuery¶

Google Cloud Storage (GCS) to Google Cloud Storage (GCS)¶

Facebook Ads to Google Cloud Storage (GCS)¶

SSH File Transfer Protocol (SFTP) to Google Cloud Storage (GCS)¶

Microsoft Azure Data Lake Storage to Google Cloud Storage (GCS)¶

Google BigQuery to Google BigQuery¶

MySQL to Google Cloud Storage (GCS)¶

Oracle to Google Cloud Storage (GCS)¶

Google Cloud Storage (GCS) to Google Spreadsheet¶

Local to Google Cloud Storage (GCS)¶

Google BigQuery to Google Cloud Storage (GCS)¶

Google Cloud Storage (GCS) to Local¶

Salesforce to Google Cloud Storage (GCS)¶

Google Ads to Google Cloud Storage (GCS)¶