apache-airflow-providers-microsoft-azure

Changelog

8.4.0

Note

This release of provider is only available for Airflow 2.6+ as explained in the Apache Airflow providers support policy.

Bug Fixes

  • Fix reraise outside of try block in 'AzureSynapsePipelineRunLink.get_fields_from_url' (#36009)

  • Do not catch too broad exception in 'WasbHook.delete_container' (#36034)

Misc

  • Bump minimum Airflow version in providers to Airflow 2.6.0 (#36017)

8.3.0

Features

  • Add Azure Synapse Pipeline connection-type in the UI (#35709)

  • Add task context logging feature to allow forwarding messages to task logs (#32646)

  • Add operator to invoke Azure-Synapse pipeline (#35091)

  • Extend task context logging support for remote logging using WASB (Azure Blob Storage) (#32972)

Misc

  • Check attr on parent not self re TaskContextLogger set_context (#35780)

  • Remove backcompat with Airflow 2.3/2.4 in providers (#35727)

8.2.0

Features

  • add managed identity support to AsyncDefaultAzureCredential (#35394)

  • feat(provider/azure): add managed identity support to container_registry hook (#35320)

  • feat(provider/azure): add managed identity support to wasb hook (#35326)

  • feat(provider/azure): add managed identity support to asb hook (#35324)

  • feat(provider/azure): add managed identity support to cosmos hook (#35323)

  • feat(provider/azure): add managed identity support to container_volume hook (#35321)

  • feat(provider/azure): add managed identity support to container_instance hook (#35319)

  • feat(provider/azure): add managed identity support to adx hook (#35325)

  • feat(provider/azure): add managed identity support to batch hook (#35327)

  • feat(provider/azure): add managed identity support to data_factory hook (#35328)

  • feat(provider/azure): add managed identity support to synapse hook (#35329)

  • feat(provider/azure): add managed identity support to fileshare hook (#35330)

Bug Fixes

  • Fix AzureContainerInstanceOperator remove_on_error (#35212)

  • fix(providers/microsoft): setting use_async=True for get_async_default_azure_credential (#35432)

Misc

  • Remove empty TYPE_CHECKING block into the Azure provider (#35477)

  • Refactor azure managed identity (#35367)

  • Reuse get_default_azure_credential method from Azure utils method (#35318)

  • `` make DefaultAzureCredential configurable in AzureKeyVaultBackend (#35052)``

  • Make DefaultAzureCredential in AzureBaseHook configuration (#35051)

8.1.0

Features

  • AIP-58: Add Airflow ObjectStore (AFS) (#34729)

8.0.0

Note

This release of provider is only available for Airflow 2.5+ as explained in the Apache Airflow providers support policy.

Breaking changes

Warning

In this version of the provider, we have removed network_profile param from AzureContainerInstancesOperator and AzureDataFactoryHook methods and AzureDataFactoryRunPipelineOperator arguments resource_group_name and factory_name is now required instead of kwargs

  • resource_group_name and factory_name is now required argument in AzureDataFactoryHook method get_factory, update_factory, create_factory, delete_factory, get_linked_service, delete_linked_service, get_dataset, delete_dataset, get_dataflow, update_dataflow, create_dataflow, delete_dataflow, get_pipeline, delete_pipeline, run_pipeline, get_pipeline_run, get_trigger, get_pipeline_run_status, cancel_pipeline_run, create_trigger, delete_trigger, start_trigger, stop_trigger, get_adf_pipeline_run_status, cancel_pipeline_run

  • resource_group_name and factory_name is now required in AzureDataFactoryRunPipelineOperator

  • Remove class PipelineRunInfo from airflow.providers.microsoft.azure.hooks.data_factory

  • Remove network_profile param from AzureContainerInstancesOperator

  • Remove deprecated extra__azure__tenantId from azure_container_instance connection extras

  • Remove deprecated extra__azure__subscriptionId from azure_container_instance connection extras

  • Bump azure-mgmt-containerinstance (#34738)

  • Upgrade azure-mgmt-datafactory in microsift azure provider (#34040)

Features

  • Add subnet_ids param in AzureContainerInstancesOperator (#34850)

  • allow providing credentials through keyword argument in AzureKeyVaultBackend (#34706)

Bug Fixes

  • Name params while invoking ClientSecretCredential (#34732)

  • fix(providers/microsoft-azure): respect soft_fail argument when exception is raised (#34494)

  • Error handling for when Azure container log cannot be read in properly. (#34627)

  • Fix hardcoded container name in remote logging option for Azure Blob Storage (#32779)

Misc

  • Bump min airflow version of providers (#34728)

  • Consolidate hook management in AzureBatchOperator (#34437)

  • Consolidate hook management in AzureDataExplorerQueryOperator (#34436)

7.0.0

Breaking changes

Warning

In this version of the provider, we have changed AzureFileShareHook to use azure-storage-file-share library instead of azure-storage-file this change has impact on existing hook method see below for details, removed deprecated extra__azure_fileshare__ prefix from connection extras param and removed protocol param from connection extras

  • get_conn from AzureFileShareHook return None instead FileService

  • Remove protocol param from Azure fileshare connection extras

  • Remove deprecated extra__azure_fileshare__ prefix from Azure fileshare connection extras, list_files

  • Remove share_name, directory_name param from AzureFileShareHook method check_for_directory, list_directories_and_files, create_directory in favor of AzureFileShareHook share_name and directory_path param

  • AzureFileShareHook method create_share and delete_share accept kwargs from ShareServiceClient.create_share and ShareServiceClient.delete_share

  • Remove share_name, directory_name, file_name param from AzureFileShareHook method get_file, get_file_to_stream and load_file in favor of AzureFileShareHook share_name and file_path

  • Remove AzureFileShareHook.check_for_file method

  • Remove AzureFileShareHook.load_string, AzureFileShareHook.load_stream in favor of AzureFileShareHook.load_data

Note

LocalToAzureDataLakeStorageOperator class has been removed in favor of LocalFilesystemToADLSOperator AzureDataFactoryPipelineRunStatusAsyncSensor class has been removed in favor of AzureDataFactoryPipelineRunStatusSensor

  • Update Azure fileshare hook to use azure-storage-file-share instead of azure-storage-file (#33904)

  • Remove 'AzureDataFactoryPipelineRunStatusAsyncSensor' class (#34036)

  • Remove 'LocalToAzureDataLakeStorageOperator' class (#34035)

Features

  • feat(providers/microsoft): add AzureContainerInstancesOperator.volume as template field (#34070)

  • Add DefaultAzureCredential support to AzureContainerRegistryHook (#33825)

  • feat(providers/microsoft): add DefaultAzureCredential support to AzureContainerVolumeHook (#33822)

Misc

  • Refactor regex in providers (#33898)

  • Improve docs on AzureBatchHook DefaultAzureCredential support (#34098)

  • Remove  azure-storage-common from microsoft azure providers (#34038)

  • Remove useless string join from providers (#33968)

  • Refactor unneeded  jumps in providers (#33833)

6.3.0

Features

  • Add AzureBatchOperator example (#33716)

  • feat(providers/microsoft): add DefaultAzureCredential support to AzureContainerInstanceHook (#33467)

  • Add DefaultAzureCredential auth for ADX service (#33627)

  • feat(providers/microsoft): add DefaultAzureCredential to data_lake (#33433)

  • Allow passing fully_qualified_namespace and credential to initialize Azure Service Bus Client (#33493)

  • Add DefaultAzureCredential support to cosmos (#33436)

  • Add DefaultAzureCredential support to AzureBatchHook (#33469)

Bug Fixes

  • Fix updating account url for WasbHook (#33457)

  • Fix Azure Batch Hook instantation (#33731)

  • Truncate Wasb storage account name if it's more than 24 characters (#33851)

  • Remove duplicated message commit in Azure MessageHook (#33776)

  • fix(providers/azure): remove json.dumps when querying AzureCosmosDBHook (#33653)

Misc

  • Refactor: Remove useless str() calls (#33629)

  • Bump azure-kusto-data>=4.1.0 (#33598)

  • Simplify conditions on len() in providers/microsoft (#33566)

  • Set logging level to WARNING (#33314)

  • Simplify 'X for X in Y' to 'Y' where applicable (#33453)

  • Bump azure-mgmt-containerinstance>=7.0.0,<9.0.0 (#33696)

  • Improve modules import in Airflow providers by some of them into a type-checking block (#33754)

  • Use a single  statement with multiple contexts instead of nested  statements in providers (#33768)

  • remove unnecessary and rewrite it using list in providers (#33763)

  • Optimise Airflow DB backend usage in Azure Provider (#33750)

6.2.4

Misc

  • Clean microsoft azure provider by deleting the custom prefix from conn extra fields (#30558)

6.2.3

Misc

  • Refactor account_url use in WasbHook (#32980)

6.2.2

Misc

  • Add Redis task handler (#31855)

  • Add deprecation info to the providers modules and classes docstring (#32536)

6.2.1

Note

Note: this version contains a fix to get_blobs_list_async method in WasbHook where it returned a list of blob names, but advertised (via type hints) that it returns a list of BlobProperties objects. This was a bug in the implementation and it was fixed in this release. However, if you were relying on the previous behaviour, you might need to retrieve name property from the array elements returned by this method.

Bug Fixes

  • Fix breaking change when Active Directory ID is used as host in WASB (#32560)

  • Fix get_blobs_list_async method to return BlobProperties (#32545)

Misc

  • Moves 'AzureBlobStorageToGCSOperator' from Azure to Google provider (#32306)

6.2.0

Features

  • Adds connection test for ADLS Gen2  (#32126)

  • Add option to pass extra configs to ClientSecretCredential  (#31783)

  • Added 'AzureBlobStorageToS3Operator' transfer operator (#32270)

Bug Fixes

  • Cancel pipeline if unexpected exception caught (#32238)

  • Fix where account url is build if not provided using login (account name) (#32082)

  • refresh connection if an exception is caught in "AzureDataFactory" (#32323)

Misc

  • Doc changes: Added Transfers section in Azure provider docs (#32241)

  • Adds Sensor section in the Azure providers docs  (#32299)

  • Add default_deferrable config (#31712)

6.1.2

Note

This release dropped support for Python 3.7

Misc

  • Replace unicodecsv with standard csv library (#31693)

  • Removed unused variables in AzureBlobStorageToGCSOperator (#31765)

  • Remove Python 3.7 support (#30963)

  • Add docstring and signature for _read_remote_logs (#31623)

6.1.1

Bug Fixes

  • Fix deferrable mode execution in WasbPrefixSensor (#31411)

Misc

  • Optimize deferred mode execution for wasb sensors (#31009)

6.1.0

Note

This release of provider is only available for Airflow 2.4+ as explained in the Apache Airflow providers support policy.

Features

  • Add deferrable mode to 'WasbPrefixSensor' (#30252)

Misc

  • Bump minimum Airflow version in providers (#30917)

  • Optimize deferrable execution mode 'AzureDataFactoryPipelineRunStatusSensor' (#30983)

  • Optimize deferred execution for AzureDataFactoryRunPipelineOperator (#31214)

6.0.0

Breaking changes

Warning

In this version of the provider, deprecated GCS hook’s param delegate_to is removed from AzureBlobStorageToGCSOperator. Impersonation can be achieved instead by utilizing the impersonation_chain param.

  • remove delegate_to from GCP operators and hooks (#30748)

Misc

  • Merge WasbBlobAsyncSensor to WasbBlobSensor (#30488)

5.3.1

Bug Fixes

  • Fix AzureDataFactoryPipelineRunLink get_link method (#30514)

  • Load subscription_id from extra__azure__subscriptionId (#30556)

5.3.0

Features

  • Add deferrable 'AzureDataFactoryRunPipelineOperator' (#30147)

  • Add deferrable 'AzureDataFactoryPipelineRunStatusSensor' (#29801)

  • Support deleting the local log files when using remote logging (#29772)

Bug Fixes

  • Fix ADF job failure during deferral (#30248)

  • Fix AzureDataLakeStorageV2Hook 'account_url' with Active Directory authentication (#29980) (#29981)

Misc

  • merge AzureDataFactoryPipelineRunStatusAsyncSensor to AzureDataFactoryPipelineRunStatusSensor (#30250)

  • Expose missing params in AzureSynapseHook API docs (#30099)

5.2.1

Bug Fixes

  • Handle deleting more than 256 blobs using 'WasbHook.delete_file()' (#29565)

5.2.0

Features

  • Enable individual trigger logging (#27758)

Bug Fixes

  • Fix params rendering in AzureSynapseHook Python API docs (#29041)

Misc

  • Deprecate 'delegate_to' param in GCP operators and update docs (#29088)

5.1.0

Features

  • Add hook for Azure Data Lake Storage Gen2 (#28262)

Bug Fixes

  • Hide 'extra' field in WASB connection form (#28914)

5.0.2

Misc

  • Re-enable azure service bus on ARM as it now builds cleanly (#28442)

5.0.1

Bug Fixes

  • Make arguments 'offset' and 'length' not required (#28234)

5.0.0

Note

This release of provider is only available for Airflow 2.3+ as explained in the Apache Airflow providers support policy.

Breaking changes

  • In AzureFileShareHook, if both extra__azure_fileshare__foo and foo existed in connection extra dict, the prefixed version would be used; now, the non-prefixed version will be preferred.

  • Remove deprecated classes (#27417)

  • In Azure Batch vm_size and vm_node_agent_sku_id parameters are required.

Misc

  • Move min airflow version to 2.3.0 for all providers (#27196)

Features

  • Add azure, google, authentication library limits to eaager upgrade (#27535)

  • Allow and prefer non-prefixed extra fields for remaining azure (#27220)

  • Allow and prefer non-prefixed extra fields for AzureFileShareHook (#27041)

  • Allow and prefer non-prefixed extra fields for AzureDataExplorerHook (#27219)

  • Allow and prefer non-prefixed extra fields for AzureDataFactoryHook (#27047)

  • Update WasbHook to reflect preference for unprefixed extra (#27024)

  • Look for 'extra__' instead of 'extra_' in 'get_field' (#27489)

Bug Fixes

  • Fix Azure Batch errors revealed by added typing to azure batch lib (#27601)

  • Fix separator getting added to variables_prefix when empty (#26749)

4.3.0

Features

  • Add DataFlow operations to Azure DataFactory hook (#26345)

  • Add network_profile param in AzureContainerInstancesOperator (#26117)

  • Add Azure synapse operator (#26038)

  • Auto tail file logs in Web UI (#26169)

  • Implement Azure Service Bus Topic Create, Delete Operators (#25436)

Bug Fixes

  • Fix AzureBatchOperator false negative task status (#25844)

4.2.0

Features

  • Add 'test_connection' method to AzureContainerInstanceHook (#25362)

  • Add test_connection to Azure Batch hook (#25235)

  • Bump typing-extensions and mypy for ParamSpec (#25088)

  • Implement Azure Service Bus (Update and Receive) Subscription Operator (#25029)

  • Set default wasb Azure http logging level to warning; fixes #16224 (#18896)

4.1.0

Features

  • Add 'test_connection' method to AzureCosmosDBHook (#25018)

  • Add test_connection method to AzureFileShareHook (#24843)

  • Add test_connection method to Azure WasbHook (#24771)

  • Implement Azure service bus subscription Operators (#24625)

  • Implement Azure Service Bus Queue Operators (#24038)

Bug Fixes

  • Update providers to use functools compat for ''cached_property'' (#24582)

4.0.0

Breaking changes

Note

This release of provider is only available for Airflow 2.2+ as explained in the Apache Airflow providers support policy.

Features

  • Pass connection extra parameters to wasb BlobServiceClient (#24154)

Misc

  • Apply per-run log templates to log handlers (#24153)

  • Migrate Microsoft example DAGs to new design #22452 - azure (#24141)

  • Add typing to Azure Cosmos Client Hook (#23941)

3.9.0

Features

  • wasb hook: user defaultAzureCredentials instead of managedIdentity (#23394)

Misc

  • Replace usage of 'DummyOperator' with 'EmptyOperator' (#22974)

3.8.0

Features

  • Update secrets backends to use get_conn_value instead of get_conn_uri (#22348)

Misc

  • Docs: Fix example usage for 'AzureCosmosDocumentSensor' (#22735)

3.7.2

Bug Fixes

  • Fix mistakenly added install_requires for all providers (#22382)

3.7.1

Misc

  • Add Trove classifiers in PyPI (Framework :: Apache Airflow :: Provider)

3.7.0

Features

  • Add 'test_connection' method to 'AzureDataFactoryHook' (#21924)

  • Add pre-commit check for docstring param types (#21398)

  • Make container creation configurable when uploading files via WasbHook (#20510)

Misc

  • Support for Python 3.10

  • (AzureCosmosDBHook) Update to latest Cosmos API (#21514)

3.6.0

Features

  • Add optional features in providers. (#21074)

Misc

  • Refactor operator links to not create ad hoc TaskInstances (#21285)

3.5.0

Features

  • Azure: New sftp to wasb operator (#18877)

  • Removes InputRequired validation with azure extra (#20084)

  • Add operator link to monitor Azure Data Factory pipeline runs (#20207)

3.4.0

Features

  • Remove unnecessary connection form customizations in Azure (#19595)

  • Update Azure modules to comply with AIP-21 (#19431)

  • Remove 'host' from hidden fields in 'WasbHook' (#19475)

  • use DefaultAzureCredential if login not provided for Data Factory (#19079)

Bug Fixes

  • Fix argument error in AzureContainerInstancesOperator (#19668)

3.3.0

Features

  • update azure cosmos to latest version (#18695)

  • Added sas_token var to BlobServiceClient return. Updated tests (#19234)

  • Add pre-commit hook for common misspelling check in files (#18964)

Bug Fixes

  • Fix changelog for Azure Provider (#18736)

Other

  • Expanding docs on client auth for AzureKeyVaultBackend (#18659)

  • Static start_date and default arg cleanup for Microsoft providers example DAGs (#19062)

3.2.0

Features

  • Rename AzureDataLakeStorage to ADLS (#18493)

  • Creating ADF pipeline run operator, sensor + ADF custom conn fields (#17885)

  • Rename LocalToAzureDataLakeStorageOperator to LocalFilesystemToADLSOperator (#18168)

  • Rename FileToWasbOperator to LocalFilesystemToWasbOperator (#18109)

Bug Fixes

  • Fixed wasb hook attempting to create container when getting a blob client (#18287)

  • Removing redundant relabeling of password conn field (#18386)

  • Proper handling of Account URL custom conn field in AzureBatchHook (#18456)

  • Proper handling of custom conn field values in the AzureDataExplorerHook (#18203)

Main

Changes in operators names and import paths are listed in the following table This is a backward compatible change. Deprecated operators will be removed in the next major release.

Deprecated operator name

New operator name

Deprecated path

New path

AzureDataLakeStorageListOperator

ADLSListOperator

airflow.providers.microsoft.azure.operators.adls_list

airflow.providers.microsoft.azure.operators.adls

AzureDataLakeStorageDeleteOperator

ADLSDeleteOperator

airflow.providers.microsoft.azure.operators.adls_delete

airflow.providers.microsoft.azure.operators.adls

3.1.1

Misc

  • Optimise connection importing for Airflow 2.2.0

  • Adds secrets backend/logging/auth information to provider yaml (#17625)

3.1.0

Features

  • Add support for managed identity in WASB hook (#16628)

  • Reduce log messages for happy path (#16626)

Bug Fixes

  • Fix multiple issues in Microsoft AzureContainerInstancesOperator (#15634)

3.0.0

Breaking changes

  • Auto-apply apply_default decorator (#15667)

Warning

Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+. If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded automatically and you will have to manually run airflow upgrade db to complete the migration.

  • Fixes AzureFileShare connection extras (#16388)

Azure Container Volume and Azure File Share have now dedicated connection types with editable UI fields. You should not use Wasb connection type any more for those connections. Names of connection ids for those hooks/operators were changed to reflect that.

Features

  • add oracle  connection link (#15632)

  • Add delimiter argument to WasbHook delete_file method (#15637)

Bug Fixes

  • Fix colon spacing in ``AzureDataExplorerHook docstring (#15841)``

  • fix wasb remote logging when blob already exists (#16280)

2.0.0

Breaking changes

  • Removes unnecessary AzureContainerInstance connection type (#15514)

This change removes azure_container_instance_default connection type and replaces it with the azure_default. The problem was that AzureContainerInstance was not needed as it was exactly the same as the plain “azure” connection, however its presence caused duplication in the field names used in the UI editor for connections and unnecessary warnings generated. This version uses plain Azure Hook and connection also for Azure Container Instance. If you already have azure_container_instance_default connection created in your DB, it will continue to work, but the first time you edit it with the UI you will have to change its type to azure_default.

Features

  • Add dynamic connection fields to Azure Connection (#15159)

Bug fixes

  • Fix 'logging.exception' redundancy (#14823)

1.3.0

Features

  • A bunch of template_fields_renderers additions (#15130)

Bug fixes

  • Fix attributes for AzureDataFactory hook (#14704)

1.2.0

Features

  • Add Azure Data Factory hook (#11015)

Bug fixes

  • BugFix: Fix remote log in azure storage blob displays in one line (#14313)

  • Fix AzureDataFactoryHook failing to instantiate its connection (#14565)

1.1.0

Updated documentation and readme files.

Features

  • Upgrade azure blob to v12 (#12188)

  • Fix Azure Data Explorer Operator (#13520)

  • add AzureDatalakeStorageDeleteOperator (#13206)

1.0.0

Initial version of the provider.

Was this entry helpful?