BigQueryCreateExternalTableOperator doesn't trigger schema discovery

[EugeneYushin]

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

No response

Apache Airflow version

2.7.3

Operating System

Darwin Kernel Version 23.6.0

Deployment

Google Cloud Composer

Deployment details

No response

What happened

Using BigQueryCreateExternalTableOperator with autodetect=True doesn't trigger schema to be created in BQ:

What you think should happen instead

Running the similar command from CLI leads to correct schema discovery processing:

bq mk --table \
  --external_table_definition=@CSV=gs://.../test.csv \
 <dataset_name>.airflow_test_2

How to reproduce

Upload CSV file to GCS:

name,age
Bill,42
Sally,15

Run BigQueryCreateExternalTableOperator:

BigQueryCreateExternalTableOperator(
        task_id="bq_op",
        bucket="***",
        source_objects=["test.csv"],
        destination_project_dataset_table="***",
        source_format="CSV",
        autodetect=True,
        skip_leading_rows=1,
        location="us-east4",
    )

Anything else

Most probably switching from deprecated API bq_hook.create_external_table to create_empty_table led to that behavior
#24363

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[EugeneYushin]

I was able to trace it to that line:

airflow/providers/src/airflow/providers/google/cloud/operators/bigquery.py

Line 1720 in 4c5d85a

"schema": {"fields": schema_fields},

Providing empty schema_fields explicitly intervenes with BQ logic for schema auto-detection. Nominal check to pass it to BQ client if only this field is set should fix the issue.

[EugeneYushin]

Although, it looks like the related part of the API is deprecated. Hence, it might not make much sense to fix it while it's up to be deleted soon. It will be a sole responsibility of the caller to provide valid table definition and the API will just pass it through to GCP:

airflow/providers/src/airflow/providers/google/cloud/operators/bigquery.py

Lines 1603 to 1608 in 4c5d85a

	warnings.warn(
	"Passing table parameters via keywords arguments will be deprecated. "
	"Please provide table definition using `table_resource` parameter.",
	AirflowProviderDeprecationWarning,
	stacklevel=2,
	)