python – Airflow与BgQuery Error的集成

我已经安装了气流,我写了一个DAG来将
MySQL数据与BigQuery集成.

当我运行python脚本时,我收到以下错误:

ImportError: cannot import name GbqConnector

我按照指示将大熊猫降级为旧版本.当我这样做时,我得到了一个不同的错误:

ImportError: cannot import name _test_google_api_imports

编辑:来自的建议
x97Core工作.

我现在有一个不同的问题.我收到以下错误:

/usr/local/lib/python2.7/dist-packages/airflow/models.py:1927: PendingDeprecationWarning: Invalid arguments were passed to MySqlToGoogleCloudStorageOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were:

*args: ()

**kwargs: {‘google_cloud_storage_connn_id’: ‘podioGCPConnection’}
category=PendingDeprecationWarning

/usr/local/lib/python2.7/dist-packages/airflow/models.py:1927: PendingDeprecationWarning: Invalid arguments were passed to GoogleCloudStorageToBigQueryOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were:

*args: ()

**kwargs: {‘project_id’: ‘podio-data’}
category=PendingDeprecationWarning

根据这个链接,问题在于气流与python2和pythn3的兼容性.我已经尝试在两者上运行代码,但同样的错误仍然出现.
Airflow mysql to gcp Dag error

有谁知道这是否有解决方案?

最佳答案 刚遇到这个问题.

降级大熊猫版本似乎有效(在Airflow v1.8.0上测试):

pip install pandas==0.18.1

有关详细信息:https://issues.apache.org/jira/browse/AIRFLOW-1179

或者,如果您使用的是Airflow 1.8.2或更高版本.

pip install pandas-gbq
点赞