Snippets Collections
1. Create File "Dockerfile" in same directory as "docker-compose.yaml"
2. Write 
FROM apache/airflow:2.0.2
RUN pip install --no-cache-dir geopandas another-package
3. Build new image:
docker build . --tag projectname-airflow:2.0.2
4. In "docker-compose.yaml" replace
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.0.2}
with 
image: ${AIRFLOW_IMAGE_NAME:-projectname-airflow:2.0.2}
5. In Shell move to directory an run
docker build . --tag projectname-airflow:2.0.2
6. Epic win
def pull_file(URL, savepath):
    r = requests.get(URL)
    with open(savepath, 'wb') as f:
        f.write(r.content)   
    # Use the print method for logging
    print(f"File pulled from {URL} and saved to {savepath}")

from airflow.operators.python_operator import PythonOperator

# Create the task
pull_file_task = PythonOperator(
    task_id='pull_file',
    # Add the callable
    python_callable=pull_file,
    # Define the arguments
    op_kwargs={'URL':'http://dataserver/sales.json', 'savepath':'latestsales.json'},
    dag=process_sales_dag
)
star

Wed Dec 06 2023 14:37:41 GMT+0000 (Coordinated Universal Time)

#python #airflow
star

Thu Sep 21 2023 07:32:08 GMT+0000 (Coordinated Universal Time)

#python #airflow #dag

Save snippets that work with our extensions

Available in the Chrome Web Store Get Firefox Add-on Get VS Code extension