run_this = BashOperator(
task_id="run_after_loop",
bash_command="echo 1",
also_run_this = BashOperator(
task_id="also_run_this",
bash_command='echo "ti_key={{ task_instance_key_str }}"',
Warning
Care should be taken with “user” input or when using Jinja templates in the
bash_command, as this bash operator does not perform any escaping or
sanitization of the command.
This applies mostly to using “dag_run” conf, as that can be submitted via
users in the Web UI. Most of the default template variables are not at
risk.
For example, do not do this:
bash_task = BashOperator(
task_id="bash_task",
bash_command='echo "Here is the message: \'{{ dag_run.conf["message"] if dag_run else "" }}\'"',
Instead, you should pass this via the env kwarg and use double-quotes
inside the bash_command, as below:
bash_task = BashOperator(
task_id="bash_task",
bash_command="echo \"here is the message: '$message'\"",
env={"message": '{{ dag_run.conf["message"] if dag_run else "" }}'},
Skipping
In general a non-zero exit code produces an AirflowException and thus a task failure. In cases where it is desirable
to instead have the task end in a skipped state, you can exit with code 99 (or with another exit code if you
pass skip_exit_code).
this_will_skip = BashOperator(
task_id="this_will_skip",
bash_command='echo "hello world"; exit 99;',
dag=dag,
Jinja template not found
Add a space after the script name when directly calling a Bash script with
the bash_command argument. This is because Airflow tries to apply a Jinja
template to it, which will fail.
t2 = BashOperator(
task_id="bash_example",
# This fails with 'Jinja template not found' error
# bash_command="/home/batcher/test.sh",
# This works (has a space after)
bash_command="/home/batcher/test.sh ",
dag=dag,
However, if you want to use templating in your bash script, do not add the space
and instead put your bash script in a location relative to the directory containing
the DAG file. So if your DAG file is in /usr/local/airflow/dags/test_dag.py, you can
move your test.sh file to any location under /usr/local/airflow/dags/ (Example:
/usr/local/airflow/dags/scripts/test.sh) and pass the relative path to bash_command
as shown below:
t2 = BashOperator(
task_id="bash_example",
# "scripts" folder is under "/usr/local/airflow/dags"
bash_command="scripts/test.sh",
dag=dag,
Creating separate folder for bash scripts may be desirable for many reasons, like
separating your script’s logic and pipeline code, allowing for proper code highlighting
in files composed in different languages, and general flexibility in structuring
pipelines.
It is also possible to define your template_searchpath as pointing to any folder
locations in the DAG constructor call.
Example:
dag = DAG("example_bash_dag", template_searchpath="/opt/scripts")
t2 = BashOperator(
task_id="bash_example",
# "test.sh" is a file under "/opt/scripts"
bash_command="test.sh ",
dag=dag,
BashSensor
Use the BashSensor to use arbitrary command for sensing. The command
should return 0 when it succeeds, any other value otherwise.
t3 = BashSensor(task_id="Sensor_succeeds", bash_command="exit 0")
t4 = BashSensor(task_id="Sensor_fails_after_3_seconds", timeout=3, soft_fail=True, bash_command="exit 1")
Apache Airflow, Apache, Airflow, the Airflow logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.
All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.
