The Airflow Scheduler Daemon: How to Update Your Environment's Python Version

When setting up an environment for your Apache Airflow workflows, it's essential to consider the compatibility of your Python version. Amazon MWAA installs a specific version of Python that is compatible with your environment, which means you can't update it manually when running yum update in a startup script.

However, if you need to update the operating system of an Apache Airflow component or install additional runtime libraries for your workflows, you'll need to use a startup script. For example, you can run yum update to update the operating system using a script. This process is crucial for ensuring that your environment runs smoothly and efficiently.

To get started with updating your environment's Python version, you'll first need to exclude Python from the update process using the -exclude=python* command as shown in the following example:

Once you've excluded Python from the update process, you can proceed with installing additional runtime libraries for your workflows. This will enable you to use the latest versions of Apache Airflow components and take full advantage of their features.

For those who want to view the logs of their Airflow scheduler daemon, there are a few steps you can follow. First, ensure that your startup script has finished running. Then, navigate to the Log events pane in the CloudWatch console, where you'll see the output of the command printing the value for MWAA_AIRFLOW_COMPONENT.

Next, choose a stream with the following prefix: startup_script_exection_ip. Finally, select the log group for which you want to view logs, such as the Airflow scheduler log group. This will give you a comprehensive overview of your Airflow scheduler daemon's performance and help you identify any issues that may arise during its operation.

In conclusion, updating your environment's Python version is a crucial step in setting up an Apache Airflow workflow. By following these simple steps and using a startup script to update your operating system and install additional runtime libraries, you'll be well on your way to creating efficient and effective workflows with the Airflow scheduler daemon.