Required Componenets
Scheduler
- Triggers scheduled workflows
- Submits tasks to the executor to run
Executor
- Configuration property of the scheduler (not a separate component)
- Runs within the scheduler process
Webserver
- Presents the user interface
DAG Directory
- Read by the scheduler to figure out what tasks to run and when to run them
- Used by Airflow components to store state of workflows and tasks
Optional Components
Worker
- Executes the tasks given to it by the scheduler (might be part of the scheduler)
- Can be run as:
- a running process in the CeleryExecutor
- a pod in the KubernetesExecutor
Triggerer
- Executes deferred tasks in an asyncio event loop
DAG Processor
- Parses DAG files and serialises them into the metadata database
- If present, the scheduler does not need to read the DAG files directly
Plugin Directory
- Read by the scheduler, DAG processor, triggerer, and webserver
References