status-im
/
airflow-xcom-redis
mirror of https://github.com/status-im/airflow-xcom-redis.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
Alternative Backend for Airflow XCom
6 Commits 1 Branch 1 Tag 34 KiB
Python 100%
Go to file
Jakub Sokołowski e307d97bc6
change version to 0.1.0 
Signed-off-by: Jakub Sokołowski <jakub@status.im>
 		2021-03-30 18:09:29 +02:00
airflow_xcom_redis use pickle to serialize data for storing in Redis 2021-03-30 16:52:53 +02:00
LICENSE.txt add python module boilerplate 2021-03-30 14:13:46 +02:00
Pipfile add python module boilerplate 2021-03-30 14:13:46 +02:00
README.md readme: add details and a todo list 2021-03-30 18:09:28 +02:00
setup.cfg add python module boilerplate 2021-03-30 14:13:46 +02:00
setup.py change version to 0.1.0 2021-03-30 18:09:29 +02:00

README.md

Description

This module is an alternative Airflow XCom backend.

Details

Airflow XCom is a way Tasks exchange values in a DAG. By default the values are stored in the default Airflow database. This makes it unsuitable for sharing large volumes of data for two reasons:

  • PostgreSQL has a limit of 1GB per XCom value
  • Using the same database as other Airflow data may degrade performance

For these reasons Airflow allows for implementing a custom XCom backend. This module is one such implementation for use with Redis.

Requirements

Currently the module uses the Python Redis module which is already provided by Airflow.

The only requirements is for Airflow to have a Redis Connection defined called xcom_cache.

TODO

Possible future improvements:

  • Get Connection ID from configuration or environment variables
  • Splitting of values into different keys to avoid size limits