status-im
/
airflow-dags
mirror of https://github.com/status-im/airflow-dags.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
Status BI python DAGs for Airflow
85 Commits 3 Branches 0 Tags 268 KiB
Python 100%
Go to file
Alexis Pentori 4a082ba9bb
comm_extraction: fix dag 
Signed-off-by: Alexis Pentori <alexis@status.im>
 		2024-04-30 18:21:23 +02:00
providers/airbyte Provider overriding Airbyte providers 2023-12-14 13:25:21 +01:00
tasks treasure_dashboard: Updating airbyte source config 2024-01-09 14:04:52 +01:00
.gitignore gitignore: add __pycache__ 2021-03-15 19:08:48 +01:00
README.md README: update 2023-11-23 11:14:44 +01:00
comm_extraction.py comm_extraction: fix dag 2024-04-30 18:21:23 +02:00
dbt.py Merge branch 'test' into prod 2024-02-07 18:20:15 +01:00
dbt_spiff.py Fixed typo and schedule for Treasure Sync and created Mod Prod Spiff dag sync (#6) 2024-04-23 10:58:07 +02:00
forums_sync.py forums_sync: creation of separate DAG 2024-03-06 09:18:47 +01:00
github_extraction.py Refactorying DAG for github data extractions 2023-12-14 13:43:55 +01:00
infra_sync.py infra_sync: init dag 2024-04-30 18:21:22 +02:00
logos-org-map.py forums_sync: creation of separate DAG 2024-03-06 09:18:47 +01:00
spiff_extraction.py Refactoring DAG for spiff data extractions 2023-12-14 14:13:47 +01:00
treasure_dashboard.py Fixed typo and schedule for Treasure Sync and created Mod Prod Spiff dag sync (#6) 2024-04-23 10:58:07 +02:00
website_sync.py website_sync: fix execution order 2024-02-22 14:36:27 +01:00

README.md

Description

This repo contains implementations of Airflow workflows and tasks called respectively DAGs and Operators.

  • DAGs - Direct Acyclic Graphs - Python scripts defining workflows in a way that reflects their relationships.
  • Operators - Python functions which define the individual tasks that are executed as part of a DAG run.

To learn how to write DAGs and Operators read about core concepts and follow the official tutorial.

DAG

This repository contains:

  • website_sync: DAG to launch the Airbyte jobs for the status-website charts.
  • spiff_sync: DAG to synchronize Spiff workflows data
  • dbt: DAG to run all the dbt models,
  • gh_sync: DAG to synchronize data from repository (logos, waku, codex)

The DBT models run in some DAG are stored in dbt-models.

Continuous Integration

Changes pushed to master are automatically fetched to our Airflow instance by the airflow-webhook service.

Branches

This repos has 3 working branches:

Infrastructure

All Airflow infrastructure is managed in the infra-bi repository.