Cookie settings

Cookies are small text files stored on your device to identify you and can be used to remember user preferences and analyse traffic to further improve our website. We may share information about your use of our site with our social media, advertising and analytics partners. By clicking "Accept all cookies", you agree to the use of all cookies as described in our cookie statement or "Accept only essential cookies" to only use cookies that are necessary for the functioning of our site.

Read our cookie statement here.

You can choose to adjust your preferences at any time.

Wholesale Banking

Data’s inferno: 7 circles of data testing hell with Airflow

Real data behaves in many unexpected ways that can break even the most well-engineered data pipelines.

Seven layers of data testing that is used in CI setup and Apache Airflow pipelines

To catch as much of this weird behaviour as possible before users are affected, the ING Wholesale Banking Advanced Analytics team has created 7 layers of data testing that they use in their CI setup and Apache Airflow pipelines to stay in control of their data.

Read full article on Medium.