Proactive Data Quality Management: Implementing Automated Checks with Akshay Jain at AIE 2024

This talk will cover automated data quality checks performed by large organizations to execute data reliability checks on big datasets in real time using data profiling and machine learning techniques. The demo will use the open source library Deequ, Spark framework and reporting and notification tools to enforce data issues in a proactive manner. Akshay will be covering an example of a framework developed at Amazon and Visa to validate customer facing data and its integration with notification tools based on the statistical methods.