DATA VALIDATION SYSTEM

Data Management Framework

Data Valıdatıon Requırements

The data validation is of great importance for organizations such as daily resource and target transfer tests, update and data migration projects, and control and supervising of live systems from the perspective of organizations.

Source To Target Load

  • ETL Validation
  • Data Masking

Upgrade and Data Migration Projects

  • ETL Version Upgrade
  • Data Migration
  • Application or Database Upgrade

Live System’s Controls

  • ODS
  • Data Warehouse
  • Datamart
  • B2B
  • Applications

Traditional Methods and Challenges

Traditional Methods

  • Data controls are carried out at database level or within ETL systems.
  • Rule definitions are not documented.
  • Analyzes and improvements are made on script-based, non-visual screens.

Challenges

  • Access support is required for each different database and is not managed from a central system.
  • It is not known which rules are applied.
  • It causes learning difficulties and job losses, Also it is open to mistakes.
  • The results cannot be followed up.

Data Validation System

Data Validation System aims to establish a sustainable and traceable infrastructure within the company by ensuring that validation rules are managed from a single center, thereby increasing efficiency.

Data Validation Pre-Built Rules

Data Validation Pre-Built Rules

  • Built-in and Extendable Data Validation Rules
    • Reusable rules
    • New columns can be add easily to validate
    • Ability to extend with Adhoc queries
  • Ability to Define Threshold for Rules
  • Performance Management
  • Email Notification for Critical Controls
  • Provides Reporting Infastructure

Data Validation System Scoring

  • Different types of scoring
    • Domain-based scoring
    • KPI based scoring
  • Easy configuration for new KPI’s.
  • Weights and metrics can be managed easily
  • Trend analysis with scorecards

Technical Architecture

Data Validation System Approach

Slide
Analyze Capabilities
  1. Pre-defined Turkish data quality rules for name, surname, citizen number etc.
  2. Column Profiling for gathering statistics such as
  3. Value Frequency and Pattern
  4. Null and Distinct value ratios
  5. Max and Min values
  6. PK – FK Analysis
  7. Join Analysis
Slide
Measure Capabilities
  1. Rol based and drag&drop user interfaces
  2. Pre-Built transformation such as joiner, filter etc.
  3. Exception management
  4. Scorecards
  5. Drilldown data in development
  6. Definition of built-in and extendable data validation rules
  7. Displaying data validation results with different metrics such as
  8. Completeness, Uniqueness, Integrity, Accuracy etc
  9. KPI scoring
Slide
Manage Capabilities
  1. Create & Schedule jobs
  2. Monitoring Results and Mail Notifications
  3. User privileges
  4. Detailed error logs
  5. Easy integration for new sources and databases

Are you ready to

Accelerate Your Business?

LET’S GO