Effective Agile Test Framework for Big data

Abstract –

As saying goes “Simplicity is the ultimate sophistications” the Testing frame work should be simple, agile, collaborative and scalable, which we will design and develop here to address these 3 major challenges , faced while testing any Bigdata applications especially Hadoop based techniques, First  Processing Time to crunch huge data sets? Second is validation required for all the phases of data transformation and transitions? Third how should we validating transformed/ processed data?

The framework developed here will address the above challenges; framework generates a small representative data set from any original large data set using space partition testing. Using this data set for development and testing would not hinder the continuous integration and delivery when using agile processes. The test framework also accesses and validates data at various transition points when data is transferred and transformed.

Introduction—

Big data is a big topic these days, one that has made its way up to the executive level. Most organizations may not yet fully understand what big data is, exactly, but they know he or she needs a plan for managing it. The big data construct theory described by Gartner’s has 3Vs – data Volume, Velocity, and Variety), are all growing dramatically and we’re being asked to process data ever more quickly so we can respond to events as they happen, and that data is coming from an ever wider array of channels, sensors and formats.

The velocity of data generation is high and these data have to be analyzed in a timely manner. People may derive valuable information by mining high volumes of datasets in various formats. Many programs that use big data techniques (e.g., Hadoop1/2) and processes big data are currently being developed. How to test such software effectively and efficiently is challenging. Currently presented issues in big data testing and proposed solution that generates huge and realistic data for databases likeYCSB Benchmarking for NoSQL DBs, SBK conductor tool Kit for Splunk etc. This paper focuses on generating small and representative datasets from very large sets of data. This can save the cost of processing large amounts of data, which hinders continuous integration and delivery during agile development. This paper introduces an Effective Agile scalable big data test framework to test Extract, Transform, and Load (ETL/ELT) applications that use big data techniques.

ETL (Extract, Transform and Load) / ELT ( Extract, Load and Transform) process, data is extracted from multiple data sources like RDBMS, NOSQL DBs, WebServer Logs, Third party data sets, Flat files, Machine Data etc, and then is transformed into structured format to support queries. Finally data is loaded into HDFS or any NoSQL DBs for customer / data scientists etc to view.

First Technical problem is processing petabytes of data takes days or weeks. Generating small and “representative” data sets for different data sources and clients quickly is challenging. Running an entire or part of historical data hinders the overall agile development process. We seek to find a smaller data set that represents from this larger population using the characteristics of domain-specific constraints, business constraints, referential constraints, statistical distribution, and other constraints, which is called representative data set here after. Second, data is transferred and transformed at different transition points during an ETL process. Should we validate the data at one point or all of them? Third, how should we validate transferred and transformed data? Manually validating high volumes of data is prohibitively costly, so this must be automated.

Test Framework—

To solve these three technical problems above, we have a scalable big data test framework to generate representative data sets and validate data transfers and transformation. Figure 1 shows that data coming from different sources (on the left) is stored at various storage services and regular storage places (on the right). The purpose of the test framework is to generate a representative data set from a large original data set. Both data sets may be stored at the same or different services. The representative data set can be used for the validation of data transfer and transformation. The test framework validates the data transfer by comparing the data before and after the transfer. Likewise, the framework also validates the data transformation with the requirements that specify how data is transformed.

Test Data Generation—

To control the size of the test set input space partitioning is used. Input space partition testing starts with an input-domain model (IDM). The tester partitions the IDM, selects test values from partitioned blocks, and applies combinatorial coverage criteria to generate tests. Figure 2 shows the general process of test data generation.

Consider “audits” (another structural/non structured data other than database) are generated to reflect how data are changed. To create a representative data set from lots of audits using IDMs, need to extract test values for every attribute of the audits. We will have to write a grammar to describe the structure of these audits. The test framework then parses the grammar against the audits to collect all test values, including nested attributes, and computes statistical distributions of the test values, resulting in a parse tree. After analyzing the test values and constraints, the IDMs for every attribute are generated and merged into the parse tree. The final representative data set is generated from the parse tree, which has audit attributes, their relationships, constraints, and IDMs. The representative data set is used and evaluated through repeated use.  Here we can also leverage machine-learning techniques to improve the IDMs for accuracy and consistent representative data set generation. Also use parallel computing and Hadoop to speed up data generation. Even if the process is slow, only need to generate an initial representative data set once for a project. Then the data set will be refined incrementally to adjust to changing constraints and user values.

Data Validation—

During ETL processes, we need to validate data transfer and transformation to ensure that data integrity is not compromised. Validating data transfer is relatively simple because we know expected values are equal to original values. If the data is transferred from a database to another, we can validate the source and target data quickly by checking the number of columns, rows, and the columns’ names and data types. Best practice is evaluating row count of sum of each column in a table; the validation can be automated when source and target data are provided. Validating data transformation is more complicated. For instance, we may aggregate data from ten source tables into one target table. Some of the columns of the target table use the same data types as original while other columns may use different data types. We have two plans at different validation granularity levels. First, we validate whether the target data has correct data types and value ranges at a high level. The test framework derives data types and value ranges from requirements, then generates test to validate the target date.

The second plan derives detailed specifications to validate every transformation rule. The test framework compares the source data with the target data to evaluate whether the target data was transformed correctly. Both plans require testers to write the transformation specification in a format that the test framework can read. Then the framework automatically analyzes the specification and generates tests to validate the transformation.