COMPRESSING DATA CUBE IN PARALLEL OLAP SYSTEMS

This paper proposes an efficient algorithm to compress the cubes in the progress of the parallel data cube generation. This low overhead compression mechanism provides block-by-block and record-by-record compression by using tuple difference coding techniques, thereby maximizing the compression ratio and minimizing the decompression penalty at run-time. The experimental results demonstrate that the typical compression ratio is about 30:1 without sacrificing running time. This paper also demonstrates that the compression method is suitable for Hilbert Space Filling Curve, a mechanism widely used in multi-dimensional indexing.


INTRODUCTION
Data warehouses provide the primary support for Decision Support Systems (DSS) and Business Intelligence (BI) systems.One of the most interesting recent themes in this respect has been the computation and manipulation of the data cube, a relational model that can be used to support On-Line Analytical Processing (OLAP).Data cube based OLAP systems pre-compute multiple views of selected data by aggregating values across all possible attribute combinations (a group-by in database terminology).For a d-dimensional input set, there are 2 d possible group-bys.The resulting data structures can then be used to dramatically accelerate visualization and query tasks associated with large information sets [30].
Within the context of massive data volumes, data cube computation has to be very efficient with respect to speed and space.Many research studies have shown that parallel computation effectively speeds up data cube construction.Data cube compression, on the other hand, is not only crucial for computing and storing data cubes in limited space, but also reduces I/O access time.Though compression algorithms are quite common in the literature, most are poorly suited to database/cube environments since they (i) offer relatively poor compression ratios or (ii) result in significant run-time penalties.The existing database compression techniques, such as BIT, Block-BIT and TDC [20,37,39,40], preserve the structure of relations, and therefore support efficient random query and update operations.Some of them are also applied to data warehousing applications and achieve compression ratios of between 3 and 4 to 1 [20,39].This paper focuses on the investigation of even more efficient data cube compression techniques in the context of parallel OLAP computation systems, the Parallel Algorithms for New Data Warehousing Architectures (PANDA) [12,14,13,15,16] framework in particular.It is worth noting that sequential data compression algorithms can be applied in PANDA's parallel OLAP computation framework since each single view is processed by one processor at a time.

RELATED WORK
In the context of data warehouse applications, we are only interested in lossless data compression techniques, which allow the original data to be fully recovered form its compressed form.There are a variety of data compression algorithms.Current techniques of lossless data compression may be categorized into two broad classes: statistical techniques, such as Huffman coding, and dictionary techniques, such as Lempel and Ziv algorithm [17,24,25,46,47,52].There are two very important properties of these conventional data compression techniques: 1).Data is processed serially (FIFO) in either on-line or off-line compression/ decompression environments, 2).Data models, either statistical or dictionary models, must be consistent.The encoder and decoder share the same model.We note that the random, localized access of databases, such as tuple query, insertion, deletion, and update, conflicts with the serial processing and consistent data model properties of traditional data compression.
Database compression techniques have been researched since the 1990's.One class of solutions focuses on the application of traditional data compression techniques in relational database systems at four different levels, namely the file (relation) level, the page (block) level, the record (tuple) level, and attribute level.Some of them are more interested in page level compression.Since a block is a unit of I/O access, there is no extra I/O cost for maintaining the data model's consistency.Oracle applies a block-based dictionary compression technique, which reaches about a 3.1 compression ratio for a database of 55GB of data without a performance penalty in data warehouse applications [39].Another actively researched class of database compression solutions tries to find more efficient data distributions from the characteristics and knowledge of the relation, thus achieving high compression ratios.Also the tuple-structure of a relation is preserved in its compressed form in order to support high performance database operations through the avoidance of unnecessary compression and decompression.
Ray, Haritsa and Seshadri proposed the Column Based Attribute Compression algorithm (COLA) in 1995 [40].Instead of having a single frequency distribution for the entire relation, COLA uses a separate frequency distribution table for each attribute in a relation, since data belonging to the same attribute is usually semantically related and is expected to have a similar distribution.The experimental compression percentage is 21.27% for a synthetic numeric relation.The size of each tuple is 76 bytes and the relation has 9964 records.Bit compression (BIT) is a well-known technique that represents each numerical attribute in bits, instead of bytes.Goldstein, Ramakrishman and Shaft proposed a derivative algorithm of BIT in 1998 by compressing relations in blocks [20] (we refer to it as Block-BIT in this paper).The idea of Block-BIT is as follows.Each compressed block stores a 'frame of reference', the minima and maxima of all dimensions in a relation.All attributes for each tuple in the compressed block are presented by storing just enough bits to distinguish between the values in this range.The typical compression ratios on real data sets are between 3 and 4 to 1.It reaches as high as 88 to 1 on low cardinality data sets.The CPU cost of decompressing a relation is approximately 1/10 the CPU cost of GZIP.
Ng and Ravishankar proposed a block-oriented database compression technique, the Tuple Differential Coding (TDC) method [37].In TDC, all attributes in a relation are mapped into numeral domains.Tuples are converted into ordinal numbers in ascending mixed-radix order.A compressed block only stores the value of the first tuple as a reference.Each succeeding tuple is replaced by its difference with respect to its preceding tuple.The i th tuple in the relation can be reconstructed from the first tuple and the first i−1 difference values.Algorithm 1 describes the main steps of the TDC compression techniques.The typical compression ratios of TDC are between 4 and 6.6 to 1 for the tables with 106 tuples with 8 dimensions [37].
Algorithm Instead of coding tuples by using differences for each attribute as Block-BIT does, TDC uses the difference in the tuple level, which also keeps the tuple information of each attribute.In fact, the tuple level difference value removes the duplicated attribute values, and only keeps the different attribute values and compacts them into one integer and, as such, compresses the whole relation.

DATA CUBE COMPRESSION
This section proposes an efficient data cube compression algorithm -XTDC, and the corresponding data structure, as well as a number of basic compressed data cube operations based on the compacted data structure.

The XTDC Data Cube Compression Strategy
The multi-dimensional model is the most popular model used in data warehousing environments to support OLAP operations [30].Data cubes, generated from fact tables, consist of the surrogate keys of the dimensional tables plus the measure fields.These surrogate keys are usually consecutive integers, which are automatically generated during the data warehouse Extract, Transform and Load (ETL) stage.
Considering the properties of the data warehouse and data cube computation, we propose the eXtended Tuple Differential Data Cube Coding (XTDC) strategies that builds upon a number of ideas of existing techniques, specifically, TDC [37], BIT and Block-BIT [20]: 1).Treat dimensional data and measure data separately.The main I/O access task is related to the large views, which have a large number of records and high dimensions.The major objective in compressing data cubes is therefore to reduce the storage of dimensional data.Given the characteristics of data warehouse applications, the dimensional data are usually represented as integers.The compression ratio mentioned later in this paper is the ratio of the size of the original dimensional data divided by the size of the compressed dimensional data.2).Compute tuple differences to compress the dimensional data at the block level.XTDC uses the fundamental idea of the Tuple Differential Coding method [37] to code the tuples in block wise fashion.Each tuple is represented using the difference between it and its preceding tuple.In order to avoid the risk of data overflow in the case of large views with high dimensions, two tuple operations, tuple-add and tuple-minus, are proposed to support a wider range and faster data computation.3).Compact the differences into bits.4).Compact all the differences together to remove gaps caused by byte-alignment.The XTDC stores the differences together in order to remove the gaps that are caused by byte-alignment.All the measure data are stored in the second part of the block.5).Dynamically determine the number of tuples to be compressed into one block.Since the number of bits for each tuple is determined by the value of the maximum difference in one block, XTDC dynamically computed the number of tuples that can be compressed in one block in order to encode as many tuples as possible into this block.6).Use a counter mechanism to represent consecutive 1-differences.For those views that have low dimensions but a large number of tuples, there is very high probability that the difference values of conjunctive tuples are 1's because the attribute values of each dimension are usually consecutive integers.XTDC uses a counter to represent these consecutive 1's in each data block in order to improve compression efficiency in this case.7).Keep the compression information in each block.The data compression information, such as the number of tuples in the current block and number of bits of each difference value, are stored in the block header.This information is dynamically calculated during the compression process and is used during decompression.
XTDC is a block-level lossless data cube compression technique.It uses the knowledge of the characteristics of the multi-dimensional data model to guide the compression and decompression processes, rather than analyzing the data distribution statistics as with most conventional data compression techniques.XTDC preserves tuple structure in compressed views in order to get the benefits of database compression.
The following section describes the details of the XTDC compacted data structure.

The XTDC Compacted Data Structure
XTDC keeps the compression information in each block, namely the block header, in order to preserve the tuple-structure in compressed data for high performance data access and to compress data cubes in block level.In this way, each compressed data block contains all necessary information to decompress this block or to localize the required data (tuples) from the compressed data directly.It is also good for efficient indexing to store the uncompressed first tuple in the block header.XTDC compresses dimensional data separately and leaves the measure data in uncompressed form.Each tuple is encoded by its difference value and stored in bit form.In order to avoid the spare bits between tuples that may be caused by byte-alignment issues, this data structure stores all of the compressed dimensional data together and puts all of the measure data in the remaining space in the same block.Table 3.1 presents the typical structure of a compressed block.It consists of three parts: 1).The Block header contains the compression information for this block.The length and the content of the block header may vary according to the different compression algorithms.2).The Dimensional data area contains in bit form all the compressed dimensional data -the difference values -of the tuples in this block.3).The Measure data area contains all the measure data in original form (uncompressed format).The offset of this segment in the current block is given by: measure offset = length of block header + ⎡(number of tuples − counter) × (number of bits for dimensional data)/8⎤.The offset of the measure data of i th tuple is (measure offset)+(i−1)×(number of bytes for measure data).
Since most data cube operations are read-only in data warehouse applications, XTDC focuses on storing as many encoded tuples in a block as possible, rather than designing a more flexible data structure for update operations.The following section discusses the details of using this data structure in the XTDC algorithms.

The Extended Tuple Differential Data Cube Coding Algorithm -XTDC
As we discussed previously, the principle idea of the tuple differential coding algorithms is to store the difference values of the consecutive tuples.The mixed-radix values of tuples can be calculated according to Eq.(3.1) of Definition 1 [37].
The value of one tuple < a 1 , a 2 , ..., a n > is defined as:

Tuple operators
In enterprise level data warehousing environments, a view to be compressed may have high dimensions with high cardinalities.Consequently, there is a very high risk of data overflow when mapping each tuple to a mixed-radix value in such environments.However, when views are fully sorted, the differences between two conjunctive tuples are usually very small.We propose two tuple operators, tuple add and tuple minus, in order to encode views safely and efficiently.Theorem 1 presents the principle of the operators.
Theorem 1 Given two consecutive tuples: < a 1 , a 2 , ..., a n > and < a' 1 , a' 2 , ..., a' n >, the difference value of these two tuples is: Proof.Apply Eq.(3.1) of Definition 1 to the left hand side of Eq.(3.2), we have: Combining the right hand side of Eq.(3.3), we get the right hand side of Eq.(3.2).
Algorithm 2 calculates the difference by directly manipulating the attribute values of the tuples to avoid data overflow.It is also efficient because it reduces the multiplicative operations to addition operations.

Algorithm 2. Tuple Minus Input: Two tuples T 1 , T 2 with d-dimension; Cardinalities (C[i]) for each attribute domain (A[i])
Output: The difference between the mixed-radix value of T 2 and T 1 1: difference = 0; 2: During the decompression process of XTDC, we can exploit the fact that the preceding tuple has already been decoded into its uncompressed format when decoding the current.Algorithm 3 directly operates on attributes of tuples to avoid computing the mixed-radix values.

The XTDC Algorithms
XTDC is a block-level compression technique using the XTDC compacted data structure.The encoded dimensional data, the differences, are compacted by bit and grouped together to save maximal space.The block information, such as the number of tuples per block and the first tuple (in uncompressed format) of the current block are stored in the block header.The number of bits for each tuple difference value is dynamically determined by the maximum value of the differences.Therefore, given the fixed block size, the number of tuples per block varies according to the number of bits per tuple during compression.A counter of consecutive single increments of tuples is stored in the block header as well.In some cases, such as compressing views that have a large number of tuples but relatively low dimensions and cardinalities, it is most likely that the values of consecutive tuples are consecutive integers, which means the differences are 1's.
As Algorithm 4 presents, the XTDC data cube compression algorithm dynamically determines how many tuples can be compressed in one block.For each block, XTDC employs two phases: 1).The Computation phase collects all the information for one block.XTDC calculates the differences of conjunctive tuples by using the tuple_minus operation shown in Algorithm 2 (line 4) and dynamically computes the number of tuples in one compressed block (line 5).XTDC checks every tuple to determine if it can fit in the current block according to the changing value of the maximum difference(max_difference) and the number of consecutive 1's(counter).The differences of those tuples are stored in a buffer(difference_buf) in integer form in the first phase.
2).The Compact phase creates a compressed block.After collecting enough differences values for one block (or all tuples have been encoded), XTDC computes the offset of the measure segment of current block (line 15).This offset(offset) is calculated according to the block header.The measure data is copied to the measure area of the block in uncompressed format (line 20).All of the differences, calculated in the first phase, are compacted in bit format into the dimensional area of the block (line18).Each of these differences occupies ⎡log 2 (max_difference)⎤ bits.Algorithm 5 shows the XTDC decompression process.Note that the XTDC technique supports access to each tuple at the block level without loading the whole view and decompressing it.However, in this particular project, we use the XTDC interface to decompress the whole view immediately after loading it into main memory.So, the algorithm presented here loads the whole compressed view.In any case, XTDC retrieves the compression information from the block header (line 3), and computes the offset of the measure segment of the current block (line 4, same as Algorithm 4).The first tuple is retrieved from the block header as well (line 5) and each tuple is computed by adding the difference value to its preceding tuple using the tuple_add (line 7,12) shown in Algorithm 3. The differences are either 1's (for the first counter tuples) or else are retrieved from the dimensional area (line 11).The measure data is simply copied to the output buffer (line 8, 13).The next block is processed (if necessary) when all tuples of the current block are decoded.

Applying the Hilbert Space Filling Curve Technique to XTDC
The XTDC algorithm uses the tuple differential coding method to compress data cubes.Tuples are sorted in a specific order, then converted into an integer representation.The difference between consecutive integers is calculated and used to represent the original data.The method that performs the integer mapping must be "one-to-one" in order to convert (decompress) the compressed integer back to a unique tuple representation.The Hilbert Space Filling Curves technique traces a unique pathway though the points of a multidimensional space.In this sense, it may be used to provide a clean one-to-one mapping of a tuple to its ordinal or "indexed" position in the hypercubic space.We apply the Hilbert ordering to XTDC compression process using the following two phases: 1).Hilbert ordering.Maps the tuples (dimensional data) of the views to the sequence of the Hilbert Space Filling Curve and sorts the views by these sequential values.We note that a Hilbert re-sorting is required here since the core cube aggregation algorithms use a lowX ordering.2).XTDC encoding.Uses steps similar to those found in the standard XTDC approach shown in Algorithm 4 except that the tuple_minus is the simple integer minus and the first tuple is stored in its Hilbert sequence value in the block header.
The decompression process is composed of two phases: 1).XTDC decoding.Uses steps similar to those found in the standard XTDC approach shown in Algorithm 5 except that the tuple_add is the simple integer add.2).Hilbert de-ordering.Maps the sequence of the Hilbert Space Filling Curve to the dimensional data of the tuples.
It is very important to note that the PANDA system utilizes the Hilbert Space Filling Curve to compute the sorted views for multidimensional indexing.As a result, there is a significant potential to improve the data cube compression performance since we effectively can get the Hilbert sorted views for "free".

Compressed Data Cube Computation
Unlike conventional data compression techniques, XTDC preserves the tuple structure in compressed data, thereby allowing the OLAP computation system to often manipulate the data cube in compressed format.By avoiding unnecessary decompression and compression computations, XTDC not only reduces the storage requirement and I/O bandwidth, but also reduces main memory requirements.This section describes how the XTDC data structure supports high performance data cube computations based on compressed data.Two of the most important operations -random query and sub view generation -are discussed.

Random Query
The XTDC algorithms are able to retrieve one single tuple from a compressed view without decoding the whole block.They also improve the quality of index structures such as B-trees and R-trees by reducing the number of leaf blocks.Algorithm 6. Locating One Specific Tuple in a Compressed view.Input: A compressed view in XTDC data format.Dimensional data of the specific tuple, t Output: Measure data of t (NULL for non-existing tuple) 1: Locate the block that may contain t by checking the first tuple in block headers 2: Load the entire block in to main memory 3: Compute the difference (v) between the required tuple (t) and the first tuple 4: Accumulate the first i different values until we reach one that is equal to OR greater than v 5: Return NULL if the different value is greater than v 6: Compute the offset of the measure data segment in the current block according to the header information 7: Return the i th measure data Algorithm 6 presents the major steps of localizing a specific tuple in a compressed view.The details may vary slightly due to the different algorithms used in the actual implementation.In any case, the block header stores the information about how this block was compressed.The block which may contain the specific tuple (t) can itself be located as per ordinary indexing methods.Since the compressed view is sorted, the tuple t (if it exists) must be in the block which has the biggest header tuple value that is smaller than or equal to the value of tuple t.By using indexes, we only load the block when absolutely necessary.The difference between tuple t and the first tuple is computed using the Tuple_Minus method shown in algorithm 2 (line 3).The sequence number (i) of the tuple(t) is calculated by accumulating the difference values retrieved from the dimensional data area of the block (line 4).If the summation of the difference value is greater than v, we know that the tuple t is not in this view at all.A NULL value is returned at line 5.The offset of the measure data area can be computed (line 6) according to the header information (as Section 3.2 discussed).The i th value of the measure data area is the measure data of t.

Sub View Generation
Generating sub views from a given view (parent view) is one of the primary operations of data cube computation [16,32].The XTDC technique allows OLAP computation systems, such as PANDA, to compute a compressed sub view from a compressed parent view directly.By doing so, we avoid decompressing the whole parent view and then compressing the target sub view since the output of the computation is in compressed format.Again, the main memory requirements are reduced significantly because the compressed data is much smaller than the original one.
In the XTDC technique, we only use dimensional attributes to compute the tuple value.The tuple value of a sub view can be calculated by using the value of the corresponding tuple of its parent view.
Theorem 2 Given a parent view R, the tuple value, ϕ', of its k-subview, R', is: Proof.According to Eq.(3.1) of Definition 1, the tuple value, ϕ', of k-subview is The value, ϕ, of the corresponding tuple in the parent view can be expanded as: Applying Eq.(3.6), we have 3.7 and 3.8: Load one block of V p .

4:
for all tuple(t i ) of Vp do 5: Compute the value v pi of tuple t i .

6:
Get measure data m pi of t i .

7:
Compute the corresponding tuple value, v si , in V s .

8:
Accumulate the measure data of V s : view_buf[v si ]+= m pi .9: end for 10: until all blocks of V p are processed.11: Construct the k-subview: Compact the view_buf in XTDC format.
Theorem 2 presents a way of computing the tuple values of a sub view without decoding the corresponding tuples of the parent view.Algorithm 7 shows the main steps to compute a k-subview from a parent view using the XTDC data structure.A temporary buffer(view_buf) is used to construct the k-subview(Vs).For each tuple (t i ) of the parent view, its value, v pi , is calculated by using Tuple Add in line 5. Line 7 computes the corresponding value (v si ) of the sub view according to Theorem 2. Line 6 locates its measure value in the measure data area as shown in algorithm 5.The measure data is summarized in line 8.After processing all of the tuples of the parent view, line 11 generates the sub view by compacting view_buf: For those entries with values, their indices are the values of tuples in the k-subview and their values are the measure data of the corresponding tuples.The compaction process simply calculates the differences of conjunctive indices and stores the data in XTDC format.

Compressing Data Cube in the PANDA System
PANDA supports high performance parallel data cube computations.Its I/O Manager is a significant feature that handles efficient I/O access during the manipulation of massive data sets.We implement the XTDC algorithm as a Compression Interface and plug the interface into the I/O Manager in order to compress (and decompress) data cubes.In the data-writing phase, the tuples in the view buffer are compressed by block before they are physically written to disk .In the data-loading phase, the entire compressed view is loaded from disk and decompressed in main memory (Input Buffer).The details of the system structure and the XTDC Compression Interface implementation are discussed in [16,33].
In this section, we discussed the efficient data cube compression algorithm, XTDC, and its corresponding compact data structure, as well as two OLAP operations -random point query and sub view generationbased on this data structure.We also demonstrated that the XTDC algorithms can utilize the Hilbert Space Filling Curve technique.Therefore, it has potential for use in OLAP systems that use the Hilbert space technique for multidimensional indexing.Finally, we introduced the strategy of applying the XTDC algorithms to a parallel OLAP computing system -PANDA.In next section, we will demonstrate the experimental results and evaluate the performance of the XTDC algorithms.

EVALUATION
This section evaluates the performance of data cube compression techniques implemented in the PANDA System.The main goal of data cube compression is to reduce the space requirements of data cube computation while maintaining reasonable response time.Our tests therefore focus on two main issues: compression ratio and compression/decompression speed.
1).Compression ratio (CR): In the context of data cube compression, our implementations compress the dimensional data and leave the measure data in uncompressed form.Our evaluations use the dimensional data compression ratio, CR =(dimension size without compression) / (dimension size with compression).Given the compression ratio (CR), the expected data size reduction is REDUC = (1−1/CR)*100% [42].2).Compression and decompression speed: To compute a view in PANDA with data cube compression, the parent node (view) of it is read and decompressed, and then the target view is compressed before being written to disk.Both compression and decompression processes are involved in data cube computation.We use wall-clock running time to evaluate the speed performance for both single view computation and full data cube computation.
In the multi-dimensional model, a data cube is organized in exactly the same format as that of a conventional relational table.The database compression techniques can be simply applied directly to data cube compression in order to reduce the storage space.In order to compare the compression efficiency between XTDC and the existing database compression techniques, we implemented the TDC [37] and the BIT database compression algorithms.It is worth noting that when we apply the TDC database compression algorithm to achieve data cube compression in PANDA, we follows the fundamental ideas of [37] except that we use our tuple computation algorithms, tuple_minus and tuple_add, to avoid data overflow during the computation of tuple differences.As proposed by [37], TDC uses Run Length Coding (RLC) to encode the number of leading zero components in each difference.In our particular implementation, we store the differences in integer form (4-byte), which costs less than using RLC encoding.
We also plug an Open Source conventional compression library, BZIP [6,45], into the PANDA system.We choose maximal buffer size, 900KB, plus the size of the output to achieve the highest compression ratio that BZIP could reach.We divide our evaluation into two major parts.Section 4.2 evaluates the compression efficiency, both in terms of compression ratio and compression time, for single view computation.Section 4.3 offers the same analysis, but does so in the context of a fully materialized PANDA data cube.This section also demonstrates the speedup evaluation in parallel OLAP computation.In both sections, we provide a performance comparison of the BIT, TDC and XTDC data cube compression algorithms, as well as a conventional data compression algorithm, BZIP.All of our tests were conducted on a Linux cluster, whose primary characteristics are listed below [26] With respect to time oriented tests, such as the full cube pipesort comparison shown in Figure 4.5, the running time is the average of three timed runs using the same input datasets.For the compression ratio evaluation tests, the results depend on the meta data associated with the views, i.e. cardinalities, dimensions and number of tuples.All of the output data is also decompressed by using the same module that is plugged into the IO Manager and is compared with the results obtained from the original system.

Test Cases
Real data warehouse systems usually contain between 4 and 15 dimensions [30].We will look at a sequence of data cube compression tests, each designed to highlight one important characteristic.We evaluate fact tables with 6 to 10 dimensions.The number of tuples in these fact tables ranges from 100K to 2M.The fact tables themselves are created with PANDA's Data Generator [16] by specifying parameters such as the number of tuples in the data set, the number of feature attributes (dimensions), and the number of unique values (cardinality) in each dimension.In effect, we utilize a set of base parameters and then vary exactly one of these parameters in each of the tests.These base parameters are (with defaults listed in parenthesis): a) Fact Table Size (1M), b) Dimension Count (10).The meta data, which is a group of arbitrary cardinalities that is used by all of our test cases, is listed in Table 4.1

Single View Compression
The efficiency of compression techniques can be clearly evaluated on single view tests.We use the Data Generator [16] to generate a group of fact tables with 500K, 1M, 2M, 5M, and 10M tuples respectively.All of these fact tables have 10 dimension attributes and 1 measure field.We arbitrarily create a group of single views corresponding to each fact table by using the Partial Data Cube generation module in PANDA.Each of these views has 7 dimensions and 1 measure field.The number of tuples and the original size of these views are listed in Table 4.2.For each single view, we apply different compression techniques, including BIT, TDC, XTDC, BZIP, and Linux GZIP.Because GZIP and BZIP are global range data compression techniques, their compression ratios are computed as total compression ratio.The BZIP libraries [45] are plugged into the same test harness as BIT, TDC and XTDC.Both options for best compression (gzip -9) and for fast compression (gzip -1) of GZIP are used to evaluate compression ratio and running speed.4.2.The compression ratios of BIT, TDC, BZIP and GZIP are between 5 and 12 to 1.With the increasing size of views, the ratios of conventional compression techniques (BZIP, GZIP) slightly increase because there are better data distributions in a larger range.The number of tuples in a view does not affect the compression ratio of BIT.In fact, the compression ratio is only determined by the number of bits for every tuple, which corresponds to the cardinalities of the dimensions.The compression ratio of TDC remains stable since TDC stores differences in integer form, which costs 4 bytes in our system, no matter how small the differences are.The experiments show that XTDC reaches compression ratios between 26 and 51 to 1, which are much higher than the other techniques.In XTDC, the number of bits required to store the differences in a block is determined by the maximal difference of consecutive tuples in that block.With the same dimension cardinalities, the higher the number of tuples in a view, the smaller the maximal difference in that block might be, and the greater the likelihood of consecutive 1-differences.Both the bit compaction technique and the counter mechanism help XTDC to reach higher compression ratios with an increasing number of tuples in a view.

Full Cube Compression
In full cube tests, 2 d single views are created.These views cover all the possible combinations from 1dimension to d-dimensions.The efficiency of single view compression will definitely affect the full cube compression.Figure 4.3 presents the average of compression ratios for full cube computation.The fact tables of these cubes have 10 dimensions and the same cardinality distribution as listed on Table 4.1.
Consistent with the result in Figure 4.1, XTDC reaches a much higher compression ratio than the others (BIT, TDC) do.It is worth noting that the fully materialized data cube is much bigger than the fact table.In one of our test cases -using a fact table with 10 dimensions and 10 6 tuples -the dimensional data of the fact table is 40MB, while the total dimensional data in the full data cube generated by this fact table is 9778 MB.XTDC reaches a 29.4 to 1 compression ratio, which reduces the dimensional data from 9778 MB to 333MB. 3 also shows that XTDC is more efficient in terms of compressing the full cube that has been generated by the fact table with the same dimensions but a larger number of tuples.In this experiment, XTDC reaches a 31.8:1compression ratio when the fact table of the cube contains 2×10 6 tuples.Note that the compression ratios are lower on the full cube than the single views we tested in Section 4.2.As we discussed previously, XTDC uses one difference value (several bits in many cases) to represent the dimensional data of one tuple.As the number of dimensions decrease (and most views have less than the 7 dimensions used in the single view test), the ability to compress diminishes as well.Conversely, the greater the number of dimensions, the greater the benefit for compression.We also note that the compression ratios of BIT and TDC are not significantly affected as the number of tuples increases.Since the BIT algorithm uses bit form to express each attribute, the compression ratio only depends on the cardinalities of the dimensions.With respect to the TDC algorithm, the tuple difference value is represented as an integerfour bytes in our implementation.The compression ratio depends on the number of dimensions instead of the number of tuples.5 to 1 for a larger data cube, which is generated by using a fact table with 10 dimensions and 10 6 tuples, and does so in less than 900 seconds.

CONCLUSION AND FUTURE WORK
This paper proposes an efficient data cube compression algorithm, XTDC, and its corresponding compact data structure.Building upon a number of existing compression algorithms, XTDC is effectively a combination of the following techniques: 1).Tuple differential coding: Tuples in the views are mapped to integer values and the differences of the conjunctive tuples are used to represent the views.2).Bit compaction: The tuple differences are stored in bit form and are compacted.
3).Block-wise compression: The tuples are compressed in blocks (pages).This not only increases the compression ratio by reducing the value range of the differences, but also makes more efficient data access since all the compression information is localized in individual blocks.4).Handling dimensional data and measure data separately to get rid of the gaps caused by bytealignment.5).Using meta data information: Knowledge of the data cube is used when compressing and decompressing.
The XTDC technique preserves the tuple structure in compressed data cubes.Its data structure makes the compressed blocks accessible to common indexing methods such as B-trees or the packed R-trees that are actually used by PANDA.Since all information about tuples is encoded in individual blocks, the data cube operation can be done when the data cubes are still compressed.We propose two algorithms for random access and sub cube generation based on compressed data, which shows the possibility of manipulating the compressed data cube without decoding the whole views, thereby improving OLAP computing performance.We also discuss that the Hilbert Space Filling Curve technique is well suited to the XTDC algorithms.Therefore, the XTDC technique has great potential for use in practical cube systems that use space filling curves for multidimensional indexing.
The experimental results show that the XTDC technique is well suited for parallel OLAP computing systems.By integrating the XTDC algorithms into the PANDA system, the storage space requirements for OLAP computation are greatly reduced with very little performance penalty.The typical compression ratio is 29.4 to 1 for a full cube generation, in which the fact table has 10 dimensions and 10 6 tuples.The dimensional data reduction is from 9778MB to 332MB (96.6%).The experiments also demonstrate that the XTDC algorithms have the ability to achieve higher compression ratios for larger data cubes which have more dimensions and more tuples.
Since the XTDC technique preserves the structural information of the data cube in compressed form, it would be possible to extend the data cube operations on compressed data using the current design.Below we identify a number of these possibilities: 1).Indexing compressed data blocks.As noted previously, the compressed data cubes are sorted and organized at the block level.The first tuple of a block is stored in the block header in uncompressed format.It is very convenient to build an index for fast random tuple location.Since the number of blocks is reduced for the compressed view, the size of the index file is significantly decreased as well.2).Computing compressed data cubes.In our current implementation, an XTDC interface is plugged into PANDA to compress and decompress data during the physical I/O phases.PANDA computes views using uncompressed data in main memory.Section 3.5 discussed the XTDC technique is able to generate compressed sub views from a compressed view.By doing so, we will not only save main memory space during data cube computation, but also avoid most of the data compression and decompression processes.
As we noted that the XTDC algorithms is suitable to the Hilbert Space Filling Curve and the PANDA system utilizes the Hilbert ordering to compute the sorted views for multidimensional indexing.As a result, there is a significant potential to improve the data cube compression performance since we effectively can get the Hilbert sorted views for "free".Both data cube compression and multidimensional indexing can share the full benefit of Hilbert ordering.

Algorithm 7 .
3.7 and 3.8 to the right hand side of 3.4, we get the right hand side of 3.5, which is ϕ'.Construct a Compressed k-subview From a Compressed Parent View.Input: V p , a compressed parent view in XTDC data structure.Output: V s ,the compressed k-subview in XTDC data structure.1: Initialize view buffer: view_buf.2: repeat 3:

Figure 3 . 1 .
Figure 3.1.An illustration of data cube I/O with the compression interface Figure 3.1 illustrates the PANDA's I/O access mechanism with respect to the data cube compression feature.In the data-writing phase, the tuples in the view buffer are compressed by block before they are physically written to disk .In the data-loading phase, the entire compressed view is loaded from disk and decompressed in main memory (Input Buffer).The details of the system structure and the XTDC Compression Interface implementation are discussed in[16,33].

Figure 4 .
Figure 4.1 compares the compression ratios attained by applying data cube compression techniques and global compression libraries (tools) to the views listed in Table4.2.The compression ratios of BIT, TDC, BZIP and GZIP are between 5 and 12 to 1.With the increasing size of views, the ratios of conventional compression techniques (BZIP, GZIP) slightly increase because there are better data distributions in a larger range.The number of tuples in a view does not affect the compression ratio of BIT.In fact, the compression ratio is only determined by the number of bits for every tuple, which corresponds to the cardinalities of the dimensions.The compression ratio of TDC remains stable since TDC stores differences in integer form, which costs 4 bytes in our system, no matter how small the differences are.The experiments show that XTDC reaches compression ratios between 26 and 51 to 1, which are much higher than the other techniques.In XTDC, the number of bits required to store the differences in a block is determined by the maximal difference of consecutive tuples in that block.With the same dimension cardinalities, the higher the number of tuples in a view, the smaller the maximal difference in that block might be, and the greater the likelihood of consecutive 1-differences.Both the bit compaction technique and the counter mechanism help XTDC to reach higher compression ratios with an increasing number of tuples in a view.

Figure 4 .
Figure 4.2 presents the total running time (compression time plus decompression time) of these compression algorithms.Data cube compression techniques (BIT, TDC, and XTDC) have the same range of running times, which are much faster than conventional ones.

Figure 4 .
Figure 4.3 also shows that XTDC is more efficient in terms of compressing the full cube that has been generated by the fact table with the same dimensions but a larger number of tuples.In this experiment, XTDC reaches a 31.8:1compression ratio when the fact table of the cube contains 2×106 tuples.Note that the compression ratios are lower on the full cube than the single views we tested in Section 4.2.As we discussed previously, XTDC uses one difference value (several bits in many cases) to represent the dimensional data of one tuple.As the number of dimensions decrease (and most views have less than the 7 dimensions used in the single view test), the ability to compress diminishes as well.Conversely, the greater the number of dimensions, the greater the benefit for compression.We also note that the compression ratios of BIT and TDC are not significantly affected as the number of tuples increases.Since the BIT algorithm uses bit form to express each attribute, the compression ratio only depends on the cardinalities of the dimensions.With respect to the TDC algorithm, the tuple difference value is represented as an integerfour bytes in our implementation.The compression ratio depends on the number of dimensions instead of the number of tuples.

Figure 4 . 4 .
Figure 4.4.Runtime for parallel full cube generation with compression Figure 4.4 presents the speedup of PANDA with data cube compression on multiple processors.The result shows that XTDC, as well as BIT and TDC, works very well with PANDA's parallel data cube computation.The running times are very close to the original ones.We do not include conventional compression techniques in the full cube tests because of the results of the single view experiments.In fact, the BZIP compression libraries significantly slow down full cube computation.In one of our experiments, PANDA with BZIP compression takes 872 seconds to generate a full cube using a fact table that has 10 dimensions and 5 × 10 4 tuples.The total compression ratio is 7.7 to 1.As we can see from Figure4.3 and Figure4.4,XTDC reaches a compression ratio of 22.5 to 1 for a larger data cube, which is generated by using a fact table with 10 dimensions and 10 6 tuples, and does so in less than 900 seconds.

Table 3 . 1 :
The XTDC data structure of a compressed data view block Finally, XTDC completes the block header (line 22) and starts to compute the next block.
Input: A compressed view(in_buf) and its meta data