JP2012168671A - Data compression device, data compression method and data compression program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a data compression device capable of increasing compression efficiency without decreasing information accuracy.SOLUTION: A data compression device 1 comprises a storage part 10, a grouping part 11, an information accuracy determination part 13, an event conversion part 15 and a data compression part 17. The storage part 10 stores: an identification determination table that shows the weight of consistency when item information is consistent, and a retention level of information accuracy; an information accuracy determination table that shows an original data retention level determined from a distribution of each item information and the retention level; and an identification determination table that shows conditions for grouping based on a distribution of the weight of consistency between data and on an appearance frequency. The grouping part 11 groups identifiable data together based on setting information, with respect to each of data groups in event data 2. The information accuracy determination part 13 obtains the distribution of item information, and determines, based on the setting information, the original data retention level of each item, with respect to each group. The event conversion part 15 rewrites data information in accordance with the original data retention level. The data compression part 17 compresses a data group including the converted data.

Description

  The present invention relates to a data compression technique, and more particularly to a data compression technique for data having data items.

  In recent years, a large amount of data such as sensor data and log data has been generated, and there is a movement to obtain some knowledge from a large amount of data. Therefore, it is necessary to store or transfer a large amount of data, and data compression processing can be used to reduce the cost of storing and transferring data.

  As data compression processing, processing combining lexicographic compression processing and entropy compression processing is well known. The lexicographic compression process is a process of registering a character string in data in a dictionary and replacing the character string with a code indicating a registered part of the dictionary when an existing character string appears again. The entropy compression process is a process for compressing data by determining the length of a code to be assigned based on the appearance frequency of the code.

  Even if the above data compression processing is used, depending on the amount of data to be compressed, the amount of compressed data may be large, and improvements in compression efficiency have been studied.

  As a conventional method, the data compression rate has been improved by reducing the information accuracy of the data to be compressed (original data). As an example, when the original data includes time data indicated by “hour: minute: second (for example, 10:36:24)”, the value of “minute: second” is deleted and “hour (10)” is deleted. The pre-processing to convert the data into only time data was performed to increase the data compression rate. As another example, pre-processing for converting a numerical value (0.47924927897) included in the original data into a numerical value (0.479) with a small number of significant digits is reduced.

  Also, a process is known in which only necessary field data is extracted, compressed, and transmitted according to the analysis process, thereby reducing the data to be transferred.

JP 2004-199377 A

  Conventionally, in order to increase the efficiency of data compression, information accuracy has to be lowered. When the original data is preprocessed to reduce the information accuracy of a specific data item, the information accuracy of the data item is uniformly reduced, and the use value of the data item is reduced when the data is used. There was a problem that.

  An object of the present invention is to provide a data compression technique that increases a data compression rate without degrading information accuracy.

  A data compression apparatus disclosed as one embodiment of the present invention includes: 1) a degree of coincidence and a data group calculated based on a matching weight given to an item when the item information matches between a plurality of data And the same level determination table in which conditions for grouping data into one group using the appearance frequency of data are set, and 2) a holding level indicating the information accuracy set for the item based on the distribution of information for each item And a storage unit that stores an information accuracy determination table in which a correspondence relationship with the minimum principal holding level indicating the converted information accuracy is defined, and 3) for each item of the data group to be processed, for each item Based on the setting information in which the matching weight and the holding level are set, a matching degree based on the matching weight with respect to other data in the data group is obtained, and data matching the conditions of the sameness determination table is obtained. 4) a distribution of information that appears in each of the items in each of the groups, and holding the distribution of the appearing information and the setting information based on the information accuracy determination table An information accuracy determining unit that determines a principal holding level of each item according to the level; and 5) according to a conversion rule for rewriting data information corresponding to the principal holding level, for each group, An event conversion unit that converts information of each item of data based on the principal holding level; and 6) a data compression unit that performs data compression on the data group including the converted data.

  A data compression program disclosed as another embodiment of the present invention is for causing a computer to execute a process that is realized by the apparatus.

  According to another aspect of the present invention, there is provided a data compression method in which a computer executes processing steps that are realized by the apparatus.

  According to the data compression apparatus described above, the data of the data group to be compressed is grouped according to the distribution of coincidence of the item information, and the data in which the information is rewritten according to the set information accuracy is compressed. Thus, the efficiency of data compression can be improved.

It is a figure which shows the structural example in the Example of the data compression apparatus disclosed as one form of this invention. It is a figure which shows the example of the event data group which a data compression apparatus acquires. It is a figure which shows the example of setting information. It is a figure which shows the example of an identification determination table. It is a figure which shows the example of an information precision determination table. It is a figure which shows the example of a conversion rule. It is a figure which shows the example of the group division | segmentation of event data. It is a figure which shows the example of rewriting of the information of event data. It is a figure which shows the example of rewriting of the information of event data. It is a figure which shows the example of the event data group by which the rewriting process of information was carried out. It is a figure which shows the data image at the time of compressing an event data group with the dictionary type compression method. It is a figure which shows the example of a processing flow of a data compression apparatus. It is a figure which shows the hardware structural example of a data compression apparatus.

  FIG. 1 is a diagram illustrating a configuration example in an embodiment of a data compression device disclosed as one embodiment of the present invention.

  The data compression apparatus 1 shown in FIG. 1 performs compression processing on a data group (event data group) including information corresponding to a plurality of items, and outputs compressed data 3.

  A data group to be processed by the data compression apparatus 1 is a set of event data 2 (event data group) that is generated for each event in a POS system (Point Of Sales system).

  The data compression apparatus 1 includes a storage unit 10, a group division unit 11, an information accuracy determination unit 13, an event conversion unit 15, and a data compression unit 17. Further, the data compression apparatus 1 may include a setting unit 19.

  The storage unit 10 stores an identification determination table 101, an information accuracy determination table 103, a conversion rule 105, and setting information 107.

  The identification determination table 101 uses the degree of coincidence calculated based on the matching weight assigned to an item when the information of the item matches among a plurality of data and the appearance frequency of the data in the data group. It is a data table in which conditions for grouping data into one group are set.

  Based on the unique number of information for each item, the information accuracy determination table 103 shows the correspondence between the holding level indicating the information accuracy set for the item and the principal holding level indicating the information accuracy of the converted information. It is a defined data table.

  The conversion rule 105 is information in which a conversion rule 105 for converting information corresponding to the principal holding level is set.

  The setting information 107 is information in which a matching weight and a holding level for each item of the event data 2 are set.

  Based on the setting information 107, the group dividing unit 11 obtains a distribution of matching weights with respect to other event data 2 for each event data 2 of the event data group to be processed, and satisfies the conditions of the sameness determination table 101 Data is collected into one group (event set).

  The information accuracy determination unit 13 obtains the distribution of information appearing in each item of the event data 2 in the group for each group collected by the group division unit 11, and distributes the information based on the information accuracy determination table 103. From the holding level of the setting information 107, for each item, a principal holding level indicating the accuracy of information held when converting item information is determined.

  The distribution of information is specified by, for example, a unique number of information.

  The event conversion unit 15 converts the information of each event data item in the group according to the conversion rule 105 based on the principal holding level. The event conversion unit 15 adds the event data 2 group (event set) whose information has been converted to the conversion event group, and among the input event data group, the event data remaining without being collected into the event set 2 is added to the conversion event group.

  The data compression unit 17 performs data compression on the conversion event group and generates compressed data 3.

  The setting unit 19 receives input of matching weight and holding level for each item of the event data 2, generates setting information 107, and stores it in the storage unit 10. Alternatively, the setting unit 19 receives the setting information 107 and stores it in the storage unit 10.

  FIG. 2 is a diagram illustrating an example of an event data group acquired by the data compression apparatus 1.

  Each row of the event data group shown in FIG.

  The event data 2 includes a plurality of items set in advance, each item of “year, month, day, hour, minute, second, customer ID, cash register number, cashier charge, product ID, color, size” in this embodiment. Contains information.

  FIG. 3 is a diagram illustrating an example of the setting information 107.

  In the setting information 107, “weight of matching” and “holding level” are set for each item of the event data 2 shown in FIG.

  The “matching weight” is a value given to the corresponding item of the matching event data 2 when the item information matches between the plurality of event data 2. The matching weight is a value indicating how much importance is placed on matching of item information when determining the degree of matching between the event data 2. In the example of the setting information 107 in FIG. 3, the matching weight takes a value from 0 to 1 in 0.1 units, and the closer the value is to 1, the more important the matching of item information is. .

  When grouping event data 2 into groups, it is important that information of items whose matching weights are close to 1 match. In the example of FIG. 3, importance is placed on the coincidence of information in the items “product ID” and “size”.

  The “holding level” is a value indicating a setting with which accuracy the information of the item of the event data 2 is held. In the example of the setting information 107 in FIG. 3, it is assumed that four levels of “complete, range, distribution, deleteable” are set.

  “Complete” indicates information accuracy for maintaining a plurality of pieces of information appearing in the item as they are. “Complete” is set for items that require individual information to be identified.

  “Range” indicates information accuracy in which a plurality of pieces of information appearing in an item are represented by a set of limit values of the information. “Range” is set for items that are recognized as useful if the range of information can be specified.

  “Distribution” indicates information accuracy that represents a plurality of pieces of information appearing in an item as a percentage of all pieces of information in each information group. “Distribution” is set for items that are recognized as useful if the distribution of information can be identified.

  “Can be deleted” is an information accuracy that indicates that information can be deleted when multiple items appearing in the item are dispersed and all information is less than a certain number. Show. When there is a variation in information, it is set for an item for which the information itself is not recognized as significant.

  FIG. 4 is a diagram illustrating an example of the sameness determination table 101.

  In the sameness determination table 101, “frequency” associated with each “matching degree” is set.

  The “matching degree” is a range of matching degrees based on matching weights among a plurality of event data 2. In this embodiment, the degree of coincidence is the ratio of the sum of the matching weights given to the items for which the information matches to the sum of the matching weights for all items. In the example of FIG. 4, the degree of coincidence is shown as a percentage. The degree of coincidence = 10% −29% represents a range where the degree of coincidence is 10% to 29%.

  “Frequency” is the number of appearances of event data 2 required when grouping event data 2 having a corresponding degree of matching (combining them into one event set).

  In the example shown in FIG. 4, when there are six or more event data 2 having a matching degree of 30% in the event data group, a process is performed in which the corresponding event data 2 is identified and grouped into one group. Is called.

  FIG. 5 is a diagram illustrating an example of the information accuracy determination table 103.

  In the information accuracy determination table 103, for each retention level that can be set for the item in the setting information 107, the principal retention level that is the information accuracy retained by the information conversion appears in the items of the event data 2 grouped together. It is set according to the unique number of information indicating the distribution of information to be performed.

  The principal holding level is information indicating which information accuracy of the information of each item of the group is to be converted, and “complete, distribution, range, deletion” is defined.

  “Complete” indicates information accuracy for maintaining a plurality of pieces of information appearing in the item as they are. “Range” indicates information accuracy in which a plurality of pieces of information appearing in an item are represented by a set of information limit values. “Distribution” indicates information accuracy that represents a plurality of pieces of information appearing in an item as a percentage of all pieces of information in each information group. “Delete” indicates that a plurality of pieces of information appearing in the item are deleted.

  In the information accuracy determination table 103 of FIG. 5, when the retention level is “complete”, “complete” is set as the principal retention level for any unique number.

  When the retention level is “range”, the principal retention level “complete” is when the unique number = 1, the principal retention level “distribution” when the unique number = 2, and the unique number = 3. Alternatively, the principal holding level “range” is set when the number is 4 or more.

  FIG. 6 is a diagram illustrating an example of the conversion rule 105.

  The conversion rule 105 indicates how to convert information corresponding to each principal holding level, that is, how information of each item of the event data 2 is based on the principal holding level determined for each group. Indicates the rules for converting to notation.

  In the conversion rule 105 of FIG. 6, for example, in the case of “complete”, the information appearing in the item is written as it is without being converted. In the case of “range”, the upper and lower limit values (for example, the maximum value and the minimum value) of the information appearing in the item are obtained and expressed as “maximum value−minimum value”. “Maximum value; Minimum value”. Further, in the case of “distribution”, the number of each information appearing in the item is obtained, the ratio (%) of the number of each information to the total number of information is calculated, and the information A, B, C is a%, b%, c %, It is expressed as “A−B−C = a% −b% −c%”. In addition, in the case of “delete”, the information appearing in the item is deleted.

  Next, the process of the data compression apparatus 1 will be described more specifically.

  It is assumed that the group division unit 11 acquires the event data group shown in FIG.

  The group division unit 11 refers to the setting information 107 when the information stored in the item for each event data 2 in the event data group matches the other event data 2, and sets the corresponding item. The matching weight is assigned, and the degree of matching is obtained by calculating the ratio obtained by dividing the sum of the matching weights given to the items by the sum of the matching weights set for all items.

  Further, the group division unit 11 refers to the sameness determination table 101 and groups together event data 2 having a high degree of coincidence and a high frequency (number of appearances) for the event data group shown in FIG. Let it be a set. When data having the same degree of coincidence appears in a certain number or more, the data is grouped by giving priority to the weight of coincidence with respect to the item with high priority.

  FIG. 7 is a diagram illustrating an example of group division of the event data 2.

  Three event data 2 shown in FIG. 7 indicate event data 2 extracted as an event set from the event data group shown in FIG.

  In the event data 2 of FIG. 7, among the items, “year (0.1), month (0.5), day (0.1), hour (0.5), cashier charge (0.1), product” The information of “ID (1), color (0.5), size (1)” matches, and the sum of matching weights = 3.8. Further, from the matching weight of all items of the event data 2 = 4.6, the matching degree = 83% is calculated.

  Furthermore, since the condition of frequency (≧ 3) corresponding to the corresponding degree of coincidence (70-100%) is satisfied based on the sameness determination table 101, three or more event data 2 shown in FIG. As a result, they are grouped into one event set.

  Next, the information accuracy determination unit 13 refers to the setting information 107 and the information accuracy determination table 103 to determine the information accuracy of each item of the event data 2 for the event set collected by the group division unit 11. That is, the information accuracy determination unit 13 counts the unique number of information for each item. Then, the information accuracy determination unit 13 refers to the information accuracy determination table 103 and determines the principal retention level based on the retention level set in the setting information 107 and the obtained unique number.

  More specifically, in the case of the event set shown in FIG. 7, “year” has the retention level = complete, and the information (values) are all the same (unique number = 1). Is set. Since “Month, Day” has a retention level = range and a unique number = 1, the principal retention level = complete is set. “Time” is set as “maintenance level = complete” because the retention level = deletable and the unique number = 1. “Minute and Second” is set such that the retention level = deletable and the unique number = 3, so that the principal retention level = range.

  “Customer ID” is set as “maintenance holding level = complete” because holding level = complete and unique number = 3. The “registration number” is set as “maintenance holding level = range” because holding level = range and unique number = 2. “Responsible for cashier” is set as “maintenance level = complete” because the retention level = deletable and the unique number = 1.

  “Product ID” has a retention level = complete and a unique number = 1, so that the principal retention level = complete. Since “color” has a retention level = distribution and a unique number = 1, the principal retention level = complete is set. Since “size” is retention level = distribution and unique number = 1, principal retention level = complete is set.

  Next, the event conversion unit 15 refers to the conversion rule 105, and based on the information accuracy (principal holding level) determined by the information accuracy determination unit 13, the event conversion unit 15 sets each item of the event data 2 collected in the event set. Rewrite information. That is, the event conversion unit 15 rewrites the information of the item of each event data 2 according to the conversion rule 105 corresponding to the principal holding level of the item.

  8 and 9 are diagrams showing examples of rewriting information of the event data 2. FIG.

  The three event data 2 shown in FIG. 8 shows the event data 2 of the event set shown in FIG.

  Here, with respect to the information (17, 24, 53) of “minutes”, the principal holding level = range is determined. The event conversion unit 15 obtains the minimum value and the maximum value from the information of each event data 2 in the event set, and the “minute” information of all event data 2 in the event set is rewritten as “17-53”. ing.

  Also, since the principal holding level = range is determined for the “second” information (01, 11, 24), similarly, the minimum value and the maximum value are obtained, and all event data 2 of the event set are determined. The information of “second” is rewritten as “01-24”.

  For other items, since the principal holding level is determined to be complete, the information of each event data 2 of the event set is maintained as it is.

  The seven event data 2 shown in FIG. 9A shows another event set extracted from the event data group shown in FIG.

  For the event set shown in FIG. 9A, the principal holding level is determined and the information of each item is rewritten, as in the process described with reference to FIG.

  In the event set shown in FIG. 9A, when the information accuracy determining unit 13 determines that the “color” principal holding level = distribution, the event converting unit 15 sets each information of “color” (white, The appearance ratio of (red, sky blue) is calculated and expressed according to the conversion rule 105. That is, as shown in FIG. 9B, the “color” information is rewritten as “white-red-sky blue = 43-29-29”.

  Similarly, the “size” information is also rewritten as a distribution notation “ML−LL = 29−43−29”.

  The information accuracy determination and the information rewriting process as described above are performed on all event sets made up of event data groups.

  The event conversion unit 15 adds an event set (converted event data 2) extracted from the event data group and rewritten to the converted event group for compression processing. Then, the event conversion unit 15 adds the event data 2 remaining without being collected into the event set to the conversion event group.

  FIG. 10 is a diagram illustrating an example of an event data group subjected to information rewrite processing.

  The event data group shown in FIG. 10 corresponds to the event data group shown in FIG.

  In the event sets G1, G2, and G3, it is shown that the item information is rewritten according to the corresponding principal holding level to be the same information.

  The event data 2 in the lowermost row of the event data group shown in FIG. 10 represents that the event data 2 remains without being grouped into an event set by the group dividing unit 11. In the event data group, the event data 2 that is not grouped because it does not meet the conditions of the sameness determination table 101 indicates that the item information remains as it is (information accuracy).

  From the event data group shown in FIG. 10 that is subject to data compression, statistical information about the item of interest is maintained at a set retention level (information accuracy) or higher, and is read as useful information for later analysis processing, etc. You can see that it is possible.

  More specifically, from the event data group shown in FIG. 10, as a knowledge of the sales trend, only “pink” color is sold at 10 o'clock on April 6, 2010 for the S size of a certain product. You can see that it is not. In addition, it can be seen that the “M” size products are sold in the “pink” period from April 3 to 14, 2010 at 10 o'clock.

  Furthermore, other size and color combinations are retained in the statistical information and the information accuracy is low, but the information accuracy of the set “distribution” is maintained.

  On the other hand, event data 2 including unique information that is deleted or buried in other information as in the case of the April 15 record is also maintained with the same information accuracy. ing.

  The data compression unit 17 performs data compression on the conversion event group collected by the event conversion unit 15 to generate compressed data 3. The data compression unit 17 performs processing using a known data compression method. In this example, lexicographic compression processing and entropy compression processing are used. Since these compression processes are known, description thereof is omitted.

  FIG. 11 is a diagram showing a data image when the event data group shown in FIG. 10 is compressed by the lexicographic compression method.

  As shown in FIG. 11, since the information of each item is rewritten to the same notation for each event set, it can be seen that the data is compressed at a higher compression rate.

  FIG. 12 is a diagram illustrating a processing flow example of the data compression apparatus 1.

  Step S10: The setting unit 19 of the data compression apparatus 1 receives the input of matching weight and holding level for each item of the event data 2 via the input device (not shown in FIG. 1), and generates the setting information 107. , Stored in the storage unit 10. Alternatively, the setting unit 19 acquires the setting information 107 and stores it in the storage unit 10.

  Step S11: The group dividing unit 11 acquires the data group of the event data 2, and for each event data 2, based on the setting information 107, items having the same information with each other event data 2 are displayed. If there is, the matching weight set for the item is assigned. Further, the group dividing unit 11 calculates the sum of the matching weights, and calculates the degree of matching (%) indicating the ratio to the matching weights set for all items. The group dividing unit 11 groups a group of event data 2 having a high degree of coincidence and a high frequency based on the sameness determination table 101, and takes out as a set of events.

  Step S12: The information accuracy determination unit 13 obtains the distribution of information for each item of the event data 2 for the extracted event set, and based on the holding level set for the item in the setting information 107 and the distribution of information. Next, determine the principal retention level for each item.

  Step S13: The event conversion unit 15 converts the information of each item of all event data 2 of the event set into a notation representing the distribution of the corresponding information according to the determined principal holding level and the conversion rule 105, Event data 2 of the converted event set is added to a conversion event group for compression processing.

  Step S14: The group dividing unit 11 checks whether there is a group of event data 2 having a high degree of matching and a high frequency in the remaining event data group of the acquired event data 2. If there is a group of event data 2 having a high degree of coincidence and a high frequency (Y in step S14), the process returns to the process in step S11. If there is no group of event data 2 having a high degree of coincidence and a high frequency (N in step S14), the process proceeds to step S15.

  Step S15: The data compressing unit 17 summarizes the event data 2 that is the remaining event data group of the acquired event data 2 and the event data 2 of the conversion event group, into the event data group of the combined event data 2 On the other hand, data compression processing is performed and compressed data 3 is output.

  The data compression apparatus 1 is realized by a computer system including hardware having a CPU and a memory and a software program, or dedicated hardware.

  FIG. 13 is a diagram illustrating a hardware configuration example of the data compression apparatus 1.

  The data compression apparatus 1 includes a computing device (CPU) 101, a temporary storage device (DRAM, flash memory, etc.) 102, a computer 100 having a permanent storage device (HDD, flash memory, etc.) 103, and an input device (keyboard, mouse, etc.). ) 120 and an output device (display, printer, etc.) 130.

  The data compression apparatus 1 can be implemented by a program that can be executed by the computer 100. In this case, a program describing the processing contents of the functions that the data compression apparatus 1 should have is provided. When the computer 100 executes the provided program, the processing functions of the data compression apparatus 1 described above are realized on the computer 100.

  The computer 100 can also read a program directly from a portable recording medium and execute processing according to the program.

  As shown in the present embodiment, according to the data compression apparatus 1, by setting a holding level that is information accuracy for each item in the setting information 107, information on items related to information to be used is minimized. The data compression can be performed while maintaining the information with higher information accuracy according to the distribution of information.

  That is, when the data compression apparatus 1 is applied to the data compression process of the event data 2, the following effects can be obtained.

  (1) Since data with a high compression rate is collected and grouped for each item included in data to be compressed, the compression efficiency can be improved without significantly reducing the information accuracy depending on the data.

  (2) Since data including special information is compressed without being grouped, information accuracy can be maintained high.

  In the present embodiment, the case where the data compression apparatus 1 is applied to the data compression of event data has been described. However, the scope of application of the data compression apparatus 1 is not limited to this, and the scope of the gist of the description Of course, various modifications are possible.

DESCRIPTION OF SYMBOLS 1 Data compression apparatus 10 Memory | storage part 11 Group division | segmentation part 13 Information accuracy determination part 15 Event conversion part 17 Data compression part 19 Setting part 101 Homogeneity determination table 103 Information accuracy determination table 105 Conversion rule 107 Setting information 2 Event data 3 Compression data

Claims (5)

A data compression apparatus for compressing a data group having multiple items of information,
A condition for grouping data into a group using the degree of coincidence calculated based on the matching weight given to the item and the appearance frequency of the data in the data group when the item information matches between a plurality of data An identification determination table in which is set,
Based on the distribution of information for each item, an information accuracy determination table in which a correspondence relationship between a holding level indicating the information accuracy set for the item and a minimum principal holding level indicating the information accuracy after conversion is defined. A storage unit for storing;
For each data in the data group to be processed, based on the setting information in which the matching weight and the holding level are set for each item, the degree of matching based on the matching weight for the other data in the data group is obtained, A group division unit for grouping data satisfying the conditions of the same identification determination table;
In each of the groups, a distribution of information appearing in each of the items is obtained, and based on the information accuracy determination table, a principal of each of the items according to the distribution of the appearing information and the holding level of the setting information An information accuracy determination unit for determining a retention level;
In accordance with a conversion rule for rewriting data information corresponding to the principal holding level, for each group, an event converter that converts information of each item of data in the group based on the principal holding level;
A data compression apparatus comprising: a data compression unit that performs data compression on the data group including the converted data. The principal holding level includes any two or more levels of complete, range, distribution, or deletion. The complete indicates information accuracy for maintaining information appearing in each item as it is, and the range includes each item. Indicates the information accuracy for converting the information appearing in the information into the maximum value and the minimum value of the information, and the distribution indicates the information accuracy for converting the information appearing in each item into the ratio of the information in the group, The data compression apparatus according to claim 1, wherein "deletable" indicates information accuracy for deleting information appearing in each item. The information processing apparatus includes: a setting unit that acquires the setting information and stores the setting information in the storage unit; or receives an input related to the matching weight and the holding level to generate setting information and store the setting information in the storage unit. The data compression apparatus according to claim 1 or 2. A data compression program for causing a computer to execute compression processing of a data group having multiple items of information,
When the computer matches the item information among a plurality of pieces of data, the computer uses the matching degree calculated based on the matching weight assigned to the item and the appearance frequency of the data in the data group to set the data to 1 The identification table with the conditions to be grouped together,
Based on the distribution of information for each item, an information accuracy determination table in which a correspondence relationship between a holding level indicating the information accuracy set for the item and a minimum principal holding level indicating the information accuracy after conversion is defined. It has a storage unit to store,
In the computer,
For each data in the data group to be processed, based on the setting information in which the matching weight and the holding level are set for each item, the degree of matching based on the matching weight for the other data in the data group is obtained, A process of grouping data satisfying the conditions of the sameness determination table;
In each of the groups, a distribution of information appearing in each of the items is obtained, and based on the information accuracy determination table, a principal of each of the items according to the distribution of the appearing information and the holding level of the setting information Processing to determine the retention level;
In accordance with a conversion rule for rewriting data information corresponding to the principal holding level, for each group, processing for converting information of each item of data in the group based on the principal holding level;
A process of performing data compression on the data group including the converted data;
Data compression program to be executed. A data compression method in which a computer compresses a data group having multiple items of information,
The computer is
A condition for grouping data into a group using the degree of coincidence calculated based on the matching weight given to the item and the appearance frequency of the data in the data group when the item information matches between a plurality of data An identification determination table in which is set,
Based on the distribution of information for each item, an information accuracy determination table in which a correspondence relationship between a holding level indicating the information accuracy set for the item and a minimum principal holding level indicating the information accuracy after conversion is defined. It has a storage unit to store,
For each data in the data group to be processed, based on the setting information in which the matching weight and the holding level are set for each item, the degree of matching based on the matching weight for the other data in the data group is obtained, A processing step for grouping data satisfying the conditions of the identification table,
In each of the groups, a distribution of information appearing in each of the items is obtained, and based on the information accuracy determination table, a principal of each of the items according to the distribution of the appearing information and the holding level of the setting information Processing steps to determine the retention level;
In accordance with a conversion rule for rewriting data information corresponding to the principal holding level, for each group, a process step for converting information of each item of data in the group based on the principal holding level;
A data compression method comprising: performing a data compression on the data group including the converted data.

Images