KR20220060401A - Method and apparatus for data reduction based on locality - Google Patents

Abstract

According to one embodiment of the present invention, a method for reducing data based on locality comprises: a step of acquiring a plurality of pieces of input data configured in a time sequential manner; a step of storing, for the plurality of pieces of input data, individual input data when the individual input data is compared with input data previous to the individual input data and the individual input data is changed; and a step of storing change information, which is information on whether the individual input data is changed when the individual input data is compared with input data previous to the individual input data.

Description

Method and device for data reduction based on locality

The present invention relates to a method and apparatus for efficiently condensing data in consideration of locality when storing data having locality.

With the recent advent of the 4th industrial revolution, IoT technology is being applied in many places. From smart home services that integrate home appliances such as robot vacuum cleaners and air conditioners used at home, to connected cars called the future of autonomous driving, and smart factories for facility management used in the production process, IoT devices in many places are It provides users with various convenient functions based on

To provide these services, IoT devices produce and store data in real time. In addition, various types of databases are used for efficient data management. Representatively, there are relational databases such as MySQL, and key-value based databases such as Redis and Cassandra. The main role of these databases is to organize and maintain a set of data to efficiently process data by avoiding duplication of data shared and used by several people and unifying it.

On the other hand, most IoT devices have very small hardware resources, unlike other general-purpose computer systems, so it is very important to use them efficiently. Nevertheless, existing databases waste storage space because they do not consider the characteristics of data with high locality.

Accordingly, there is a need for a method for reducing data based on locality and an apparatus for storing and managing data more efficiently in consideration of locality of data.

In order to solve the above-mentioned problem, the present invention provides a data reduction method and apparatus capable of more efficiently storing and managing data in consideration of locality of data.

The problem to be solved by the present invention is not limited to the problem(s) mentioned above, and another problem(s) not mentioned will be clearly understood by those skilled in the art from the following description.

A data reduction method based on locality according to an embodiment of the present invention includes: acquiring a plurality of input data configured in time series; comparing the individual input data with the input data immediately preceding the individual input data with respect to the plurality of input data and storing the individual input data when changed; and storing change information, which is information about whether the individual input data is changed by comparing the individual input data with the input data immediately before the individual input data, for each of the plurality of input data.

Preferably, the acquiring of the plurality of input data may include sequentially receiving and acquiring individual input data constituting the plurality of input data, or acquiring the plurality of input data at once.

Preferably, the input data includes a predetermined data value and information about a time corresponding to the data value, and the storing of the change information may further store time information, which is information about the time.

Preferably, the storing of the change information may include storing the change information in a bitmap indicating a bit value corresponding to each of a change and a no change.

In addition, according to an embodiment of the present invention, an apparatus for reducing data based on locality includes: an acquisition unit configured to acquire a plurality of input data configured in time series; and when the plurality of input data is changed by comparing the individual input data with the input data immediately preceding the individual input data, the individual input data is stored, and for each of the plurality of input data, the individual input data and and a storage unit that compares the input data immediately before the individual input data and stores change information, which is information about whether the input data has been changed.

Preferably, the acquiring unit may sequentially receive and acquire individual input data constituting the plurality of input data, or acquire the plurality of input data at once.

Preferably, the input data includes a predetermined data value and information about a time corresponding to the data value, and the storage unit may further store time information, which is information about time, together with the change information.

Preferably, the storage unit may store the change information in a bitmap indicating a bit value corresponding to each of change and no change.

According to an embodiment of the present invention, the size of the stored data can be reduced by abbreviated and stored in consideration of the locality of the acquired data, and a range-query for a time value of the stored data is also performed. It has the effect of being able to do it faster.

1 is a flowchart illustrating a data reduction method based on locality according to an embodiment of the present invention.
2 is a block diagram illustrating an apparatus for reducing data based on locality according to an embodiment of the present invention.
3 is a diagram for explaining a data storage capacity according to an embodiment of the related art.
4 is a diagram for explaining a data storage capacity according to an embodiment of the present invention.
5 and 6 are diagrams for explaining the effect of a data reduction method based on locality according to an embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals have been used for like elements.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it is understood that other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart illustrating a data reduction method based on locality according to an embodiment of the present invention.

In step S110, the data reduction device acquires a plurality of input data configured in time series.

For example, the data reduction device may be directly mounted on an IoT device used in a smart home service, a connected car, a smart factory, etc., or may be connected to the IoT device by wire or wirelessly to obtain input data. Alternatively, the data reduction device may be connected to various types of sensor devices by wire or wirelessly to obtain input data.

In this case, the plurality of input data may be configured according to the order of time, and may be obtained according to a predetermined time interval determined in advance, or may be obtained together with information about the obtained time.

In another embodiment, the data reduction apparatus may sequentially receive and acquire individual input data constituting a plurality of input data, or may acquire a plurality of input data at once.

That is, the data reduction device may acquire individual input data constituting a plurality of input data from the IoT device one by one in real time. For example, the data reduction device may continuously receive and acquire speed-related data from the connected car at 1-second intervals.

Also, the data reduction device may receive and acquire a plurality of input data corresponding to a predetermined time period from the IoT device at once. For example, the data reduction apparatus may acquire data obtained by measuring the temperature at predetermined time intervals from the temperature sensor in units of one day.

In step S120, the data reduction device compares the individual input data with the input data immediately preceding the individual input data with respect to the plurality of input data and, when changed, stores the individual input data.

Here, referring to FIGS. 3 and 4 , it can be seen that 64 pieces of speed data of the connected car are arranged in chronological order. At this time, the speed data of the connected car is moving between 86km/h and 94km/h, and when the same speed value as the speed value measured immediately before is continuously measured, it is confirmed that the data overlap occurs frequently in 43 times. can

At this time, assuming that the data reduction device uses the conventional technology, 8*64= It requires 512 bytes of storage space.

However, the data reduction apparatus can store the individual input data only when the individual input data included in the plurality of input data is changed by comparing the individual input data with the previous input data. At this time, since the data reduction device stores only 21 pieces of speed data indicated by circles in FIG. 4 , a storage space of 8*21=168 bytes is required.

For example, referring to FIGS. 3 and 4 , the velocity data is {89, 89, 89, 89, 90, 90, 90, 92, 92, ... }, the data reduction device compares the previous speed data to {89, 90, 92, . } can only be stored. In this case, the data reduction device may consider that '89', which is the first speed data, has been changed compared with the previous speed data, and may store it.

Finally, in step S130, for each of the plurality of input data, the data reduction device compares the individual input data with the input data immediately preceding the individual input data, and stores change information, which is information about whether the data has been changed.

That is, the data reduction apparatus may store change information, which is information about a result of the apparatus comparing individual input data with input data immediately preceding the individual input data with respect to all input data included in the plurality of input data.

For example, the data reduction apparatus may store 0 as change information if the individual input data and the input data immediately preceding the individual input data are the same, and 1 if they are different. In the description of step S120 above, the data reduction device indicates that the speed data is {89, 89, 89, 89, 90, 90, 90, 92, 92, ... }, {89, 90, 92, . } can be stored. At this time, the data reduction device is the speed data {89, 89, 89, 89, 90, 90, 90, 92, 92, ... } Change information for each {1, 0, 0, 0, 1, 0, 0, 1, 0, … } can be saved.

That is, the data reduction device is the {89, 90, 92, ... } and change information {1, 0, 0, 0, 1, 0, 0, 1, 0, … }, the speed data of the connected car can be abbreviated and stored.

Conversely, in order to restore the abbreviated data, {89, 90, 92, ... }, and change information {1, 0, 0, 0, 1, 0, 0, 1, 0, … } to add the unchanged speed data with the same value as the previous speed data using {89, 89, 89, 89, 90, 90, 90, 92, 92, ... } can be obtained.

For example, {89, 90, 92, . } for change information {1, 0, 0, 0, 1, 0, 0, 1, 0, … }, list {89, 90, 92}, and since there are 3 0s of change information between 89 and 90, add 3 89s, the same value as the previous speed data, and change between 90 and 92 Since there are 2 0s in the information, two 90s, the same value as the previous speed data, are added, and since there is 1 0 of the changed information after 92, 1 92, the same value as the previous speed data, is added, and the original speed data In {89, 89, 89, 89, 90, 90, 90, 92, 92, … } can be restored.

In another embodiment, the input data includes information about a predetermined data value and a time corresponding to the data value, and the data reduction apparatus may further store time information, which is information about time, when storing the change information.

In the example described above, it was assumed that the time interval for acquiring data was uniform. By additionally storing time information when storing , data reduction can be performed without a problem even when the time interval at which data is acquired is not uniform.

In another embodiment, the data reduction device may store the change information in a bitmap indicating a bit value corresponding to each of change and no change.

That is, when the data reduction device assumes change to 1 and no change to 0, change information {1, 0, 0, 0, 1, 0, 0, 1, 0, ... } can be reconstructed in bits and stored in a bitmap. For example, the data reduction device includes the change information {1, 0, 0, 0, 1, 0, 0, 1, 0, ... } as a bitmap to represent 9 bits of 100010010.

Meanwhile, referring to FIG. 4 , since the data reduction device stores only 21 pieces of speed data, the point requiring a storage space of 8*21=168 bytes has been described above. At this time, since the data reduction device needs to store change information for 64 speed data in 64 bits (ie, 8 bytes), a total storage space of 168+8=176 bytes is required. As a result, the data reduction device can save storage space as much as 512-176=336 bytes through application of the present invention.

Meanwhile, referring to FIG. 5, when the size of the time value in data is designated as 4 bytes and the size of the speed value is increased from 1 byte to 8 bytes and stored, the conventional technique (Traditional) and the present invention (Bitmap Encoding Scheme, BES) storage This is a graph comparing space usage. As expected, the space saved increases as the size of the data pair increases, and in the case of the conventional technology, it increases by 20% to 50%. Although necessary, in the case of the present invention, it increased by 18% to 47% and increased by a total of 1.3 times, showing a capacity saving effect of about 64 to 66%.

Conversely, however, there is a case in which a waste of storage space occurs in the present invention compared to the prior art. This is a case in which data duplication does not occur frequently. In the worst case, even if no duplication occurs at all, the wasted capacity is very small at 1 bit per piece of data.

Also, referring to FIG. 6 , the rate of occurrence of data duplication for saving space in the present invention (BES) for each size of the data pair compared to the conventional technique is shown for each size of the data pair. As the size of a data pair increases, the space saved by one duplication increases, so it was confirmed that the ratio was significantly lowered from 12% when the size was 1 byte to 1.5% when the size was 8 bytes. The redundancy rate of the entire data was much higher at 24%, and considering that the duplicate rate of the section having the worst number of duplicates in 200 consecutive pieces of data is also 9%, the present invention (BES) causes wastage of storage space. It is considered that there are not many cases.

Furthermore, the present invention (BES) can perform range-query for a time interval faster than the prior art. In the case of the prior art, all 64 data pairs must be read in order to read the speed values corresponding to 64 sections of FIG. 3 or 4 when all data for the whole are stored, whereas the present invention (BES) uses 21 data pairs. After reading the data pair, the speed value of the empty time period is filled with the previous value, so that the number of disk accesses during range-query can also be reduced.

2 is a block diagram illustrating an apparatus for reducing data based on locality according to an embodiment of the present invention.

2 is a data reduction apparatus 200 based on locality according to an embodiment of the present invention includes an acquisition unit 210 and a storage unit 220 .

On the other hand, the data reduction device 200 based on locality according to an embodiment of the present invention is mounted on an IoT device, a connected car, a sensor device, a smart phone, a tablet, a computer, a laptop computer, etc. or is connected through a wired or wireless network. It can be used for data reduction.

The acquisition unit 210 acquires a plurality of input data configured in time series.

In another embodiment, the acquiring unit 210 may sequentially receive and acquire individual input data constituting the plurality of input data, or acquire a plurality of input data at once.

The storage unit 220 compares the individual input data with the input data immediately before the individual input data with respect to the plurality of input data and stores the individual input data, and stores the individual input data for each of the plurality of input data. By comparing the input data with the input data immediately preceding the individual input data, change information, which is information about whether the input data has been changed, is stored.

In another embodiment, the input data includes information about a predetermined data value and time corresponding to the data value, and the storage unit 220 may further store time information, which is information about time, together with change information.

In another embodiment, the storage unit 220 may store the change information in a bitmap indicating a bit value corresponding to each of change and no change.

Although specific embodiments according to the present invention have been described so far, various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by the claims described below as well as the claims and equivalents.

As described above, although the present invention has been described with reference to the limited examples and drawings, the present invention is not limited to the above-described examples, which are various modifications and variations from these descriptions by those skilled in the art to which the present invention belongs. Transformation is possible. Accordingly, the spirit of the present invention should be understood only by the claims described below, and all equivalents or equivalent modifications thereof will fall within the scope of the spirit of the present invention.

Claims (8)

acquiring a plurality of input data configured in time series;
comparing the individual input data with the input data immediately preceding the individual input data with respect to the plurality of input data and storing the individual input data when changed; and
storing change information, which is information about whether the individual input data is changed by comparing individual input data with input data immediately preceding the individual input data, for each of the plurality of input data;
A data reduction method based on locality, comprising: According to claim 1,
The step of acquiring the plurality of input data includes:
A data reduction method based on locality, characterized in that sequentially receiving and acquiring individual input data constituting the plurality of input data, or acquiring the plurality of input data at once. According to claim 1,
The input data includes information about a predetermined data value and a time corresponding to the data value,
The step of storing the changed information is
A data reduction method based on locality, characterized in that further storing time information, which is information about time. According to claim 1,
The step of storing the changed information is
The data reduction method based on locality, characterized in that the change information is stored in a bitmap representing a bit value corresponding to each change and no change. an acquisition unit acquiring a plurality of input data configured in time series; and
With respect to the plurality of input data, when the individual input data is compared with the input data immediately preceding the individual input data and changed, the individual input data is stored;
a storage unit for storing change information, which is information about whether the individual input data is changed by comparing the input data immediately before the individual input data with respect to each of the plurality of input data;
Data reduction device based on locality, characterized in that it comprises a. 6. The method of claim 5,
the acquisition unit
The apparatus for reducing data based on locality, characterized in that sequentially receiving and acquiring individual input data constituting the plurality of input data, or acquiring the plurality of input data at once. 6. The method of claim 5,
The input data includes information about a predetermined data value and a time corresponding to the data value,
the storage unit
The apparatus for reducing data based on locality, characterized in that it further stores time information, which is information about time, together with the change information. 6. The method of claim 5,
the storage unit
and storing the change information in a bitmap representing a bit value corresponding to each of change and no change.

Images