KR20240040922A - Method, apparatus, system and computer program for data compaction in storage with enhanced processing capability - Google Patents

Abstract

The present invention relates to a security-enhanced image storage management method, device, system, and computer. The present invention relates to a data compaction method, device, system, and computer program, and more specifically, to storage that is stored in a manner that does not allow modification. ) relates to a data compaction method, device, system, and computer program that can effectively process data compaction.
In the present invention, in a data compaction system, a storage step of storing a first base file containing original values of data and a first delta file containing changed values of the data in a storage unit; Based on the request for the data, performing merge processing on the original value of the data and the changed value of the data and performing merge processing corresponding to the request; And a compaction processing step of generating a second base file in which the first base file and the first delta file are merged based on whether the merge processing is performed and replacing the first base file in the storage unit. Disclosed is a data compaction method characterized in that:

Description

Method, apparatus, system and computer program for data compaction in storage with enhanced processing capability}

The present invention relates to a data compaction method, device, system, and computer program, and more specifically, to a data compaction method capable of effectively processing data compaction in storage stored in a manner that does not allow modification. Relates to data compaction methods, devices, systems, and computer programs.

Recently, technologies for efficiently storing and managing large amounts of data have been widely used in cloud systems and database systems.

In this regard, data changes are stored in a separate delta file without allowing changes to the base file where the original data is stored, such as block storage or object storage. Various attempts are being made to secure data stability through management methods and further secure efficient processing performance in distributed environments.

More specifically, data compaction refers to combining delta files for changed values into one in a system where changes to the base file where the original data is stored are not allowed, as described above. This refers to the operation of merging with a base file.

However, in the past, data compaction was usually performed in batches at predetermined intervals. In this case, a large amount of data was processed at once, concentrating computing resources. This could cause an overload on the system, and if the user searches data before the batch data compaction task is performed, the merge process for the base file and delta file is not only performed repeatedly. In addition, in the batch data compaction task, the merge process for the base file and delta file was performed redundantly, resulting in a waste of computing resources.

In addition, conventionally, in order to perform the above data compaction task, the administrator must set the cycle of the data compaction task based on his or her experience, and furthermore, data compaction for the entire file. Priority had to be determined, such as whether to perform the operation or only some files. At this time, problems may occur in which data compaction cannot be performed with optimal settings depending on the administrator's experience level or the operating environment of the system. I was able to.

Accordingly, overload of the system and repetitive merge processing that may occur during data compaction in storage stored in a way that does not allow changes in cloud systems or database systems, etc. There is a continuous need for methods that can effectively suppress unnecessary consumption of resources and further effectively improve problems such as the environment in which data compaction is performed depending on the manager's experience level.

Republic of Korea Patent Publication No. 10-2020-0104856 (published on September 4, 2020)

The present invention was created to solve the problems of the prior art as described above, which may appear in the process of data compaction in storage stored in a manner that does not allow changes in cloud systems or database systems. The purpose is to provide data compaction methods, devices, systems, and computer programs that can effectively suppress unnecessary consumption of computing resources due to overload of existing systems and repetitive merge processing.

Additionally, the purpose of the present invention is to provide a data compaction method, device, system, and computer program that can effectively improve problems such as changes in the data compaction performance environment depending on the manager's experience level, etc.

The data compaction method according to one aspect of the present invention for solving the above problem is, in a data compaction system, a first base file containing original values of data and a first delta file containing changed values of the data are stored in a storage unit. storage storage step; A merge processing step of performing merge processing on the original value of the data and the changed value of the data based on the request for the data; And a compaction processing step of generating a second base file in which the first base file and the first delta file are merged based on whether the merge processing is performed and replacing the first base file in the storage unit. It is characterized by:

Here, in the storage unit, changes to data included in the first base file may not be permitted.

In addition, in the merge processing step, the first base file and the first delta file are loaded into memory to perform merge processing, and in the compaction processing step, merge information for the merge processing performed in the memory is collected and collected. Based on this, the storage unit may generate the second base file to replace the first base file.

At this time, in the compaction processing step, the merge information may be collected from the memory and stored in the cache storage device.

In addition, in the compaction processing step, the second base file is stored in the storage unit based on the merge information stored in the cache storage device using a separate application that runs independently of the application that runs the merge processing step. can be created.

Furthermore, in the compaction processing step, the first base file may be deleted from the storage unit and replaced with the second base file.

Additionally, in the compaction processing step, the merged information can be stored in the cache storage device only when the size of the merged information is larger than a predetermined standard value.

Additionally, in the compaction processing step, the compaction operation on the second base file can be performed again only when a predetermined reference time has elapsed since the first base file was replaced with the second base file. there is.

Furthermore, in the compaction processing step, the merged information accumulated in the cache storage device during the predetermined reference time may be deleted, leaving only the final merged information.

In addition, in the compaction processing step, the compaction completion list for which a specific predetermined time has elapsed can be automatically deleted from the meta database.

Additionally, a computer program according to another aspect of the present invention is stored in a computer-readable medium for executing each step of the data compaction method described above on a computer. It could be a computer program.

In addition, a data compaction system according to another aspect of the present invention includes a storage unit that stores a first base file containing original values of data and a first delta file containing changed values of the data; a merge processing unit that performs merge processing on the original value of the data and the changed value of the data based on the request for the data; And a compaction processing unit that generates a second base file in which the first base file and the first delta file are merged based on whether the merge processing is performed and replaces the first base file in the storage unit. It is characterized by

Here, in the storage unit, changes to data included in the first base file may not be permitted.

In addition, the merge processing unit loads the first base file and the first delta file into memory and performs merge processing, and the compaction processing unit collects merge information for the merge processing performed in the memory and based on this, The storage unit may generate the second base file to replace the first base file.

At this time, the compaction processing unit may collect the merge information from the memory and store it in a cache storage device.

Furthermore, the compaction processing unit may generate the second base file in the storage unit based on the merge information stored in the cache storage device using a separate application that runs independently of the merge processing unit. .

Additionally, the compaction processing unit may delete the first base file from the storage unit and replace it with the second base file.

In the data compaction method, device, system, and computer program according to an embodiment of the present invention, the data compaction process in storage is stored in a manner that does not allow changes in a cloud system, database system, etc. It is possible to effectively suppress unnecessary consumption of computing resources due to system overload and repetitive merge processing that may occur in the system.

Additionally, the data compaction method, device, system, and computer program according to an embodiment of the present invention can effectively improve problems such as changes in the data compaction performance environment depending on the manager's experience level, etc.

The accompanying drawings, which are included as part of the detailed description to aid understanding of the present invention, provide embodiments of the present invention and explain the technical idea of the present invention together with the detailed description.
1 is a diagram illustrating an operating environment of a data storage system according to an embodiment of the present invention.
Figure 2 is a diagram illustrating a flow chart of a data compaction method according to an embodiment of the present invention.
3 to 9 are diagrams illustrating specific operations of the data compaction method according to an embodiment of the present invention.
Figure 10 is a diagram illustrating the operation of a data compaction method according to another embodiment of the present invention.
Figure 11 is a diagram illustrating a block diagram of a data compaction system according to another embodiment of the present invention.
Figure 12 is a diagram illustrating a specific configuration of a data compaction system according to another embodiment of the present invention.

Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. The objectives, specific advantages and novel features of the present invention will become more apparent from the following detailed description and preferred embodiments taken in conjunction with the accompanying drawings.

Prior to this, the terms and words used in this specification and claims should be interpreted as meanings and concepts consistent with the technical idea of the present invention as those defined by the inventor appropriately to explain the invention in the best way. , is only for illustrating embodiments and should not be construed as limiting the present invention.

In assigning reference codes to components, identical or similar components will be assigned the same reference codes regardless of the reference codes, and duplicate descriptions thereof will be omitted. The suffixes “module” and “part” for components used in the following description are given or used interchangeably in consideration of the ease of preparing the specification, and do not have distinct meanings or roles in and of themselves, and do not constitute software or hardware components. It can mean an element.

In describing the elements of the present invention, if an element is expressed in singular form, it should be understood that the element also includes plural forms unless otherwise specified. Additionally, terms such as “first”, “second”, etc. are used to distinguish one component from another component, and the components are not limited by these terms. Additionally, when a component is connected to another component, it means that another component may be connected between the component and the other component.

Additionally, in describing the embodiments disclosed in this specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed descriptions will be omitted. In addition, the attached drawings are only for easy understanding of the embodiments disclosed in this specification, and the technical idea disclosed in this specification is not limited by the attached drawings, and all changes included in the spirit and technical scope of the present invention are not limited. , should be understood to include equivalents or substitutes.

Hereinafter, exemplary embodiments of a data compaction method, apparatus, system, and computer program with improved processing performance according to the present invention will be described in detail with reference to the accompanying drawings.

First, Figure 1 illustrates the configuration of a data storage system 100 to which the table compaction system 120 according to an embodiment of the present invention can be applied. As can be seen in FIG. 1, the data storage system 100 according to an embodiment of the present invention includes a storage unit 110 in which data is stored and compaction of the data stored in the storage unit 110. It may be configured to include a data compaction system 120 that performs the task.

Here, as can be seen in FIG. 2, the data compaction method according to an embodiment of the present invention uses a first base file containing the original value of the data and the changed value of the data in the data compaction system 120. A storage storage step (S110) of storing the first delta file including the first delta file in the storage unit 110, performing a merge process of merging the original value of the data and the changed value of the data based on the request for the data. Based on step S120 and whether the merge process is performed, a second base file in which the first base file and the first delta file are merged is generated to replace the first base file in the storage unit 110. It may include a compaction processing step (S130).

Here, in the storage unit 110, changes to data included in the first base file may not be permitted.

In addition, in the merge processing step (S120), the first base file and the first delta file are loaded into memory and merge processing is performed on the original value of the data and the changed value of the data, and the compaction processing step is performed. In (S130), merge information about the merge process performed in the memory is collected, and based on this, the second base file is generated in the storage unit to replace the first base file.

At this time, in the compaction processing step (S130), the merge information may be collected from the memory and stored in the cache storage device.

Additionally, in the compaction processing step (S130), the merge information stored in the cache storage device using a separate application that runs independently of the application that runs the merge processing step (S120). Based on this, the second base file can be created in the storage unit 110.

Furthermore, in the compaction processing step (S130), the first base file may be deleted from the storage unit 110 and replaced with the second base file.

The method shown in FIG. 2 may be performed, for example, by a data compaction system 120, and further, the data compaction system 120 may include a computing device described below with reference to FIGS. 12 and 12 . It can be implemented including: For example, the data compaction system 120 includes a processor 10, and the processor 10 can correct table coordinate information by executing instructions configured to implement an operation for correcting table coordinate information.

Accordingly, in the data compaction method, device, and system according to an embodiment of the present invention, the data of the system that may appear during data compaction in storage stored in a manner in which changes are not permitted It is possible to prevent unnecessary consumption of computing resources due to overload and repetitive merge processing, and furthermore, it is possible to effectively improve the problem that the data compaction performance environment varies depending on the administrator's experience level, etc.

Hereinafter, the data compaction method, device, and system according to an embodiment of the present invention will be examined in more detail with reference to FIGS. 1 and 2.

First, in the storage storage step (S110), the data compaction system 120 stores the first base file containing the original value of the data and the first delta file containing the changed value of the data in the storage unit 110. I do it.

At this time, the storage unit 110 may be a storage device that stores data, for example, a cloud storage device that stores files of multiple users, or a database device that stores various data. However, the present invention does not necessarily apply to It is not limited to this, and various storage devices capable of storing data may be used.

More specifically, the storage unit 110 does not allow changes to the base file where the original data is stored, such as block storage, object storage, etc., and separate changes to the data. It may be configured to include a device for storing delta files.

Additionally, the data compaction system 120 performs a compaction operation on data stored in the storage unit 110.

At this time, data compaction refers to merging delta files for changed values into one in a system that does not allow changes to the base file where the original data is stored, as described above, or to the base file. This refers to the operation of merging with (base file).

For example, referring to FIG. 3, first, a first base file (DataFile1) containing 5 pieces of data is created in the storage unit 110 ((a) of FIG. 3).

Subsequently, when it is desired to add data number 6 to the first base file (DataFile1), the first base file (DataFile1) cannot be changed, and a separate second base file (DataFile2) is created (see Figure 3 (b)).

In addition, if you want to modify data number 5 of the first base file (DataFile1), the first base file (DataFile1) cannot be changed, and a separate first delta file (DataFile3) must be added (Figure 3) (c)).

Furthermore, if you want to delete data number 4 of the first base file (DataFile1), the first base file (DataFile1) cannot be changed, and a separate second delta file (DataFile4) must be added (Figure 3) (d)).

Subsequently, when data compaction is performed, some or all of the base file and the delta file are reflected and the delta files for the changes are combined into one (minor Compaction), an operation of merging the base file and delta file is performed (major compaction).

Accordingly, when major compaction is performed on (d) of FIG. 3, as can be seen in FIG. 3 (e), the entire base file and delta file All of these are merged and a new base file is created to replace the existing base file. Accordingly, the existing base file is deleted and the new base file ( A base file) may be stored in the storage unit 110 and the address of the new base file may be linked.

Next, in the merging processing step (S120), merging processing is performed on the original value of the data and the changed value of the data based on the request for the data, and further responds to the request based on this.

More specifically, as can be seen in FIG. 4, in the conventional data compaction system 120, when a request for data inquiry, modification, deletion, etc. by a user or application is received (S210), a response corresponding to the request is received (S210). The location of the data is confirmed in the database (S220).

Next, data is loaded from the storage unit 110 based on the confirmed location (S230), and it is determined whether it is necessary to merge the original and changed values of the data (S240).

At this time, if it is determined that merging is necessary, the original and changed values of the data are merged in a memory such as RAM (S250).

Accordingly, the data compaction system 120 responds to a request for inquiry of the data based on information merged in the memory (S260).

In this way, the data compaction system 120 responds to requests from users, etc. by merging data in real time.

However, in the conventional data compaction system 120, apart from the above real-time data processing, data compaction is performed in batches at predetermined cycles.

More specifically, as can be seen in FIG. 4, in the conventional data compaction system 120, when a batch compaction operation is started periodically based on a schedule predetermined by an administrator, etc. (S270), the The data compaction system 120 checks the entire list of files subject to compaction (S280), loads the files subject to compaction into memory, merges them, and then stores them in the story. It is stored in the branch 110 and the location information is updated (S290).

Accordingly, in the conventional data compaction system 120, overload of the system could be caused by periodically performing data compaction work on a large amount of data, and furthermore, repetitive data searches, etc. As the merge process was carried out, unnecessary consumption of computing resources could have occurred.

In contrast, in the present invention, a second base file in which the first base file and the first delta file are merged is generated based on whether the merge process is performed in the compaction processing step (S130), and the storage unit 110 ) can replace the first base file.

More specifically, as can be seen in FIG. 5, in the present invention, in the data compaction system 120, when a request for data inquiry, modification, deletion, etc. by a user or application is received (S310), the request is responded to. The location of the data is checked in the database, etc. (S320).

Next, data is loaded from the storage unit 110 based on the confirmed location (S330), and it is determined whether it is necessary to merge the original and changed values of the data (S340).

At this time, if it is determined that merging is necessary, the original and changed values of the data are merged in a memory such as RAM (S350).

Accordingly, the data compaction system 120 responds to the request based on the information merged in the memory (S360).

Furthermore, in the present invention, in the data compaction system 120, when data merge is performed in memory, etc. according to the request of the user, etc., the merged information is intercepted in real time. And, a new file is created based on the merged information to replace the base file and/or delta file in the storage unit 110 in real time.

More specifically, as can be seen in FIG. 5, in the present invention, when the data compaction system 120 detects the performance of merge processing, the merged information (including file location, memory address, etc.) is stored in memory, etc. can be intercepted (S410).

Additionally, the data compaction system 120 may store information intercepted from the memory, etc. in a separate cache storage device 126 (S420).

Subsequently, the data compaction system 120 stores the existing base file and/or delta file in the storage unit 110 based on the merged information stored in the cache storage device 126. A new base file and or delta file that replace the (delta file) can be created (S430).

Accordingly, the data compaction system 120 can modify the file location information to the location of the created new file and delete existing files (S440).

Accordingly, in the data compaction system 120 according to an embodiment of the present invention, system overload and It is possible to effectively prevent unnecessary consumption of computing resources due to repetitive merge processing.

In addition, the data compaction system 120 may be implemented using one or two or more servers, but the present invention is not necessarily limited thereto and may be implemented as a dedicated device or the data storage system 100. It is possible to implement it in various forms, such as being implemented as a process running on the same server or other device included in .

In addition, in FIG. 1, the data network 130 connecting the storage unit 110 and the data compaction system 120 may include a wired network and a wireless network, and specifically, a local area network (LAN). Network), metropolitan area network (MAN: Metropolitan Area Network), and wide area network (WAN: Wide Area Network). Additionally, the data network 130 may include the known World Wide Web (WWW). Furthermore, the data network 130 may be implemented using a data bus or the like configured to transmit and receive data.

In addition, in the present invention, the data compaction system 120 does not necessarily have to transmit and receive certain data with the storage unit 110 through the data network 130, and directly without going through the data network 130. It is also possible to implement it in various forms, such as transmitting and receiving data, or furthermore, the data compaction system 120 is integrated with the storage unit 110.

Additionally, Figures 6 and 7 describe in detail specific operations in the data compaction method according to an embodiment of the present invention.

The method shown in FIGS. 6 and 7 may be performed, for example, by the data compaction system 120, and further, the data compaction system 120 may be used as described later in relation to FIGS. 12 and 12. It may be implemented including a computing device. For example, the data compaction system 120 includes a processor 10, and the processor 10 can correct table coordinate information by executing instructions configured to implement an operation for correcting table coordinate information.

First, as can be seen in FIG. 6, when a request for data inquiry, etc. is received by a user, etc. (① in FIG. 6), the data compaction system 120 stores each base file and delta file stored in the storage unit 110. is loaded into memory, etc., and merge processing is performed on it to generate merge information (② and ③ in FIG. 6).

At this time, the memory interceptor 125 of the data compaction system 120 detects the performance of the merge process, collects the merge information from the memory, and transmits it to the cache storage device 126 (④ in FIG. 6), The cache storage device 126 stores the merged information (including file location, memory address, etc.) (⑤ in FIG. 6).

Accordingly, the smart compactor 127 can quickly process the compaction task based on the cache storage device 126 (⑥ in FIG. 6).

Furthermore, as can be seen in FIG. 7, the data compaction system 120 performs a compaction operation on merged data according to a request such as a data inquiry from a user, etc. to store the storage unit 110. It can be updated in real time, and accordingly, even if there are repeated requests for data inquiry by users, etc. (⑦ in FIG. 7), it is possible to prevent the merge process for data from being repeatedly performed.

In addition, Figure 8 illustrates a specific implementation example of the data compaction system 120 according to an embodiment of the present invention.

More specifically, as can be seen in FIG. 8, the operation of the module that processes requests such as user inquiries in the conventional data compaction system 120 is as follows.

First, the query according to the user's request is analyzed (S510), and the list of files corresponding to it is checked (S520).

Next, the files in the confirmed list are loaded into memory (S530), undergo additional computational processing (S540), and provide the final result to respond to the request (S550).

At this time, as can be seen in FIG. 8, the existing memory load module may have the same structure as (A-1) in FIG. 8. In this case, the data compaction system according to an embodiment of the present invention At 120, a separate application can be run in the background so that data can be saved from the MemoryLoad module to the cache storage device 126 (see (Figure 8). A-2)).

Accordingly, the data compaction system 120 according to an embodiment of the present invention performs the logic of the existing memory load module as is, minimizes the impact on the existing logic, and performs merge processing on the data. When detected, the task of saving the corresponding merge information to the cache storage device 126 can be performed in the background.

Additionally, in the data compaction method according to an embodiment of the present invention, in the compaction processing step (S130), the merged information is stored in the cache storage device 126 only when the size of the merged information is larger than a predetermined standard value. You can save it in .

Furthermore, in the data compaction method according to an embodiment of the present invention, in the compaction processing step (S130), when a predetermined reference time has elapsed since the time of replacing the first base file with the second base file Only then can the compaction operation on the second base file be performed again.

More specifically, as can be seen in FIG. 9, in the data compaction method according to an embodiment of the present invention, when the storage function for the cache storage device 126 is called (S610), an intercept is performed in memory. ) It is determined whether the size of the merged information is greater than or equal to a predetermined standard value (S620), and only when the size is greater than or equal to the standard value, the storage function for the cache storage device 126 can be performed (S630).

Accordingly, when the data compaction system 120 receives a data addition signal for the cache storage device 126 (S710), a predetermined reference time elapses based on the time when the data compaction task was previously performed. By determining whether or not the cache storage device 126 has been used (S720), the storage function for the cache storage device 126 can be performed only when a predetermined reference time has elapsed.

Additionally, it is possible to determine whether a file corresponding to the data to be added exists in the storage unit 110 and, if not, stop the storage function (S730).

Subsequently, the data compaction system 120 creates a new file in the storage unit 110 using the merge information intercepted from the memory (S740), and modifies the location information for the existing file (S740). S750), existing files can be deleted from the storage unit 110 (S760).

Accordingly, the data compaction system 120 deletes the merge information stored in the cache storage device 126 (S770), and information on the compaction performance details is stored in the meta as shown in FIG. 10. It can be created in the database 128 (S780).

At this time, as can be seen in FIG. 10, in the data compaction system 120, the memory interceptor 125 detects that the size of the merged information to be stored in the cache storage device 126 is larger than a predetermined reference value. It can only be stored in the cache storage device 126 (① in FIG. 10).

In addition, as can be seen in FIG. 10, in the data compaction system 120, the smart compactor 126 waits for a predetermined reference time to elapse after replacing the first base file with the second base file. Only in one case can the compaction operation on the second base file be performed again (② in FIG. 10).

At this time, the data compaction system 120 may delete the remaining merged information accumulated in the cache storage device 126 during the predetermined reference time, leaving only the final merged information.

In addition, in the data compaction system 120, the details for which the compaction has been completed can be deleted from the cache storage device 126 (③ in FIG. 10), and further, a specific time predetermined in the meta database 128 has elapsed. It is also possible to automatically delete the compaction completion list.

At this time, the method shown in FIGS. 9 and 10 may be performed, for example, by the data compaction system 120, and further, the data compaction system 120 may use the method described later with reference to FIGS. 12 and 12. Implementations may include a computing device as described. For example, the data compaction system 120 includes a processor 10, and the processor 10 can correct table coordinate information by executing instructions configured to implement an operation for correcting table coordinate information.

In addition, the computer program according to another aspect of the present invention is characterized as being a computer program stored in a computer-readable medium in order to execute each step of the data compaction method discussed above on a computer. The computer program may be a computer program including machine language code created by a compiler, as well as a computer program including high-level language code that can be executed on a computer using an interpreter or the like. At this time, the computer is not limited to a personal computer (PC) or laptop computer, but any information processing device equipped with a central processing unit (CPU) such as a server, smartphone, tablet PC, PDA, or mobile phone capable of executing a computer program. Includes device.

Additionally, a computer-readable medium may continuously store a computer-executable program or temporarily store it for execution or download. Additionally, the medium may be a variety of recording or storage means in the form of a single or multiple pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Accordingly, the above detailed description should not be construed as restrictive in all respects and should be considered illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Additionally, Figure 11 illustrates a configuration diagram of a data compaction system 120 according to an embodiment of the present invention.

As can be seen in FIG. 11, the data compaction system 120 according to an embodiment of the present invention includes a storage storage unit 121, a merge processing unit 122, and a compaction processing unit 123. You can.

Next, the data compaction system 120 according to an embodiment of the present invention will be examined by dividing it into each component. More detailed information about the data compaction system 120 according to an embodiment of the present invention can be inferred from the description of the data compaction method according to an embodiment of the present invention described above, and will be described in more detail below. is omitted.

First, the storage unit 121 stores a first base file containing original data and a first delta file containing changed values of the data in the storage unit 110.

In addition, the merge processing unit 122 performs merge processing on the original value of the data and the changed value of the data based on the request for the data, and can further use this to respond to the request.

Finally, when the compaction processing unit 123 detects that the merge process is performed, the compaction processing unit 123 generates a second base file in which the first base file and the first delta file are merged, and stores the first base file in the storage unit. It will be replaced.

Here, in the storage unit 110, changes to data included in the first base file may not be permitted.

In addition, the merge processing unit 122 loads the first base file and the first delta file into memory and performs merge processing, and the compaction processing unit 123 provides merge information for the merge processing performed in the memory. may be collected and based on this, the storage unit 110 may generate the second base file to replace the first base file.

At this time, the compaction processing unit 123 may collect the merge information from the memory and store it in the cache storage device 126.

In addition, the compaction processing unit 123 uses a separate application that runs independently of the merge processing unit 122 to store the story based on the merge information stored in the cache storage device 126. The second base file can be created in branch 110.

Furthermore, the compaction processing unit 123 may delete the first base file from the storage unit 110 and replace it with the second base file.

Additionally, Figure 12 illustrates a device 200 to which the proposed method of the present invention can be applied.

Referring to FIG. 12, the device 200 may be configured to implement a data compaction process according to the proposed method of the present invention. As an example, the device 200 may be a server device 200 that provides enterprise data storage services.

For example, devices 200 to which the proposed method of the present invention can be applied include network devices such as repeaters, hubs, bridges, switches, routers, gateways, computer devices such as desktop computers, workstations, etc., and mobile terminals such as smartphones. , portable devices such as laptop computers, home appliances such as digital TVs, and means of transportation such as cars. As another example, the device 200 to which the present invention can be applied may be included as part of an Application Specific Integrated Circuit (ASIC) implemented in the form of a System On Chip (SoC).

The memory 20 may be connected to the processor 10 when operating, and may store programs and/or instructions for processing and controlling the processor 10, data and information used in the present invention, and data according to the present invention. and control information required for information processing, data, and temporary data generated during the information processing process. The memory 20 includes Read Only Memory (ROM), Random Access Memory (RAM), Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory, and Static RAM (SRAM). , can be implemented as a storage device such as a hard disk drive (HDD), solid state drive (SSD), etc.

The processor 10 may be operatively connected to the memory 20 and/or the network interface 30 and controls the operation of each module within the device 200. In particular, the processor 10 can perform various control functions to perform the proposed method of the present invention. The processor 120 may also be called a controller, microcontroller, microprocessor, microcomputer, etc. The proposed method of the present invention may be implemented by hardware, firmware, software, or a combination thereof. When implementing the present invention using hardware, an application specific integrated circuit (ASIC) or a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), or an FPGA (FPGA) configured to perform the present invention is used. A field programmable gate array, etc. may be provided in the processor 10. Meanwhile, when the proposed method of the present invention is implemented using firmware or software, the firmware or software includes instructions related to modules, procedures, or functions that perform the functions or operations necessary to implement the proposed method of the present invention. It may include, and the instructions are stored in the memory 20 or in a computer-readable recording medium (not shown) separate from the memory 20, and when executed by the processor 10, the device 120 operates as the proposed method of the present invention. It can be configured to implement.

Additionally, device 200 may include a network interface device 30. The network interface device 30 is connected to the processor 10 when operating, and the processor 10 controls the network interface device 30 to carry information and/or data, signals, messages, etc. through a wireless/wired network. /Can transmit or receive wired signals. The network interface device 30 supports various communication standards, such as IEEE 802 series, 3GPP LTE(-A), 3GPP 5G, etc., and can transmit and receive control information and/or data signals according to the corresponding communication standards. The network interface device 30 may be implemented outside the device 200 as needed.

Accordingly, in the data compaction method, device, system, and computer program according to an embodiment of the present invention, data compaction (data) in storage stored in a manner that does not allow changes in a cloud system or database system, etc. It is possible to effectively suppress unnecessary consumption of computing resources due to system overload that may occur during the compaction process and repetitive merge processing.

Additionally, the data compaction method, device, system, and computer program according to an embodiment of the present invention can effectively improve problems such as changes in the data compaction performance environment depending on the manager's experience level, etc.

The above embodiments and drawings described in this specification are merely illustrative and do not limit the scope of the present invention in any way. In addition, the connections or connection members of lines between components shown in the drawings exemplify functional connections and/or physical or circuit connections, and in actual devices, various functional connections or physical connections can be replaced or added. Can be represented as connections, or circuit connections. Additionally, if there is no specific mention such as “essential,” “important,” etc., it may not be a necessary component for the application of the present invention.

In the specification (particularly in the claims) of the present invention, the use of the term “above” and similar referential terms may refer to both the singular and the plural. In addition, when a range is described in the present invention, the invention includes the application of individual values within the range (unless there is a statement to the contrary), and each individual value constituting the range is described in the detailed description of the invention. It's the same. In addition, the steps presented in the method invention of the present invention are not necessarily intended to restrict the sequential order, and the order may be appropriately changed as needed unless a step must be preceded depending on the nature of each process. there is. The use of any examples or illustrative terms (e.g., etc.) in the present invention is merely for explaining the present invention in detail, and unless limited by the claims, the scope of the present invention is limited by the examples or illustrative terms. It doesn't work. Additionally, those skilled in the art will understand that various modifications, combinations and changes may be made according to design conditions and elements within the scope of the appended claims or their equivalents.

10: processor
20: memory
30: interface device
100: data storage system
110: storage unit
120: data compaction system
121: storage storage unit
122: Merge processing execution unit
123: compaction processing unit
125: memory interceptor
126: Cache storage device
127: Smart compactor
128: Metadatabase
130: data network
200: device

Claims (17)

In a data compaction system, a storage step of storing a first base file containing original values of data and a first delta file containing changed values of the data in a storage unit;
A merge processing step of performing merge processing on the original value of the data and the changed value of the data based on the request for the data; and
A compaction processing step of generating a second base file in which the first base file and the first delta file are merged based on whether the merge processing is performed and replacing the first base file in the storage unit. A data compaction method characterized by: According to paragraph 1,
In the storage unit,
A data compaction method, characterized in that changes to data included in the first base file are not permitted. According to paragraph 1,
In the merge processing step, the first base file and the first delta file are loaded into memory and merge processing is performed,
In the compaction processing step, merge information for the merge process performed in the memory is collected, and based on this, the second base file is generated in the storage unit to replace the first base file. method. According to paragraph 3,
In the compaction processing step,
A data compaction method characterized in that the merged information is collected from the memory and stored in a cache storage device. According to paragraph 4,
In the compaction processing step,
Data compact, characterized in that generating the second base file in the storage unit based on the merge information stored in the cache storage device using a separate application that runs independently of the application that drives the merge processing step. Sean Method. According to clause 5,
In the compaction processing step,
A data compaction method characterized in that the first base file is deleted from the storage unit and replaced with the second base file. According to paragraph 4,
In the compaction processing step,
A data compaction method characterized in that the merged information is stored in the cache storage device only when the size of the merged information is larger than a predetermined standard value. According to paragraph 4,
In the compaction processing step,
A data compaction method characterized in that the compaction operation on the second base file is performed again only when a predetermined reference time has elapsed since the time of replacing the first base file with the second base file. According to clause 8,
In the compaction processing step,
A data compaction method, wherein the merged information accumulated in the cache storage device during the predetermined reference time is deleted, leaving only the final merged information and deleting the rest. According to clause 8,
In the compaction processing step,
A data compaction method characterized by automatically deleting a list of completions of compaction after a certain predetermined time has elapsed from the metadatabase. A computer program stored in a computer-readable recording medium for executing each step of the data compaction method according to any one of claims 1 to 10 on a computer. a storage unit that stores a first base file containing original data and a first delta file containing modified data;
a merge processing unit that performs merge processing on the original value of the data and the changed value of the data based on the request for the data; and
A compaction processing unit that generates a second base file in which the first base file and the first delta file are merged based on whether the merge processing is performed and replaces the first base file in the storage unit. Featured data compaction system. According to clause 12,
In the storage unit,
A data compaction system, wherein changes to data included in the first base file are not permitted. According to clause 12,
The merge processing unit loads the first base file and the first delta file into memory and performs merge processing,
A data compaction system wherein the compaction processing unit collects merge information for the merge process performed in the memory, and based on this, the storage unit generates the second base file to replace the first base file. . According to clause 14,
In the compaction processing unit,
A data compaction system characterized in that the merged information is collected from the memory and stored in a cache storage device. According to clause 15,
In the compaction processing unit,
A data compaction system, wherein the second base file is generated in the storage unit based on the merge information stored in the cache storage device using a separate application that runs independently of the merge processing unit. According to clause 16,
In the compaction processing unit,
A data compaction system characterized in that the first base file is deleted from the storage unit and replaced with the second base file.

Images