JP2020149229A - Duplicate eliminating apparatus, duplicate eliminating method, program and storage media - Google Patents

Abstract

To provide a duplicate eliminating apparatus capable of low-cost and short-time processing while minimizing high-load processing.SOLUTION: In a duplicate eliminating apparatus 1 of the present invention, an identification unit 11 identifies the type of operating system information based on file system information existing in a first area of a data group stored in a data storage area having identification information in a storage, a duplicate area detection unit 12 detects an area in which duplicate data exists from the data group having the same type of the operating system information in the data storage area, and a duplicate eliminating unit 13 stores the duplicate data in at least one of the areas and eliminates the duplicate data from the other areas.SELECTED DRAWING: Figure 1

Description

The present invention relates to deduplication devices, deduplication methods, programs and recording media.

Conventionally, in order to suppress the storage capacity cost in storage, a technique for eliminating duplicated data from the storage has been used (Patent Document 1).

Japanese Patent No. 4990828

However, comparing all the data in the storage and confirming the duplication of data requires a heavy processing. Therefore, in low-performance hardware, there is a problem that the processing takes time. On the other hand, high-performance hardware that enables short-time processing even if the processing load is high has a problem that it is difficult to implement because of its high cost.

Therefore, an object of the present invention is to provide a deduplication device, a deduplication method, a program, and a recording medium that can perform processing in a short time while minimizing high-load processing at low cost.

In order to achieve the above object, the deduplication device of the present invention is
Including a specific part, an overlapping area detection part, and a deduplication part
The specific unit identifies the type of operating system information based on the file system information existing in the first area of the data group stored in the data storage area having the identification information in the storage.
In the data storage area, the overlapping area detection unit detects an area in which duplicate data exists from the data group having the same type of operating system information.
The deduplication unit is a device that stores the duplication data in at least one of the regions and eliminates the duplication data from the other regions.

The deduplication method of the present invention
Including a specific step, a duplication region detection step, and a duplication elimination step
In the specific step, the type of operating system information is specified based on the file system information existing in the first area of the data group stored in the data storage area having the identification information in the storage.
The overlapping area detection step detects an area in which duplicate data exists from the data group having the same type of operating system information in the data storage area.
The deduplication step is a method of storing the duplicated data in at least one of the regions and eliminating the duplicated data from the other regions.

According to the present invention, for example, low-cost, high-load processing can be minimized, and short-time processing can be performed.

FIG. 1 is a block diagram showing a configuration of an example of the device of the first embodiment. FIG. 2 is a block diagram showing an example of the hardware configuration of the apparatus of the first embodiment. FIG. 3 is a flowchart showing an example of processing in the apparatus of the first embodiment. FIG. 4 is a schematic diagram showing an example of the configuration of the data group in the apparatus of the first embodiment. FIG. 5 is a schematic diagram showing an example of a case where the storage is virtualized storage in the device of the first embodiment. FIG. 6 is a block diagram showing a configuration of an example of the device of the second embodiment. FIG. 7 is a schematic diagram showing an example of specifying a data storage area in the apparatus of the second embodiment.

The device of the present invention further includes a data storage area specifying unit, which identifies the data storage area having the same type of operating system information by using the identification information, and the duplication. The area detection unit may be in an embodiment in which an area in which duplicate data exists is detected from the data group having the same type of operating system information between the specified data storage areas.

In the apparatus of the present invention, the specific unit specifies the type of operating system information based on the file system information existing in the first area of the virtual disk area of the data group in the virtualized storage. There may be.

The method of the present invention further includes a data storage area specifying step, in which the data storage area specifying step identifies the data storage area having the same type of operating system information using the identification information, and the duplication. The area detection step may be an embodiment in which an area in which duplicate data exists is detected from the data group having the same type of operating system information between the specified data storage areas.

In the method of the present invention, the specific step identifies the type of operating system information based on the file system information existing in the first area of the virtual disk area of the data group in the virtualized storage. There may be.

The program of the present invention is a program capable of executing the method of the present invention on a computer.

The recording medium of the present invention is a computer-readable recording medium on which the program of the present invention is recorded.

The duplicated data is data having the same contents due to, for example, an OS area of the same type of operating system (hereinafter, OS), data backup and copying, and the like. In general, the situation where data is duplicated is often saved on the same OS. For example, in order for data to be duplicated in the OS area, it is essential that the same OS is used. Further, for example, in the case of the backup, the OS is also duplicated, so that the OS is the same. Further, for example, in the case of the copy, it is assumed that the copy is performed in order to retain the past data, so that the OS is assumed to be the same.

An embodiment of the present invention will be described with reference to the drawings. The present invention is not limited to the following embodiments. In each of the following figures, the same parts are designated by the same reference numerals. Further, the explanations of the respective embodiments can be referred to each other unless otherwise specified, and the configurations of the respective embodiments can be combined unless otherwise specified.

[Embodiment 1]
FIG. 1 is a block diagram showing a configuration of an example of the deduplication device 1 of the present embodiment. As shown in FIG. 1, the apparatus 1 includes a specific unit 11, an overlap region detection unit 12, and a deduplication unit 13.

The device 1 may be, for example, one device including the above-mentioned parts, or a device in which the above-mentioned parts can be connected via a communication network. Further, although not shown, the present device 1 can be connected to an external device described later via the communication network. The communication network is not particularly limited, and a known network can be used. For example, it may be wired or wireless. Examples of the communication line network include an Internet line, a telephone line, a LAN (Local Area Network), WiFi (Wireless Fidelity), and the like. The apparatus 1 may be incorporated in the server as a system, for example. Further, the present device 1 may be, for example, a terminal in which the program of the present invention is installed. The terminal is not particularly limited, and examples thereof include a personal computer (PC), a smartphone, a tablet, and a mobile phone.

FIG. 2 illustrates a block diagram of the hardware configuration of the present device 1. The device 1 includes, for example, a CPU (central processing unit) 101, a memory 102, a bus 103, a storage device 104, an input device 105, a display device (display) 106, a communication device 107, and the like. The device 1 is, for example, a device in which the program of the present invention or the like is built in the storage device 104 of the computer 100. The computer 100 is a general computer. Each part of the apparatus 1 is connected to each other via the bus 103 by each interface (I / F).

The CPU 101 is responsible for controlling the entire device 1. In the present device 1, for example, the program of the present invention and other programs are executed by the CPU 101, and various information is read and written. In the present invention, another arithmetic unit such as a GPU may be used instead of the CPU.

The bus 103 can also be connected to, for example, an external device. Examples of the external device include an external storage device (external storage and the like), a printer, and the like, which will be described later. The device 1 can be connected to the external network (communication network) 3 by the communication device 107 connected to the bus 103, and can also be connected to the external device via the external network 3.

Examples of the memory 102 include a main memory (main storage device). When the CPU 101 performs processing, for example, the memory 102 reads various operation programs such as the program of the present invention stored in the storage device 104 described later, and the CPU 101 receives data from the memory 102 and programs. To execute. The main memory is, for example, a RAM (random access memory). Further, the memory 102 may be, for example, a ROM (read-only memory).

The storage device 104 is also referred to as a so-called auxiliary storage device with respect to the main memory (main storage device), for example. As described above, the storage device 104 stores an operation program including the program of the present invention. The storage device 104 may be, for example, a combination of a storage medium and a drive for reading and writing to the storage medium. The storage medium is not particularly limited, and may be an internal type or an external type, and examples thereof include HD (hard disk), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, and memory card. Be done. The storage device 104 may be, for example, a hard disk drive (HDD) in which a storage medium and a drive are integrated. Further, the storage device 104 may store the OS.

That is, in FIG. 2, it can be said that the apparatus 1 is a computer 100 in which the memory 102 reads the program of the present invention, the CPU 101 receives data from the memory 102, and executes the program of the present invention.

The device 1 further includes, for example, an input device 105 and a display 106. The input device 105 is, for example, a touch panel, a keyboard, a mouse, or the like. Examples of the display 106 include an LED display and a liquid crystal display.

The identification unit 11 identifies the type of operating system information (hereinafter, also referred to as OS information) based on the file system information existing in the first area of the data group stored in the data storage area having the identification information in the storage. To do. Examples of the identification information include LUN (Logical Unit Number) and the like. The data group is data in which information is recorded in an area for each file or directory in the file system. In the present embodiment, the first area means the earliest area in which the file system information exists and the reading is started.

FIG. 4 shows an example of the configuration of the data group. In FIG. 4, the data group starts reading data from the left side. The file system information exists in the first area as described above. Further, for example, an OS file in which OS information is recorded exists behind (on the right side) the file system information. Further, behind (on the right side) the OS file, for example, there is another file such as an AP file in which active page (AP) information is recorded.

The storage is not particularly limited, and for example, as shown in FIG. 2, the storage may be the storage in the storage device 104, or the external storage 2 connected to the device 1 by the communication network 3 may be targeted. The external storage 2 is not particularly limited, and may be, for example, virtualized storage or storage in the user device. When the storage is virtualized storage, the identification unit 11 further obtains operating system information based on the file system information existing in the first area of the virtual disk area of the data group in the virtualized storage. The type may be specified.

An example of the case where the storage is a virtualized storage will be described with reference to FIG. 5 (A) is a schematic diagram showing the relationship between the virtual server and the storage, and FIG. 5 (B) is a schematic diagram showing an example of the structure of the data group in the storage of FIG. 5 (A). is there. In FIG. 5A, each server is a virtual server existing on a server virtualization board, and each of the virtual servers includes an OS and a file system, respectively. In FIG. 5A, the number of each virtual server is three, but the figure is an example and is not limited thereto. As shown in FIG. 5A, each of the virtual servers shares and uses the data storage area of the virtualized storage. On the other hand, the data group shown in FIG. 5B starts reading data from the left side. As shown in FIG. 5B, the data group in the virtualized storage has a virtual disk area for each virtualization machine of the virtual server. Further, the virtualization board file system information exists in the area on the start side rather than the virtual disk area on the start side. The file system information exists in the earliest area (first area) on the start side of each area in each virtual disk area, and the OS file is behind (right side) of the file system information. Exists in. Although not shown, the other file may exist behind the OS file.

The overlapping area detection unit 12 detects an area in which duplicate data exists from the data group having the same type of OS information in the data storage area. The duplicated data is, for example, data having the same contents due to the OS area of the same type of OS, data backup, copy, etc., as described above. The area where the duplicated data exists is also referred to as a duplicated area.

The deduplication unit 13 stores the duplicated data in at least one of the regions and eliminates the duplicated data from the other regions. The storage area may be one or two or more. In the storage, for example, a plurality of the duplicated data may be accumulated and stored, or only specific duplicated data may be stored. The specific duplicated data is not particularly limited, and examples thereof include data in which the latest date and time are recorded, original data of a copy, and the like. The exclusion may be eliminated, for example, by deleting the duplicate data from the other area (that is, the area other than the storage area), or the area for storing the duplicate data and an external device. It may be eliminated by moving to or the like.

Next, an example of the processing in the present device 1 will be described with reference to the block diagram of FIG. 1 and the flowchart of FIG.

First, in the specific step, the type of operating system information is specified based on the file system information existing in the first area of the data group stored in the data storage area having the identification information in the storage (S1). Further, when the storage is virtualized storage, the specific step further obtains operating system information based on the file system information existing in the first area of the virtual disk area of the data group in the virtualized storage. The type may be specified.

Next, the overlapping area detection step detects an area in which the duplicated data exists from the data group having the same type of operating system information in the data storage area (S2).

Next, by the deduplication step, the duplication data is stored in at least one of the regions, and the duplication data is excluded from the other regions (S3), and the process ends (END).

According to the present embodiment, the detection range of the duplicated data can be narrowed down by specifying the OS, so that even low-cost hardware (that is, low-performance hardware) has a high load. Can be minimized and processed in a short time.

[Embodiment 2]
In this embodiment, a plurality of the data storage areas are further present in the storage. Unless otherwise specified, the description of the first embodiment can be incorporated in the present embodiment.

As shown in FIG. 6, the present device 1 is the same as the deduplication device 1 of the first embodiment except that the data storage area specifying unit 14 is further included.

FIG. 7 is a schematic diagram showing an example of specifying the data storage area. Note that FIG. 7 is an example and is not limited thereto. In FIG. 7, each server 20 (20a, 20b, and 20c) includes an OS and a file system. In FIG. 7, the number of the OS indicates the type of the OS, and for example, the OS1 of the server 20a and the OS1 of the server 20b indicate the same type of OS information. On the other hand, for example, it is shown that OS1 of the server 20a and OS2 of the server 20c are different types of OS information. Further, the number of the file system indicates the type of the file system as well as the OS. Each server 20 shares and uses one storage 21. Further, in the storage 21, each data storage area 22 (22a, 22b, and 22c) having each LUN (identification information) associated with each server 20 exists. In this way, when a plurality of the data storage areas exist in the storage, the data storage area specifying unit 14 identifies the data storage area having the same type of operating system information by using the identification information. That is, the data storage area specifying unit 14 can specify the data storage areas 22a and 22b having the respective LUNs associated with the servers 20a and 20b including the OS1.

In the present embodiment, the overlapping area detection unit 12 detects an area in which duplicate data exists from the data group having the same type of operating system information between the specified data storage areas. That is, the overlapping area detection unit 12 detects the overlapping area from the data group between the data storage areas 22a and 22b.

In addition, the deduplication method of the present embodiment further includes a data storage area specifying step. In the data storage area specifying step, the data storage area having the same type of operating system information is specified by using the identification information. In addition, the overlapping area detection step detects an area in which duplicate data exists from the data group having the same type of operating system information between the specified data storage areas.

Similar to the deduplication device 1 of the first embodiment, the deduplication device 1 of the present embodiment also enables low-performance hardware to minimize high-load processing and to perform short-time processing. ..

[Embodiment 3]
The program of the present embodiment is a program capable of executing the deduplication method of the first and second embodiments on a computer. Further, the program of the present embodiment may be recorded on a computer-readable recording medium, for example. The recording medium is not particularly limited, and examples thereof include a read-only memory (ROM), a hard disk (HD), and an optical disk.

Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the structure and details of the present invention within the scope of the present invention.

According to the present invention, for example, low-cost, high-load processing can be minimized, and short-time processing can be performed. For this reason, the present invention is particularly useful when, for example, data deduplication is performed using low performance hardware.

1 Deduplication device 2 External storage 3 Communication network 11 Specific unit 12 Overlapping area detection unit 13 Deduplication unit 14 Data storage area identification unit 20 (20a, 20b, 20c) Server 21 Storage 22 (22a, 22b, 22c) Data storage Area 100 Computer 101 CPU
102 Memory 103 Bus 104 Storage device 105 Input device 106 Display 107 Communication device

Claims (8)

Including a specific part, an overlapping area detection part, and a deduplication part
The specific unit identifies the type of operating system information based on the file system information existing in the first area of the data group stored in the data storage area having the identification information in the storage.
In the data storage area, the overlapping area detection unit detects an area in which duplicate data exists from the data group having the same type of operating system information.
The deduplication unit is a deduplication device that stores the duplication data in at least one of the regions and eliminates the duplication data from the other regions. In addition, it includes a data storage area identification unit.
The data storage area identification unit identifies the data storage area having the same type of operating system information by using the identification information.
The deduplication device according to claim 1, wherein the duplication area detection unit detects an area in which duplicate data exists from the data group having the same type of operating system information between the specified data storage areas. The deduplication according to claim 1 or 2, wherein the specific unit specifies the type of operating system information based on the file system information existing in the first area of the virtual disk area of the data group in the virtualized storage. apparatus. Including a specific step, a duplication region detection step, and a duplication elimination step
In the specific step, the type of operating system information is specified based on the file system information existing in the first area of the data group stored in the data storage area having the identification information in the storage.
The overlapping area detection step detects an area in which duplicate data exists from the data group having the same type of operating system information in the data storage area.
The deduplication step stores the duplicated data in at least one of the regions and eliminates the duplicated data from the other regions.
Deduplication method. In addition, it includes a data storage area identification step.
In the data storage area specifying step, the data storage area having the same type of operating system information is specified by using the identification information.
The deduplication method according to claim 4, wherein the duplication area detection step detects an area in which duplicate data exists from the data group having the same type of operating system information between the specified data storage areas. The deduplication according to claim 4 or 5, wherein the specific step specifies the type of operating system information based on the file system information existing in the first area of the virtual disk area of the data group in the virtualized storage. Method. A program capable of executing the method according to any one of claims 4 to 6 on a computer. A computer-readable recording medium on which the program according to claim 7 is recorded.

Images