JP7274057B1 - Deduplication system, server device, deduplication method, and deduplication program - Google Patents

Abstract

A deduplication system (500) comprises a device (100) loaded with an image including a plurality of layers, each layer including a plurality of files, and a server device (200) for distributing the layers to the device (100). A duplication inspection unit (210) of a server device (200) inspects whether a duplicate file (321) is included in a distribution layer (31) distributed to a device (100) based on existing file information (22). . The deduplication unit (220) of the server device (200) distributes the deduplication layer (32) obtained by removing duplicate files (321) from the distribution layer (31) to the device (100). A layer management unit (110) of the device (100) restores the distribution layer (31) from the deduplication layer (32).

Description

The present disclosure relates to a deduplication system, equipment, server device, deduplication method, and deduplication program.

A deduplication technique is used to eliminate data duplication in a system in which a plurality of different virtual images are installed for each device and the virtual images are added, updated, or deleted. Such devices are assumed to be, for example, next-generation in-vehicle ECUs or edge devices with individual customization. ECU is an abbreviation for Electronic Control Unit.
Deduplication technology refers to reducing the amount of data transfer or storage capacity during backup or archiving. It also includes detecting and eliminating duplication of data on a file-by-file or block-by-block basis. Duplication can be determined by comparing data as it is, comparing data in combination with a checksum, or comparing data using a cryptographic hash function.

Japanese Patent Application Laid-Open No. 2002-200002 discloses a technology capable of effectively deduplicating files that constitute a virtual machine provided to a plurality of users.

JP 2009-276859 A

Patent Literature 1 discloses a deduplication technique for providing virtual computers to a plurality of users from a template of one virtual computer volume image. In Patent Document 1, reading and writing of each file in the image are traced to determine whether to provide the virtual machine for each user with an individual entity or to refer to a common file. In this way, the technique of Patent Document 1 does not check for duplicate use of the same file between different images. Therefore, Patent Document 1 has no effect on file duplication between different images. Also, by using a software container, file duplication between different images can be alleviated by sharing layers, but there is a problem that file duplication cannot be prevented when the same file exists in different layers.

An object of the present disclosure is to prevent duplication of files even when the same file exists in different layers.

A deduplication system according to the present disclosure includes:
An image formed from a plurality of layers and used for a software container, wherein each layer of said plurality of layers contains a plurality of files. In the exclusion system,
The server device
A duplication inspection unit that inspects whether a duplicate file that duplicates a file that exists in the image is included in a distribution layer that is a layer that is distributed to the device, based on existing file information that is information on the file that exists in the image. and,
a deduplication unit that generates a deduplication layer by removing the duplicate files from the distribution layer and distributes the deduplication layer to the device;
Said device comprises:
A layer management unit that restores the delivery layer from the deduplication layer is provided.

In the deduplication system according to the present disclosure, the duplication inspection unit, based on the existing file information, which is the information of the file that exists in the image, puts a duplicate that overlaps with the file that exists in the image in the distribution layer that is the layer that is distributed to the device. Check if the file is included. Then, the duplicate elimination unit generates a duplicate elimination layer by eliminating duplicate files from the delivery layer, and delivers the duplicate elimination layer to the device. Therefore, according to the deduplication system according to the present disclosure, it is possible to prevent file duplication even when the same file exists in different layers.

FIG. 2 is a diagram showing an example of an image used for a software container that is a premise of the first embodiment; FIG. 1 is a diagram showing an overall configuration example of a deduplication system according to Embodiment 1; FIG. 1 is a diagram showing a configuration example of a device according to Embodiment 1; FIG. 1 is a diagram showing a configuration example of a server device according to Embodiment 1; FIG. 4 is a flow chart showing the operation of the deduplication system according to Embodiment 1; FIG. 4 is a schematic diagram showing an example of the operation of the deduplication system according to Embodiment 1; FIG. 4 is a schematic diagram showing another example of the operation of the deduplication system according to Embodiment 1; FIG. 4 is a schematic diagram showing another example of the operation of the deduplication system according to Embodiment 1; FIG. FIG. 10 is a diagram showing a configuration example of a device according to Modification 2 of Embodiment 1; FIG. 10 is a diagram showing a configuration example of a server device according to Modification 2 of Embodiment 1; FIG. 5 is a diagram showing a configuration example of a device according to Embodiment 2; FIG. 10 is a diagram showing a configuration example of a server device according to Embodiment 2; FIG. 11 is a diagram showing a configuration example of an inter-layer duplication table according to the second embodiment; FIG. FIG. 10 is a flowchart showing the operation of the deduplication system according to Embodiment 2; FIG. 8 is a schematic diagram showing an example of the operation of the deduplication system according to the second embodiment;

The present embodiment will be described below with reference to the drawings. In each figure, the same reference numerals are given to the same or corresponding parts. In the description of the embodiments, the description of the same or corresponding parts will be omitted or simplified as appropriate. Arrows in the figure mainly indicate the flow of data or the flow of processing. Also, in the following drawings, the size relationship of each component may differ from the actual size. Also, in the description of the embodiments, directions or positions such as up, down, left, right, front, back, front, and back may be indicated. These notations are for the convenience of explanation, and do not limit the arrangement, direction, and orientation of devices, instruments, parts, and the like.

Embodiment 1.
*** Configuration description ***
FIG. 1 is a diagram showing an example of an image used for a software container that is a premise of this embodiment.
The virtualization method using the software container 10 is a method in which part of the OS of the host is separated, a dedicated area isolated from other software containers 10 is prepared, and software is operated thereon. OS is an abbreviation for Operating System.
The image 11 used for the software container 10 is formed from multiple layers 12 . Layers 12 are superimposed to form image 11 . Each layer 12 is a collection of files. An identifier 121 for uniquely identifying the layer 12 is assigned to the layer 12 . The identifier 121 is, for example, a layer 12 hash value and is also called a digest.
Typically, the same layers are shared between software containers 10 . Layer 12 is read-only.
The image 11 has a thin read/write layer added by the software container 10 and is used by the software container 10 .

FIG. 2 is a diagram showing an overall configuration example of the deduplication system 500 according to this embodiment.
A deduplication system 500 includes a device 100 and a server device 200 .
An image 11 is mounted on the device 100 . The image 11 to be mounted is different for each device.
The server apparatus 200 distributes the image 11 to the device 100 layer by layer as the image 11 is newly added, changed, or updated in the device 100 .

FIG. 3 is a diagram showing a configuration example of the device 100 according to this embodiment.
FIG. 4 is a diagram showing a configuration example of the server device 200 according to this embodiment.

In FIG. 3, device 100 is a computer. The device 100 comprises a processor 910 and other hardware such as a memory 921 , a secondary storage device 922 , an input interface 930 , an output interface 940 and a communication device 950 . The processor 910 is connected to other hardware via signal lines and controls these other hardware.
In FIG. 4, server device 200 is a computer. The server device 200 includes a processor 910 and other hardware such as a memory 921 , an auxiliary storage device 922 , an input interface 930 , an output interface 940 and a communication device 950 . The processor 910 is connected to other hardware via signal lines and controls these other hardware.
In addition, in FIG. 3 and FIG. 4, the same reference numerals are given to each piece of hardware for the purpose of simplifying the description. The device 100 and the server device 200 each have individual hardware.

In FIG. 3, the device 100 includes a layer management unit 110 and a storage unit 120 as functional elements. The storage unit 120 stores the existing layer 21 and the existing file information 22 .
The functions of the layer management unit 110 are realized by software. Storage unit 120 is provided in memory 921 . Note that the storage unit 120 may be provided in the auxiliary storage device 922 , or may be distributed between the memory 921 and the auxiliary storage device 922 .

In FIG. 4, the server device 200 includes a duplication checker 210, a duplicate remover 220, and a storage 230 as functional elements. The storage unit 230 stores the distribution layer 31 , the deduplication layer 32 and the existing file information 22 .
The functions of the duplicate checker 210 and the duplicate remover 220 are implemented by software. The storage unit 230 is provided in the memory 921 . Note that the storage unit 230 may be provided in the auxiliary storage device 922 , or may be distributed between the memory 921 and the auxiliary storage device 922 .

In the hardware description below, the server device 200 in FIG. 4 will be used as an example. The description of the hardware of the device 100 is the same.

Processor 910 is a device that executes a deduplication program. The deduplication program is a program that implements the functions of the server device 200 . Specifically, the deduplication program is a program that implements the functions of the duplication checker 210 and the duplicate remover 220 . A program that implements the functions of the device 100 is also called a deduplication program.

The processor 910 is an IC that performs arithmetic processing. Examples of processor 910 are CPU, DSP, GPU. The IC is the back of the Integrated Circuit. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. GPU is an abbreviation for Graphics Processing Unit.

The memory 921 is a storage device that temporarily stores data. A specific example of the memory 921 is SRAM or DRAM. SRAM is an abbreviation for Static Random Access Memory. DRAM is an abbreviation for Dynamic Random Access Memory.
Auxiliary storage device 922 is a storage device that stores data. A specific example of the auxiliary storage device 922 is an HDD. The auxiliary storage device 922 may be a portable storage medium such as an SD (registered trademark) memory card, CF, NAND flash, flexible disk, optical disk, compact disk, Blu-ray (registered trademark) disk, or DVD. Note that HDD is an abbreviation for Hard Disk Drive. SD® is an abbreviation for Secure Digital. CF is an abbreviation for CompactFlash®. DVD is an abbreviation for Digital Versatile Disk.

The input interface 930 is a port connected to an input device such as a mouse, keyboard, or touch panel. The input interface 930 is specifically a USB terminal. Note that the input interface 930 may be a port connected to a LAN. USB is an abbreviation for Universal Serial Bus. LAN is an abbreviation for Local Area Network.

The output interface 940 is a port to which a cable of an output device such as a display is connected. The output interface 940 is specifically a USB terminal or an HDMI (registered trademark) terminal. The display is specifically an LCD. Output interface 940 is also referred to as a display interface. HDMI (registered trademark) is an abbreviation for High Definition Multimedia Interface. LCD is an abbreviation for Liquid Crystal Display. Although one output interface 940 is illustrated in FIG. 1, there may be multiple output interfaces 940 .

Communication device 950 has a receiver and a transmitter. The communication device 950 is connected to a communication network such as LAN, Internet, WiFi (registered trademark), or telephone line. The communication device 950 is specifically a communication chip or NIC. NIC is an abbreviation for Network Interface Card.

A deduplication program is executed in the server device 200 . The deduplication program is loaded into and executed by processor 910 . The memory 921 stores not only the deduplication program but also the OS. OS is an abbreviation for Operating System. The processor 910 executes the deduplication program while executing the OS. A deduplication program and OS may be stored in the auxiliary storage device 922 . A deduplication program and OS stored in the auxiliary storage device 922 are loaded into the memory 921 and executed by the processor 910 . Note that part or all of the deduplication program may be incorporated into the OS.

Device 100 may include multiple processors in place of processor 910 . These multiple processors share the execution of the deduplication program. Each processor, like processor 910, is a device that executes a deduplication program.

The data, information, signal values and variable values used, processed or output by the deduplication program may be stored in memory 921 , secondary storage 922 , registers or cache memory within processor 910 .

The "part" of each part of the duplication inspection unit 210 and the duplicate elimination unit 220 may be read as "circuit", "process", "procedure", "processing", or "circuitry". The deduplication program causes the computer to perform duplicate check processing and deduplication processing. "Processing" of duplicate check processing and deduplication processing shall be read as "program", "program product", "computer-readable storage medium storing program", or "computer-readable recording medium storing program" good too.

The deduplication method is a method performed by each device of the deduplication system 500 executing a deduplication program.
The deduplication program may be provided by being stored in a computer-readable recording medium. Also, the deduplication program may be provided as a program product.

***Description of operation***
Next, the operation of the deduplication system 500 according to this embodiment will be described. The operation procedure of the deduplication system 500 corresponds to the deduplication method. Also, the program that implements the operation of the deduplication system 500 corresponds to the deduplication program.

FIG. 5 is a flowchart showing operations of the deduplication system 500 according to this embodiment.
FIG. 6 is a schematic diagram showing an example of the operation of the deduplication system 500 according to this embodiment.
FIG. 7 is a schematic diagram showing another example of the operation of the deduplication system 500 according to this embodiment.
FIG. 8 is a schematic diagram showing another example of the operation of the deduplication system 500 according to this embodiment.

An existing layer 21 in FIG. 5 is a layer that already exists in the device 100 . Specifically, each of all layers forming the image 11 mounted on the device 100 is called an existing layer 21 .
In the example of FIG. 6, layer A and layer B are existing layers 21 . Layer A contains files 1 and 2, and layer B contains files 3 and 4.
In the schematic diagrams of FIGS. 6 to 8, illustration of the image 11 mounted on the device 100 is omitted in order to simplify the explanation.

The existing file information 22 in FIG. 5 is a list of all files that already exist in the device 100 . Specifically, it is a list of files included in all existing layers 21 forming the image 11 loaded on the device 100 .
In the example of FIG. 6, layer A and layer B are existing layers 21 . Therefore, in the existing file information 22, file 1, file 2, file 3, and file 4 are listed.

The layer management unit 110 manages layers existing in the device 100 .
The existing file information 22 is generated by the layer management section 110 and transmitted to the server device 200 .
The server device 200 stores the existing file information 22 transmitted from the device 100 in the storage section 230 .

A distribution layer 31 in FIG. 5 is a layer that is distributed to the device 100 by the server device 200 . In other words, the distribution layer 31 is one of the layers that form the image 11 that the server device 200 intends to distribute to the device 100 .
In the example of FIG. 6, layer C is the distribution layer 31. In the example of FIG. Layer C is the layer that contains File 1 and File 5 .

<Duplicate inspection processing>
In step S<b>101 , based on the existing file information 22 , the duplication inspection unit 210 of the server device 200 checks whether the distribution layer 31 includes a duplicate file 321 that overlaps with a file that already exists in the image 11 mounted on the device 100 . to inspect. The duplication inspection unit 210 automatically inspects the duplicate file 321 from the existing file information 22 and the distribution layer 31 .
Specifically, the duplication checker 210 determines whether or not the distribution layer 31 includes the files listed in the existing file information 22 . If the distribution layer 31 includes a file listed in the existing file information 22 , the duplication checker 210 regards the file as a duplicate file 321 .

A specific example in FIG. 6 will be described.
The duplication checker 210 determines whether or not the layer C, which is the distribution layer 31 , includes the files listed in the existing file information 22 . Layer C contains File 1 listed in the existing file information 22 . Therefore, the duplication checker 210 sets the file 1 as the duplicate file 321 .

<Deduplication processing>
In step S<b>102 , the duplicate removal unit 220 generates the duplicate removal layer 32 by removing the duplicate file 321 from the distribution layer 31 . The duplication elimination unit 220 automatically eliminates duplicate files 321 from the distribution layer 31 based on the result of the duplication check. It should be noted that eliminating the duplicate file 321 from the distribution layer 31 means deleting the duplicate file 321 from the distribution layer 31 .
Then, in step S<b>103 , the deduplication unit 220 distributes the deduplication layer 32 to the device 100 via the communication device 950 . The deduplication layer 32 is provided with information about which files are excluded.

A specific example in FIG. 6 will be described.
The deduplication unit 220 generates the duplicate elimination layer 32 by eliminating the file 1, which is the duplicate file 321, from the layer C, which is the distribution layer 31. FIG. Then, the deduplication unit 220 distributes the deduplication layer 32 obtained by eliminating the file 1 from the layer C to the device 100 via the communication device 950 .

<Layer management processing>
In step S<b>104 , the layer management unit 110 of the device 100 restores the distribution layer 31 from the deduplication layer 32 . The layer management unit 110 restores the distribution layer 31 using the existing layer 21 and the deduplication layer 32 .
There are three methods for the layer management unit 110 to restore the distribution layer 31 as follows.

<<Example 1 of Layer Management Processing for Restoring Distribution Layer 31>
The layer management unit 110 restores the distribution layer by causing the duplicate file 321 included in one of the existing layers 21 to be referenced from the duplicate elimination layer 32 . Any of the existing layers 21 is any of a plurality of layers forming the image 11 mounted on the device 100 .

For example, the layer management unit 110 manages layer management information in which correspondence relationships between existing layers 21 existing in the device 100 and files included in the existing layers 21 and reference relationships between existing layers are set.
The layer management unit 110 automatically replaces the duplicate file 321 eliminated in the duplicate elimination layer 32 with a reference to the existing layer 21 in the layer management information.

In the specific example of FIG. 6, the layer management unit 110 replaces file 1 excluded in layer C with reference to file 1 in layer A, which is the existing layer 21, in the layer management information.

According to Example 1 of the layer management process, since duplicate files 321 are not transferred, the amount of transfer can be reduced. Also, since the duplicate file 321 can be shared between different layers, the storage area in the device 100 can be reduced.

<<Example 2 of Layer Management Processing for Restoring Distribution Layer 31>
The layer management unit 110 restores the distribution layer 31 by copying the duplicate file 321 included in any of the existing layers 21 to the deduplication layer 32 . By copying the duplicate file 321 to the deduplication layer 32 in this way, the distribution layer 31 is completely restored in the device 100 .

In the specific example of FIG. 7, the layer management unit 110 copies the file 1 included in the layer A which is the existing layer 21 to the layer C which is the deduplication layer 32 . As a result, the distribution layer 31 is restored.

According to Example 2 of the layer management process, since duplicate files 321 are not transferred, the amount of transfer can be reduced. In addition, since the duplicate file 321 can be copied to the deduplication layer 32 after transfer and the distribution layer 31 can be completely restored, processing such as changing file references can be reduced.

<<Example 3 of Layer Management Processing for Restoring Distribution Layer 31>
The layer management unit 110 restores the distribution layer 31 by moving duplicate files 321 included in any of the existing layers 21 to the deduplication layer 32 . Then, the layer management unit 110 refers to the duplicate file 321 included in the distribution layer 31 from the transfer source layer of the duplicate file 321 .
That is, the layer management unit 110 eliminates the duplicate file 321 from the existing layer 21 and automatically adds the duplicate file 321 to the duplicate elimination layer 32 . Then, the layer management unit 110 sets the layer management information so that the distribution layer 31 restored by the automatic addition of the duplicate file 321 is referred to from the existing layer 21 of the movement source of the duplicate file 321 .

In the specific example of FIG. 8, the server device 200 distributes the layer A' to the device 100 in order to update the layer A in the device 100 to the layer A'. Layer A′ is delivered to the device 100 as the deduplication layer 32 in which File 1, which is the duplicate file 321, has been eliminated.
The layer management unit 110 moves file 1 included in layer A to layer A′, which is the deduplication layer 32 . As a result, the layer A', which is the distribution layer 31, is restored. File 1 is excluded from layer A. The layer management unit 110 sets the layer management information so that the file 1 included in the layer A′ is referred to from the layer A of the movement source of the file 1 .

According to Example 3 of the layer management process, since duplicate files 321 are not transferred, the amount of transfer can be reduced. Also, after transfer, the duplicate file 321 is moved to the deduplication layer 32 and referred to the distribution layer 31 restored from the source layer. Therefore, even if the old layer is deleted after the source layer is updated to the distribution layer 31, there is no need to change the reference relationship.
Old layers may be retained during update due to rollback or B/G update. However, the old layers are likely to be deleted in the future. If another image file refers to the temporarily remaining old layer file, the range of influence when the old layer is deleted increases. Therefore, according to example 3 of the layer management process, it is possible to reduce the range of influence on the device 100 by operating so that the file of the new layer remains as an entity at the time of updating.

***Other Configurations***
<Modification 1>
In this embodiment, the existing file information 22 is generated by the layer management section 110 and transmitted to the server device 200 . However, the server device 200 may create the existing file information 22 .
Since the server device 200 distributes the image 11 to the device 100 in units of layers, the correspondence between layers and files and which layer has been distributed to which device 100 are recorded. Therefore, the server device 200 can create the existing file information 22, which is information about files existing in the device 100, for each device.

<Modification 2>
In this embodiment, the functions of each device of the deduplication system 500 are realized by software. As a modification, the functions of each device of the deduplication system 500 may be implemented by hardware.
Specifically, each device of deduplication system 500 includes electronic circuitry 909 in place of processor 910 .

FIG. 9 is a diagram showing a configuration example of device 100 according to Modification 2 of the present embodiment.
FIG. 10 is a diagram showing a configuration example of a server device 200 according to Modification 2 of the present embodiment.
In the following description, the server device 200 in FIG. 10 will be described as an example. The description of the device 100 is also the same.

The electronic circuit 909 is a dedicated electronic circuit that realizes the functions of the duplication checker 210 and the duplication remover 220 . Electronic circuit 909 is specifically a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, GA, ASIC, or FPGA. GA is an abbreviation for Gate Array. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field-Programmable Gate Array.

The functions of the duplication checking unit 210 and the duplication elimination unit 220 may be implemented by one electronic circuit, or may be implemented by being distributed among a plurality of electronic circuits.

As another modification, part of the functions of the duplication checker 210 and the duplicate remover 220 may be realized by electronic circuits, and the rest of the functions may be realized by software. Also, part or all of the functions of the duplicate checker 210 and the duplicate remover 220 may be realized by firmware.

Each of the processor and electronic circuitry is also called processing circuitry. In other words, the function of each device of the deduplication system 500 is realized by processing circuitry.

***Description of the effects of the present embodiment***
As described above, according to the deduplication system 500 according to the present embodiment, duplication of the same file in different layers can be eliminated.
In the deduplication system 500 according to the present embodiment, layers are transferred after removing duplicate files, so the amount of transfer can be reduced.

Further, according to Example 1 of layer management processing in the deduplication system 500 according to the present embodiment, duplicate files can be shared between different layers on the same device, so the storage area in the device can be reduced. .

Further, according to Example 2 of the layer management processing in the deduplication system 500 according to the present embodiment, duplicate files can be copied to the deduplication layer after transfer, and the distribution layer can be completely restored. Therefore, processing such as changing file references can be reduced, and the image can be easily handled by the device.

Further, according to example 3 of layer management processing in the deduplication system 500 according to the present embodiment, duplicate files are moved to the deduplication layer after transfer, and reference is made to the distribution layer restored from the source layer. Therefore, even if the old layer is deleted after the source layer is updated to the distribution layer, there is an effect that the reference relationship does not need to be changed.

Embodiment 2.
In this embodiment, points different from or added to the first embodiment will be described.
In Embodiment 1, duplication was checked for all existing files on the device. On the other hand, in the present embodiment, a description will be given of a mode in which duplicate inspection is performed for each combination of layers.

FIG. 11 is a diagram showing a configuration example of the device 100 according to this embodiment.
FIG. 12 is a diagram showing a configuration example of the server device 200 according to this embodiment.
In the present embodiment, device 100 and server device 200 have existing layer information 23 in addition to the configuration described in the first embodiment. Moreover, the server device 200 further includes an inter-layer duplication table 24 .
Note that, in the present embodiment, the existing file information 22 may not exist in the device 100 and the server device 200 .

The existing layer information 23 is a list of layers that already exist in the device 100 . Specifically, the existing layer information 23 is a list of identifiers that identify layers existing in the image 11 mounted on the device 100 . This identifier, called a digest, is a hash value for each of the layers present in image 11 . For example, the existing layer information 23 may be generated by the layer management section 110 and transmitted to the server device 200 . Alternatively, the existing layer information 23 may be generated on the server device 200 side.

FIG. 13 is a diagram showing a configuration example of the inter-layer duplication table 24 according to this embodiment.
The inter-layer duplication table 24 is a table that records the duplication relationship of files for each combination of layers that have already been delivered to the device 100 .
For example, in FIG. 13, the combination of layer A and layer D indicates that file 1 is duplicated. Also, the combination of layer B and layer D indicates that file 1 and file 3 overlap.

FIG. 14 is a flowchart showing operations of the deduplication system 500 according to this embodiment.
FIG. 15 is a schematic diagram showing an example of the operation of the deduplication system 500 according to this embodiment.

In step S<b>101 a , based on the inter-layer duplication table 24 , the duplication inspection unit 210 determines whether the distribution layer 31 is recorded in the inter-layer duplication table 24 .
If the distribution layer 31 is recorded in the inter-layer duplication table, the duplication checking unit 210 checks whether there is a duplicate file in the distribution layer 31 based on the inter-layer duplication table 24 . Note that the duplication inspection unit 210 may inspect whether or not there is a duplicate file in the distribution layer 31 based on the existing layer information 23 and the inter-layer duplication table 24 .

If the distribution layer 31 is not recorded in the inter-layer duplication table, the duplication checker 210 checks whether there is a duplicate file for each combination of the existing layer 21 and the distribution layer 31 based on the existing layer information 23 . Since the server device 200 distributes the layers themselves, it has information about which files are included in which layers. Therefore, the duplication checker 210 can check whether there is a duplicate file for each combination of the existing layer 21 and the distribution layer 31 based only on the existing layer information 23 .
In step S<b>101 b , the duplication inspection unit 210 records in the inter-layer duplication table 24 the result of checking whether or not there is a duplicate file for each combination of the existing layer 21 and the distribution layer 31 .
The processing after step S102 is the same as in the first embodiment.

***Description of the effects of the present embodiment***
As described above, according to the deduplication system 500 according to the present embodiment, it is possible to check the duplication of files using the inter-layer duplication table, so that the processing load of the duplication check can be reduced.

In the above first and second embodiments, each part of each device of the deduplication system has been described as an independent functional block. However, the configuration of each device in the deduplication system does not have to be the configuration of the embodiment described above. The functional blocks of each device of the deduplication system may have any configuration as long as they can implement the functions described in the above embodiments. Also, each device in the deduplication system may be a system composed of a plurality of devices instead of a single device.
Also, a plurality of portions of the first and second embodiments may be combined for implementation. Alternatively, one portion of these embodiments may be implemented. In addition, these embodiments may be implemented in any combination as a whole or in part.
That is, in Embodiments 1 and 2, it is possible to freely combine each embodiment, modify any component of each embodiment, or omit any component from each embodiment.

The above-described embodiments are essentially preferable examples, and are not intended to limit the scope of the present disclosure, the scope of application of the present disclosure, and the range of applications of the present disclosure. Various modifications can be made to the above-described embodiments as required. For example, procedures described using flow diagrams or sequence diagrams may be modified as appropriate.

10 software container, 11 image, 12 layer, 121 identifier, 21 existing layer, 22 existing file information, 23 existing layer information, 24 inter-layer duplication table, 31 distribution layer, 32 duplicate elimination layer, 321 duplicate file, 100 device, 110 Layer Management Unit 120, 230 Storage Unit 200 Server Device 210 Duplication Inspection Unit 220 Deduplication Unit 500 Deduplication System 909 Electronic Circuit 910 Processor 921 Memory 922 Auxiliary Storage Device 930 Input Interface 940 Output Interface, 950 Communication Device.

Claims (9)

An image formed from a plurality of layers and used for a software container, wherein each layer of said plurality of layers contains a plurality of files. In the exclusion system,
comprising an inter-layer duplication table in which a file duplication relationship for each combination of layers already delivered to the device is recorded;
The server device
A duplication inspection unit that inspects whether a duplicate file that duplicates a file that exists in the image is included in a distribution layer that is a layer that is distributed to the device, based on existing file information that is information on the file that exists in the image. and determining whether the distribution layer is recorded in the inter-layer duplication table based on the inter-layer duplication table, and if the distribution layer is recorded in the inter-layer duplication table, the inter-layer a duplication inspection unit that inspects whether there is a duplicate file in the distribution layer based on the duplication table ;
a deduplication unit that generates a deduplication layer by removing the duplicate files from the distribution layer and distributes the deduplication layer to the device;
Said device comprises:
A deduplication system comprising a layer manager that restores the distribution layer from the deduplication layer. The duplication inspection unit
If the distribution layer is not recorded in the inter-layer duplication table, whether there is a duplicate file for each combination of the existing layer and the distribution layer based on existing layer information, which is information on layers existing in the image. , and recorded in the inter-layer duplication table. The existing layer information is
a list of identifiers identifying layers present in the image;
3. The deduplication system of claim 2, wherein the identifier is a hash value for each layer present in the image. The layer management unit
2. The deduplication system of claim 1, wherein the delivery layer is restored by referencing the duplicate files included in any of the layers forming the image from the deduplication layer. The layer management unit
2. The deduplication system of claim 1, wherein the delivery layer is restored by copying the duplicate files contained in any of the layers forming the image to the deduplication layer. The layer management unit
restoring the distribution layer by moving the duplicate files included in any of the plurality of layers forming the image to the deduplication layer; and removing the duplicate files included in the distribution layer from the duplicate files. 2. The deduplication system according to claim 1, wherein reference is made from the migration source layer. In a server device that distributes an image formed from a plurality of layers and used for a software container to a device on which the image is mounted, each layer of the plurality of layers including a plurality of files,
The server device
comprising an inter-layer duplication table in which a file duplication relationship for each combination of layers already delivered to the device is recorded;
A duplication inspection unit that inspects whether a duplicate file that duplicates a file that exists in the image is included in a distribution layer that is a layer that is distributed to the device, based on existing file information that is information on the file that exists in the image. and determining whether the distribution layer is recorded in the inter-layer duplication table based on the inter-layer duplication table, and if the distribution layer is recorded in the inter-layer duplication table, the inter-layer a duplication inspection unit that inspects whether there is a duplicate file in the distribution layer based on the duplication table ;
A server device comprising: a deduplication unit that generates a deduplication layer by deduplicating the duplicate file from the distribution layer and distributes the deduplication layer to the device. An image formed from a plurality of layers and used for a software container, wherein each layer of said plurality of layers contains a plurality of files. In the deduplication method used in the deduplication system,
The deduplication system comprises an inter-layer duplication table that records duplication relationships of files for each combination of layers that have already been delivered to the device,
The server device
Based on existing file information, which is information of files existing in the image, it is a process of inspecting whether a duplicate file that overlaps with a file existing in the image is included in a distribution layer that is a layer to be distributed to the device. and determining whether the distribution layer is recorded in the inter-layer duplication table based on the inter-layer duplication table, and if the distribution layer is recorded in the inter-layer duplication table, the inter-layer duplication table Based on, perform a process to check whether there is a duplicate file in the delivery layer ,
generating a deduplication layer by eliminating the duplicate files from the delivery layer, delivering the deduplication layer to the device;
Said device comprises:
A deduplication method for recovering the delivery layer from the deduplication layer. An image formed from a plurality of layers and used for a software container, wherein each layer of said plurality of layers contains a plurality of files. In the deduplication program used in the deduplication system,
The deduplication system comprises an inter-layer duplication table that records duplication relationships of files for each combination of layers that have already been delivered to the device,
Duplication inspection processing for inspecting whether a duplicate file that overlaps with a file that exists in the image is included in a distribution layer that is a layer that is distributed to the device, based on existing file information that is information on the file that exists in the image. and determining whether the distribution layer is recorded in the inter-layer duplication table based on the inter-layer duplication table, and if the distribution layer is recorded in the inter-layer duplication table, the inter-layer Duplication inspection processing for inspecting whether there is a duplicate file in the delivery layer based on the duplication table ;
A deduplication process for generating a deduplication layer by removing the duplicate files from the distribution layer and distributing the deduplication layer to the device;
A deduplication program for causing a computer to execute layer management processing for restoring the distribution layer from the deduplication layer distributed to the device.

Images