WO2023199427A1

WO2023199427A1 - Duplicate-removal system, apparatus, server device, duplicate-removal method, and duplicate-removal program

Info

Publication number: WO2023199427A1
Application number: PCT/JP2022/017700
Authority: WO
Inventors: 康平山本; 武尚水口; 整山本; 圭祐堀井; 克久小笠原; 祐介瀬戸
Original assignee: 三菱電機株式会社
Priority date: 2022-04-13
Filing date: 2022-04-13
Publication date: 2023-10-19
Also published as: JP7274057B1; JPWO2023199427A1

Abstract

A duplicate-removal system (500) comprises: an apparatus (100) for loading an image including a plurality of layers, each layer including a plurality of files; and a server device (200) for delivering the layers to the apparatus (100). On the basis of existing file information (22), a duplicate inspection unit (210) of the server device (200) inspects for whether a duplicate file (321) is includes in a delivery layer (31) to be delivered to the apparatus (100). A duplicate-removal unit (220) of the server device (200) delivers, to the apparatus (100), a duplicate-removed layer (32) obtained by removing the duplicate file (321) from the delivery layer (31). A layer management unit (110) of the apparatus (100) restores the delivery layer (31) from the duplicate-removed layer (32).

Description

Deduplication system, equipment, server device, deduplication method, and deduplication program

The present disclosure relates to a deduplication system, a device, a server device, a deduplication method, and a deduplication program.

Deduplication technology is used to eliminate duplication of data in systems where each device is equipped with a number of different virtual images and the virtual images are added, updated, or deleted. Such devices are assumed to be, for example, next-generation in-vehicle ECUs or edge devices with individual customization. ECU is an abbreviation for Electronic Control Unit.
Deduplication technology reduces the amount of data transferred or storage capacity during backup or archiving. It also includes detecting and eliminating data duplication on a file-by-file or block-by-block basis. Duplication can be determined by comparing the data as is, by comparing it in combination with a checksum, or by using a cryptographic hash function.

Patent Document 1 discloses a technique that can effectively eliminate duplication of files that constitute a virtual machine provided to multiple users.

Japanese Patent Application Publication No. 2009-276859

Patent Document 1 discloses a deduplication technique for providing virtual machines to multiple users from a single virtual machine volume image template. In Patent Document 1, reading and writing of each file in an image is traced, and it is determined whether to provide each user's virtual machine with an individual entity or to refer to a common file. In this way, the technique disclosed in Patent Document 1 does not check the case where the same file is used redundantly between different images. Therefore, Patent Document 1 is ineffective against file duplication between different images. Furthermore, by using software containers, file duplication between different images can be alleviated by sharing layers, but there is a problem in that file duplication cannot be prevented when the same file exists in different layers.

The present disclosure aims to prevent file duplication even when the same file exists in different layers.

The deduplication system according to the present disclosure is
An image formed from a plurality of layers and used as a software container, the duplication comprising a device loading an image in which each layer of the plurality of layers includes a plurality of files, and a server device distributing the layer to the device. In the exclusion system,
The server device includes:
a duplication inspection unit that inspects whether a distribution layer that is a layer that is distributed to the device includes a duplicate file that overlaps with a file that exists in the image, based on existing file information that is information on files that exist in the image; and,
a deduplication unit that generates a deduplication layer by removing the duplicate files from the distribution layer and distributes the deduplication layer to the device,
The equipment includes:
The apparatus includes a layer management unit that restores the distribution layer from the deduplication layer.

In the deduplication system according to the present disclosure, the duplication inspection unit, based on existing file information that is information of files that exist in an image, sends duplicates that are duplicates of files that exist in an image to a distribution layer that is a layer that is distributed to devices. Check if the file is included. Then, the deduplication unit generates a deduplication layer by eliminating duplicate files from the distribution layer, and distributes the deduplication layer to the device. Therefore, according to the deduplication system according to the present disclosure, it is possible to prevent file duplication even when the same file exists in different layers.

1 is a diagram illustrating an example of an image used for a software container that is a premise of the first embodiment; FIG. 1 is a diagram showing an example of the overall configuration of a deduplication system according to Embodiment 1. FIG. 1 is a diagram showing a configuration example of a device according to Embodiment 1. FIG. 1 is a diagram illustrating a configuration example of a server device according to Embodiment 1. FIG. FIG. 2 is a flow diagram showing the operation of the deduplication system according to the first embodiment. FIG. 2 is a schematic diagram showing an example of the operation of the deduplication system according to the first embodiment. FIG. 7 is a schematic diagram showing another example of the operation of the deduplication system according to the first embodiment. FIG. 7 is a schematic diagram showing another example of the operation of the deduplication system according to the first embodiment. FIG. 7 is a diagram illustrating a configuration example of a device according to a second modification of the first embodiment. FIG. 7 is a diagram illustrating a configuration example of a server device according to a second modification of the first embodiment. FIG. 3 is a diagram illustrating a configuration example of a device according to a second embodiment. FIG. 3 is a diagram illustrating a configuration example of a server device according to a second embodiment. FIG. 7 is a diagram illustrating a configuration example of an inter-layer duplication table according to Embodiment 2; FIG. 7 is a flow diagram showing the operation of the deduplication system according to the second embodiment. FIG. 3 is a schematic diagram showing an example of the operation of the deduplication system according to the second embodiment.

Hereinafter, this embodiment will be described using figures. In each figure, the same or corresponding parts are given the same reference numerals. In the description of the embodiments, the description of the same or corresponding parts will be omitted or simplified as appropriate. The arrows in the figure mainly indicate the flow of data or processing. Further, in the following figures, the size relationship of each component may differ from the actual one. Further, in the description of the embodiments, directions or positions such as top, bottom, left, right, front, back, front, and back may be indicated. These notations are for convenience of explanation and do not limit the arrangement, direction, or orientation of devices, instruments, parts, or the like.

Embodiment 1.
***Explanation of configuration***
FIG. 1 is a diagram showing an example of an image used for a software container, which is a premise of this embodiment.
The virtualization method using the software container 10 is a method in which a part of the OS of the host is separated, a dedicated area is prepared isolated from other software containers 10, and software is operated on the dedicated area. OS is an abbreviation for Operating System.
The image 11 used for the software container 10 is formed from multiple layers 12. Layers 12 are superimposed to form image 11. Each layer 12 is a collection of multiple files. Further, the layer 12 is given an identifier 121 that uniquely identifies the layer 12. The identifier 121 is, for example, a hash value of the layer 12, and is also called a digest.
Typically, the same layer is shared between software containers 10. Layer 12 is read-only.
The image 11 has a thin read/write layer added by the software container 10 and is used by the software container 10.

FIG. 2 is a diagram showing an example of the overall configuration of the deduplication system 500 according to the present embodiment.
The deduplication system 500 includes a device 100 and a server device 200.
An image 11 is mounted on the device 100. The installed image 11 differs depending on the device.
The server device 200 distributes the image 11 to the device 100 layer by layer when the image 11 is newly added, changed, or updated in the device 100.

FIG. 3 is a diagram showing a configuration example of device 100 according to this embodiment.
FIG. 4 is a diagram showing a configuration example of server device 200 according to this embodiment.

In FIG. 3, device 100 is a computer. Device 100 includes a processor 910 and other hardware such as memory 921, auxiliary storage 922, input interface 930, output interface 940, and communication device 950. Processor 910 is connected to other hardware via signal lines and controls these other hardware.
In FIG. 4, server device 200 is a computer. The server device 200 includes a processor 910 and other hardware such as a memory 921, an auxiliary storage device 922, an input interface 930, an output interface 940, and a communication device 950. Processor 910 is connected to other hardware via signal lines and controls these other hardware.
Note that in FIGS. 3 and 4, the same reference numerals are given to each piece of hardware for the purpose of simplifying the explanation. The device 100 and the server device 200 each include separate hardware.

In FIG. 3, the device 100 includes a layer management section 110 and a storage section 120 as functional elements. The storage unit 120 stores an existing layer 21 and existing file information 22.
The functions of the layer management unit 110 are realized by software. The storage unit 120 is included in the memory 921. Note that the storage unit 120 may be provided in the auxiliary storage device 922 or may be provided in a distributed manner in the memory 921 and the auxiliary storage device 922.

In FIG. 4, the server device 200 includes a duplication checking section 210, a deduplication section 220, and a storage section 230 as functional elements. The storage unit 230 stores a distribution layer 31, a deduplication layer 32, and existing file information 22.
The functions of the duplication checking section 210 and the duplication elimination section 220 are realized by software. The storage unit 230 is included in the memory 921. Note that the storage unit 230 may be provided in the auxiliary storage device 922 or may be provided in a distributed manner in the memory 921 and the auxiliary storage device 922.

In the following hardware description, the server device 200 in FIG. 4 will be used as an example. The same applies to the description of the hardware of the device 100.

The processor 910 is a device that executes a deduplication program. The deduplication program is a program that implements the functions of the server device 200. Specifically, the deduplication program is a program that implements the functions of the duplication checking section 210 and the deduplication section 220. Note that a program that implements the functions of the device 100 will also be referred to as a deduplication program.

The processor 910 is an IC that performs arithmetic processing. Specific examples of the processor 910 are a CPU, a DSP, and a GPU. The IC is an integrated circuit. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. GPU is an abbreviation for Graphics Processing Unit.

Memory 921 is a storage device that temporarily stores data. A specific example of the memory 921 is SRAM or DRAM. SRAM is an abbreviation for Static Random Access Memory. DRAM is an abbreviation for Dynamic Random Access Memory.
Auxiliary storage device 922 is a storage device that stores data. A specific example of the auxiliary storage device 922 is an HDD. Further, the auxiliary storage device 922 may be a portable storage medium such as an SD (registered trademark) memory card, CF, NAND flash, flexible disk, optical disk, compact disc, Blu-ray (registered trademark) disc, or DVD. Note that HDD is an abbreviation for Hard Disk Drive. SD (registered trademark) is an abbreviation for Secure Digital. CF is an abbreviation for CompactFlash®. DVD is an abbreviation for Digital Versatile Disk.

The input interface 930 is a port connected to an input device such as a mouse, keyboard, or touch panel. Input interface 930 is specifically a USB terminal. Note that the input interface 930 may be a port connected to a LAN. USB is an abbreviation for Universal Serial Bus. LAN is an abbreviation for Local Area Network.

The output interface 940 is a port to which a cable of an output device such as a display is connected. Specifically, the output interface 940 is a USB terminal or an HDMI (registered trademark) terminal. The display is specifically an LCD. Output interface 940 is also referred to as a display interface. HDMI (registered trademark) is an abbreviation for High Definition Multimedia Interface. LCD is an abbreviation for Liquid Crystal Display. Although one output interface 940 is illustrated in FIG. 1, multiple output interfaces 940 may exist.

The communication device 950 has a receiver and a transmitter. The communication device 950 is connected to a communication network such as a LAN, the Internet, WiFi (registered trademark), or a telephone line. Communication device 950 is specifically a communication chip or NIC. NIC is an abbreviation for Network Interface Card.

The deduplication program is executed on the server device 200. The deduplication program is loaded into and executed by processor 910. The memory 921 stores not only the deduplication program but also the OS. OS is an abbreviation for Operating System. Processor 910 executes the deduplication program while executing the OS. The deduplication program and the OS may be stored in the auxiliary storage device 922. The deduplication program and OS stored in auxiliary storage device 922 are loaded into memory 921 and executed by processor 910. Note that part or all of the deduplication program may be incorporated into the OS.

The device 100 may include multiple processors that replace the processor 910. These multiple processors share the execution of the deduplication program. Each processor, like processor 910, is a device that executes a deduplication program.

Data, information, signal values, and variable values used, processed, or output by the deduplication program are stored in memory 921, auxiliary storage device 922, or registers or cache memory within processor 910.

The "section" of each section of the duplication checking section 210 and the duplication elimination section 220 may be read as "circuit", "process", "procedure", "processing", or "circuitry". The deduplication program causes a computer to execute a duplication check process and a deduplication process. "Processing" in the duplication checking process and deduplication process should be read as "program," "program product," "computer-readable storage medium that stores the program," or "computer-readable storage medium that stores the program." Good too.

The deduplication method is performed by each device of the deduplication system 500 executing a deduplication program.
The deduplication program may be provided stored in a computer-readable recording medium. Further, the deduplication program may be provided as a program product.

***Operation explanation***
Next, the operation of the deduplication system 500 according to this embodiment will be explained. The operating procedure of the deduplication system 500 corresponds to a deduplication method. Further, a program that realizes the operation of the deduplication system 500 corresponds to a deduplication program.

FIG. 5 is a flow diagram showing the operation of the deduplication system 500 according to this embodiment.
FIG. 6 is a schematic diagram showing an example of the operation of the deduplication system 500 according to this embodiment.
FIG. 7 is a schematic diagram showing another example of the operation of the deduplication system 500 according to the present embodiment.
FIG. 8 is a schematic diagram showing another example of the operation of the deduplication system 500 according to this embodiment.

In FIG. 5, the existing layer 21 is a layer that already exists in the device 100. Specifically, each of all layers forming the image 11 mounted on the device 100 is referred to as an existing layer 21.
In the example of FIG. 6, layer A and layer B are existing layers 21. Layer A includes file 1 and file 2, and layer B includes file 3 and file 4.
In addition, in the schematic diagrams of FIGS. 6 to 8, illustration of the image 11 mounted on the device 100 is omitted to simplify the explanation.

In FIG. 5, the existing file information 22 is a list of all files that already exist in the device 100. Specifically, it is a list of files included in all existing layers 21 that form the image 11 installed in the device 100.
In the example of FIG. 6, layer A and layer B are existing layers 21. Therefore, the existing file information 22 lists file 1, file 2, file 3, and file 4.

The layer management unit 110 manages layers existing in the device 100.
Existing file information 22 is generated by layer management section 110 and transmitted to server device 200.
The server device 200 stores the existing file information 22 sent from the device 100 in the storage unit 230.

In FIG. 5, the distribution layer 31 is a layer that the server device 200 distributes to the device 100. In other words, the distribution layer 31 is one of the layers forming the image 11 that the server device 200 is trying to distribute to the device 100.
In the example of FIG. 6, layer C is the distribution layer 31. Layer C is a layer that includes file 1 and file 5.

<Duplicate inspection process>
In step S101, the duplication checking unit 210 of the server device 200 determines whether the distribution layer 31 contains a duplicate file 321 that duplicates a file that already exists in the image 11 installed in the device 100, based on the existing file information 22. Inspect. The duplication inspection unit 210 automatically inspects the existing file information 22 and the distribution layer 31 for duplicate files 321 .
Specifically, the duplication checking unit 210 determines whether the distribution layer 31 includes a file listed in the existing file information 22. If the distribution layer 31 includes a file listed in the existing file information 22, the duplication checking unit 210 determines the file as a duplicate file 321.

This will be explained using a specific example shown in FIG.
The duplication checking unit 210 determines whether layer C, which is the distribution layer 31, includes a file listed in the existing file information 22. Layer C includes file 1 listed in existing file information 22. Therefore, the duplication checking unit 210 determines the file 1 as a duplicate file 321.

<Duplication processing>
In step S102, the deduplication unit 220 generates the deduplication layer 32 by eliminating the duplicate files 321 from the distribution layer 31. The deduplication unit 220 automatically eliminates duplicate files 321 from the distribution layer 31 based on the result of the duplication check. Note that eliminating the duplicate file 321 from the distribution layer 31 means deleting the duplicate file 321 from the distribution layer 31.
Then, in step S103, the deduplication unit 220 delivers the deduplication layer 32 to the device 100 via the communication device 950. Note that the deduplication layer 32 is given information on which files are excluded.

This will be explained using a specific example shown in FIG.
The deduplication unit 220 generates a deduplication layer 32 by eliminating file 1, which is the duplicate file 321, from layer C, which is the distribution layer 31. Then, the deduplication unit 220 delivers the deduplication layer 32, which excludes the file 1 from the layer C, to the device 100 via the communication device 950.

<Layer management processing>
In step S104, the layer management unit 110 of the device 100 restores the distribution layer 31 from the deduplication layer 32. The layer management unit 110 restores the distribution layer 31 using the existing layer 21 and the deduplication layer 32.
The layer management unit 110 can restore the distribution layer 31 in the following three ways.

<<Example 1 of layer management processing for restoring the distribution layer 31>
The layer management unit 110 restores the distribution layer by referring to the duplicate file 321 included in any of the existing layers 21 from the deduplication layer 32. Any of the existing layers 21 is any of a plurality of layers that form the image 11 mounted on the device 100.

For example, the layer management unit 110 manages layer management information in which the correspondence between the existing layer 21 existing in the device 100 and the files included in the existing layer 21 and the reference relationship between the existing layers are set.
The layer management unit 110 automatically replaces the duplicate file 321 eliminated by the deduplication layer 32 with a reference to the existing layer 21 in the layer management information.

In the specific example of FIG. 6, the layer management unit 110 replaces file 1 excluded in layer C with a reference to file 1 in layer A, which is the existing layer 21, in the layer management information.

According to the first example of layer management processing, since duplicate files 321 are not transferred, the amount of transfer can be reduced. Furthermore, since duplicate files 321 can be shared between different layers, the storage area in the device 100 can be reduced.

<<Example 2 of layer management processing for restoring the distribution layer 31>
The layer management unit 110 restores the distribution layer 31 by copying the duplicate file 321 included in any of the existing layers 21 to the deduplication layer 32. In this manner, by copying the duplicate file 321 to the deduplication layer 32, the distribution layer 31 is completely restored in the device 100.

In the specific example of FIG. 7, the layer management unit 110 copies file 1 included in layer A, which is the existing layer 21, to layer C, which is the deduplication layer 32. As a result, the distribution layer 31 is restored.

According to the second example of layer management processing, since duplicate files 321 are not transferred, the amount of transfer can be reduced. Furthermore, since the duplicate file 321 can be copied to the deduplication layer 32 after transfer and the distribution layer 31 can be completely restored, processing such as changing file references can be reduced.

<<Example 3 of layer management processing for restoring distribution layer 31>
The layer management unit 110 restores the distribution layer 31 by moving the duplicate file 321 included in any of the existing layers 21 to the deduplication layer 32. Then, the layer management unit 110 causes the duplicate file 321 included in the distribution layer 31 to be referenced from the layer from which the duplicate file 321 is moved.
That is, the layer management unit 110 eliminates the duplicate file 321 from the existing layer 21 and automatically adds the duplicate file 321 to the deduplication layer 32. Then, the layer management unit 110 sets the layer management information so that the distribution layer 31 restored by automatically adding the duplicate file 321 is referred to from the existing layer 21 from which the duplicate file 321 is moved.

In the specific example of FIG. 8, the server device 200 distributes layer A' to the device 100 in order to update layer A in the device 100 to layer A'. Layer A' is delivered to the device 100 as a deduplication layer 32 in which file 1, which is a duplicate file 321, is eliminated.
The layer management unit 110 moves file 1 included in layer A to layer A′, which is the deduplication layer 32. As a result, layer A', which is the distribution layer 31, is restored. File 1 is excluded from layer A. The layer management unit 110 sets the layer management information so that the file 1 included in the layer A' is referred to from the layer A from which the file 1 is moved.

According to the third example of layer management processing, since the duplicate file 321 is not transferred, the amount of transfer can be reduced. Further, after the transfer, the duplicate file 321 is moved to the deduplication layer 32, and the distribution layer 31 restored from the source layer is referred to. Therefore, even if an old layer is deleted after the movement source layer is updated to the distribution layer 31, there is an advantage that the reference relationship does not need to be changed.
Old layers may be retained during update for rollback or B/G update. However, the old layer is likely to be deleted in the future. If files of other images refer to files of the old layer that remain temporarily, the scope of influence when deleting the old layer increases. Therefore, according to the third example of layer management processing, the range of influence on the device 100 can be reduced by performing an operation such that the file of the new layer remains as an entity when updating.

***Other configurations***
<Modification 1>
In this embodiment, existing file information 22 is generated by layer management section 110 and sent to server device 200. However, the server device 200 may create the existing file information 22.
Since the server device 200 distributes the image 11 layer by layer to the device 100, the correspondence between layers and files and which layer was distributed to which device 100 are recorded. Therefore, the server device 200 can create existing file information 22, which is information about files existing in the device 100, for each device.

<Modification 2>
In this embodiment, the functions of each device in the deduplication system 500 are realized by software. As a modification, the functions of each device of the deduplication system 500 may be implemented in hardware.
Specifically, each device of the deduplication system 500 includes an electronic circuit 909 in place of the processor 910.

FIG. 9 is a diagram illustrating a configuration example of a device 100 according to a second modification of the present embodiment.
FIG. 10 is a diagram illustrating a configuration example of a server device 200 according to a second modification of the present embodiment.
In the following description, the server device 200 in FIG. 10 will be used as an example. The same applies to the description of the device 100.

The electronic circuit 909 is a dedicated electronic circuit that realizes the functions of the duplication checking section 210 and the duplication elimination section 220. Electronic circuit 909 is specifically a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA, an ASIC, or an FPGA. GA is an abbreviation for Gate Array. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field-Programmable Gate Array.

The functions of the duplication checking section 210 and the deduplication section 220 may be realized by one electronic circuit, or may be realized by being distributed among multiple electronic circuits.

As another modification, part of the functions of the duplication checking section 210 and the deduplication section 220 may be realized by an electronic circuit, and the remaining functions may be realized by software. Furthermore, some or all of the functions of the duplication checking section 210 and the duplication elimination section 220 may be realized by firmware.

Each of the processor and electronic circuit is also referred to as processing circuitry. In other words, the functions of each device in the deduplication system 500 are realized by processing circuitry.

***Explanation of effects of this embodiment***
As described above, the duplication elimination system 500 according to the present embodiment can eliminate duplication of the same file in different layers.
In the deduplication system 500 according to this embodiment, since layers are transferred after eliminating duplicate files, the amount of transfer can be reduced.

Furthermore, according to the first example of layer management processing in the deduplication system 500 according to the present embodiment, duplicate files can be shared between different layers on the same device, so the storage area in the device can be reduced. .

Furthermore, according to the second example of layer management processing in the deduplication system 500 according to the present embodiment, it is possible to copy duplicate files to the deduplication layer after transfer and completely restore the distribution layer. Therefore, processing such as changing file references can be reduced, making it easier to handle images in devices.

Furthermore, according to example 3 of layer management processing in the deduplication system 500 according to the present embodiment, the duplicate file is moved to the deduplication layer after transfer, and the distribution layer restored from the source layer is referred to. Therefore, even if the old layer is deleted after the source layer is updated to the distribution layer, there is no need to change the reference relationship.

Embodiment 2.
In this embodiment, points different from or added to Embodiment 1 will be explained.
In the first embodiment, all existing files on the device are checked for duplication. On the other hand, in this embodiment, an aspect will be described in which the unit of duplicate inspection is performed for each combination of layers.

FIG. 11 is a diagram showing a configuration example of device 100 according to this embodiment.
FIG. 12 is a diagram showing a configuration example of server device 200 according to this embodiment.
In this embodiment, device 100 and server device 200 include existing layer information 23 in addition to the configuration described in Embodiment 1. Furthermore, the server device 200 further includes an inter-layer duplication table 24.
Note that in this embodiment, the existing file information 22 may not exist in the device 100 and the server device 200.

The existing layer information 23 is a list of layers that already exist in the device 100. Specifically, the existing layer information 23 is a list of identifiers that identify layers existing in the image 11 installed in the device 100. This identifier is called a digest and is a hash value for each layer present in the image 11. For example, the existing layer information 23 may be generated by the layer management unit 110 and transmitted to the server device 200. Alternatively, the existing layer information 23 may be generated on the server device 200 side.

FIG. 13 is a diagram showing a configuration example of the inter-layer duplication table 24 according to this embodiment.
The inter-layer duplication table 24 is a table that records the duplication relationship of files for each combination of layers that have been delivered to the device 100.
For example, in FIG. 13, the combination of layer A and layer D indicates that file 1 is duplicated. Furthermore, the combination of layer B and layer D indicates that file 1 and file 3 overlap.

FIG. 14 is a flow diagram showing the operation of the deduplication system 500 according to this embodiment.
FIG. 15 is a schematic diagram showing an example of the operation of the deduplication system 500 according to this embodiment.

In step S<b>101 a , the duplication checking unit 210 determines whether the distribution layer 31 is recorded in the inter-layer duplication table 24 based on the inter-layer duplication table 24 .
If the distribution layer 31 is recorded in the inter-layer duplication table, the duplication checking unit 210 checks whether there is a duplicate file in the distribution layer 31 based on the inter-layer duplication table 24. Note that the duplication checking unit 210 may check whether there is a duplicate file in the distribution layer 31 based on the existing layer information 23 and the inter-layer duplication table 24.

If the distribution layer 31 is not recorded in the inter-layer duplication table, the duplication checking unit 210 checks whether there is a duplicate file for each combination of the existing layer 21 and the distribution layer 31 based on the existing layer information 23. Since the server device 200 distributes the layers themselves, it has information on which files are included in which layers. Therefore, the duplication checking unit 210 can check whether there is a duplicate file for each combination of the existing layer 21 and the distribution layer 31 based only on the existing layer information 23.
In step S<b>101 b , the duplication checking unit 210 records, in the inter-layer duplication table 24 , a check result as to whether there is a duplicate file for each combination of the existing layer 21 and the distribution layer 31 .
The processing after step S102 is the same as in the first embodiment.

***Explanation of effects of this embodiment***
As described above, according to the deduplication system 500 according to the present embodiment, file duplication inspection can be performed using the inter-layer duplication table, so that the processing load of duplication inspection can be reduced.

In the first and second embodiments above, each part of each device of the deduplication system has been described as an independent functional block. However, the configuration of each device in the deduplication system may not be the same as in the embodiments described above. The functional blocks of each device in the deduplication system may have any configuration as long as they can implement the functions described in the above embodiments. Further, each device in the deduplication system may not be one device, but may be a system composed of a plurality of devices.
Further, a plurality of parts of Embodiments 1 and 2 may be combined and implemented. Alternatively, one part of these embodiments may be implemented. In addition, these embodiments may be implemented in any combination, in whole or in part.
That is, in Embodiments 1 and 2, it is possible to freely combine each embodiment, to modify any component of each embodiment, or to omit any component in each embodiment.

Note that the embodiments described above are essentially preferable examples, and are not intended to limit the scope of the present disclosure, the scope of applications of the present disclosure, and the scope of uses of the present disclosure. The embodiments described above can be modified in various ways as necessary. For example, the procedures described using flow diagrams or sequence diagrams may be modified as appropriate.

10 Software container, 11 Image, 12 Layer, 121 Identifier, 21 Existing layer, 22 Existing file information, 23 Existing layer information, 24 Inter-layer duplication table, 31 Distribution layer, 32 Deduplication layer, 321 Duplicate file, 100 Device, 110 Layer management unit, 120, 230 storage unit, 200 server device, 210 duplication inspection unit, 220 deduplication unit, 500 deduplication system, 909 electronic circuit, 910 processor, 921 memory, 922 auxiliary storage device, 930 input interface, 940 output Interface, 950 communication device.

Claims

An image formed from a plurality of layers and used as a software container, the duplication comprising a device loading an image in which each layer of the plurality of layers includes a plurality of files, and a server device distributing the layer to the device. In the exclusion system,
The server device includes:
a duplication inspection unit that inspects whether a distribution layer that is a layer that is distributed to the device includes a duplicate file that overlaps with a file that exists in the image, based on existing file information that is information on files that exist in the image; and,
a deduplication unit that generates a deduplication layer by removing the duplicate files from the distribution layer and distributes the deduplication layer to the device,
The equipment includes:
A deduplication system comprising a layer management unit that restores the distribution layer from the deduplication layer.
The layer management section includes:
The deduplication system according to claim 1, wherein the distribution layer is restored by referencing the duplicate file included in any of the plurality of layers forming the image from the deduplication layer.
The layer management section includes:
The deduplication system according to claim 1, wherein the distribution layer is restored by copying the duplicate file included in any of the plurality of layers forming the image to the deduplication layer.
The layer management section includes:
The distribution layer is restored by moving the duplicate file included in any of the plurality of layers forming the image to the deduplication layer, and the duplicate file included in the distribution layer is moved to the deduplication layer. The deduplication system according to claim 1, wherein the deduplication system is referred to from a source layer.
The deduplication system includes:
comprising an inter-layer duplication table in which file duplication relationships for each combination of layers already distributed to the device are recorded;
The duplication inspection department is
It is determined whether the distribution layer is recorded in the inter-layer duplication table based on the inter-layer duplication table, and if the distribution layer is recorded in the inter-layer duplication table, the distribution layer is determined based on the inter-layer duplication table. The deduplication system according to any one of claims 1 to 4, wherein the deduplication system checks whether there are duplicate files in the distribution layer.
The duplication inspection department is
If the distribution layer is not recorded in the inter-layer duplication table, check whether there is a duplicate file for each combination of the existing layer and the distribution layer, based on existing layer information that is information on layers existing in the image. 6. The deduplication system according to claim 5, wherein the deduplication system examines and records in the inter-layer duplication table.
The existing layer information is
a list of identifiers identifying layers present in the image;
7. The deduplication system according to claim 6, wherein the identifier is a hash value of each layer present in the image.
A device installed with an image formed from a plurality of layers and used as a software container, each layer of the plurality of layers including a plurality of files, the device communicating with a server device that distributes the layer to the device. In equipment that
The server device includes:
a duplication inspection unit that inspects whether a distribution layer that is a layer that is distributed to the device includes a duplicate file that overlaps with a file that exists in the image, based on existing file information that is information on files that exist in the image; and,
a deduplication unit that generates a deduplication layer by removing the duplicate files from the distribution layer and distributes the deduplication layer to the device,
The equipment includes:
A device comprising a layer management unit that restores the distribution layer from the deduplication layer.
In a server device that distributes a layer to a device equipped with an image formed from a plurality of layers and used as a software container, each layer of the plurality of layers including a plurality of files,
The server device includes:
a duplication inspection unit that inspects whether a distribution layer that is a layer that is distributed to the device includes a duplicate file that overlaps with a file that exists in the image, based on existing file information that is information on files that exist in the image; and,
A server device comprising: a deduplication unit that generates a deduplication layer by eliminating the duplicate files from the distribution layer, and distributes the deduplication layer to the device.
An image formed from a plurality of layers and used as a software container, the duplication comprising a device loading an image in which each layer of the plurality of layers includes a plurality of files, and a server device distributing the layer to the device. In the deduplication method used in the deduplication system,
The server device includes:
Based on existing file information that is information on files that exist in the image, check whether a distribution layer that is a layer that is distributed to the device includes a duplicate file that overlaps with a file that exists in the image;
generating a deduplication layer by removing the duplicate files from the distribution layer, delivering the deduplication layer to the device;
The equipment includes:
A deduplication method for restoring the distribution layer from the deduplication layer.
An image formed from a plurality of layers and used as a software container, the duplication comprising a device loading an image in which each layer of the plurality of layers includes a plurality of files, and a server device distributing the layer to the device. In the deduplication program used in the elimination system,
Duplication inspection processing that checks whether a distribution layer that is a layer that is distributed to the device contains a duplicate file that overlaps with a file that exists in the image, based on existing file information that is information on files that exist in the image. and,
a deduplication process that generates a deduplication layer by removing the duplicate files from the distribution layer, and distributes the deduplication layer to the device;
A deduplication program that causes a computer to execute layer management processing for restoring the distribution layer from the deduplication layer distributed to the device.