CN107809467B - Method for deleting container mirror image data in cloud environment - Google Patents

Abstract

The invention discloses a method for deleting container mirror image data in a cloud environment, which aims at solving the problems that in the practice of container technology, due to the fact that the space occupation of a mirror image disk is overlarge, a disk and network I/O (input/output) overhead is generated in the releasing process, the deployment cost is increased, and the use flexibility of a mirror image is limited. The method is applicable to two scenarios: local storage and mirror export. When the local storage is carried out, the disk storage overhead during the mirror image storage is reduced by increasing the multiplexing rate of the local basic mirror image; when the mirror image is exported, a file export model is established through files accessed in the running process of the dynamic collection container, and the exported mirror image is constructed as required, so that the size of the exported mirror image is reduced, and the functional completeness of the exported mirror image is ensured.

Description

A method for deleting container image data in cloud environment

technical field

The invention relates to the technical field of cloud computing containers, in particular to a method for deleting container image data in a cloud environment, and more particularly to an image size optimization method for locally storing and exporting Docker container images.

Background technique

Container technology is an operating environment isolation technology similar to the sandbox mechanism. Users can create and run operating systems in containers to achieve operating system-level virtualization. Compared with traditional virtual machines, container technology achieves lightweight application isolation by sharing kernel resources. Docker is an implementation form of container technology, featuring high portability and integration of development, operation and maintenance.

Today, with the ever-expanding scale of cloud computing and big data, enterprises are increasingly demanding continuous product integration and efficient release. In the traditional virtualization system with virtual machines as the core, although the isolation of applications and services can be achieved, it needs to provide completely exclusive hardware resources, which makes the resource overhead of the entire system very large. As a lightweight virtualization concept, Docker can reduce resource and time overhead compared to virtual machines, making the packaging, release and coordination of applications and services more flexible and fast.

The image in Docker consists of a series of image layers, each layer only contains the incremental part of the previous layer, forming a stack structure. When a container is successfully created, a read-write container layer is created on top of the original image layer. The operations on the container, such as creating and deleting files, are all completed in the readable and writable container layer, and will not affect the read-only image layer. Multiple containers can share the data of the same image and also save their own data status information. The storage of container image files is in the form of tiered storage. In this way, the same part of the image can be reused, which saves the overhead of disk space and reduces the cost of transferring the container image over the network, but it will complicate the runtime of the container. For Docker images, the main structures and contents are packaged in different layers, so that the contents of the same layer can be shared and storage space can be saved.

In practice, because the image includes the entire operating environment, the image size is often large, which will generate a large disk and network I/O overhead in the process of publishing, and also limit the flexibility of the use of the image. The streamlined and convenient design of container technology runs counter to the original intention, and even increases the difficulty of deploying the entire system. Therefore, it is particularly important to delete too large and redundant image packages.

SUMMARY OF THE INVENTION

The main purpose of the present invention is to solve the above-mentioned defects in the prior art, and propose a method for deleting container image data in a cloud environment.

The purpose of the present invention can be achieved by adopting the following technical solutions:

A method for pruning container image data in a cloud environment, characterized in that the pruning method is applicable to a local image storage mode and an image export mode, and includes the following steps respectively:

Local image storage mode:

T1. Run the local image analyzer to retrieve the storage status of the local image. If the image is not stored locally, execute step T2, and if the image already exists locally, execute step T3.

T2. At this time, only analyze the newly imported image locally, and save the size of the newly imported image, the size of the basic image layer, and the SHA-256 digest value of each layer of the image to the local.

T3. If there are existing images locally, check the number of locally saved images. If there are more than 20 images, the local image analyzer will check the base images of all images, and select the one with the largest ratio through the calculation method of sharing the base image layer. As a shared base image layer, and save the absolute path, size and MD5 digest value of the files contained in it to form a base image file fingerprint library.

T4. After the shared base image layer is selected, for all new images added in the future, the SHA-256 digest value of the base image layer will be analyzed first, and compared with the SHA-256 digest value of the shared base image layer, if they are consistent Then, it can be directly stored locally without modification; if it is inconsistent, step T5 is executed.

T5. The local storage module will analyze the newly added image, obtain the MD5 digest value of the file contained in it, and compare it with the digest value in the file fingerprint database to remove all duplicates. Rebuilds a new image using the selected shared base image and the remainder.

Image export mode:

R1. When an image needs to be exported, execute the on-demand dynamic export method of the image, locate the file access information table of the image according to the name of the image to be exported, and execute step R3 if the target file access information table can be located, otherwise, execute step R2;

R2. Collect file access information in the image. When generating the container, import a file access probe to collect the files accessed by the container in real time during the running process, record it in text form, and make a file access information table, which is in step R3. Provide basis for image export;

R3. Read the file access information table, obtain the file access information table of the exported image, establish a prediction model of the image exported file, and then obtain the relevant files that are depended on when the image is running, and export these files to make a new image.

Further, the local image analyzer will collect all locally stored image information, including the size of each image, the number of layers of each image, and the sharing of each layer between the images, and calculates the sharing of Reduced disk overhead by the base image layer.

Further, the described basic image layer sharing calculation method is to calculate the ratio of the storage overhead reduced by sharing the basic image layer to the size of the total image after calculating each basic image that shares the local storage, and select the one with the largest ratio. The shared base image layer used when the base image is used as local storage.

Further, the file fingerprint library includes absolute paths, sizes and MD5 digest values of all files included in the selected base image layer to be shared.

Further, the content of the file access information table includes the name of the access file, the MD5 digest value, the size, the type, the absolute path and the process of accessing the file.

Further, the file access probe is to collect and execute the process, related configuration files, and executable programs that depend on the file in real time during the running of the container, and after the running of the container is completed, the file accessed during the running is processed. The absolute path, size, name, and MD5 digest value are written into the file access information table.

Further, the described export file prediction model is used to obtain all files included in the export image, by obtaining the absolute path and the number of visits of each file in the file access information table, the number, type and number of files accessed under each directory. The number of visits to a certain file, calculate the dependency of the image on the file, and obtain the export probability value of the directory.

Compared with the prior art, the present invention has the following advantages and effects:

(1) The present invention can effectively reduce the disk space occupied by the mirror image when it is stored locally, and also greatly reduces the size of the exported mirror image, which is convenient for the release and transplantation of the mirror image.

(2) While reducing the size of the image, the present invention ensures that the deleted image has better reliability and maintainability through the establishment of the export model, so that the image can still ensure good re-editability after being packaged and released. .

(3) The present invention can support all file drivers of the existing Docker, by firstly containerizing the local image, and then exporting after the deletion processing, so it does not depend on a specific file driver.

(4) The data acquisition of the present invention is based on the network communication with Docker, and the operation of the mirror image is also separated from the original code, and the original code is not modified, which ensures the stability of the original system.

Description of drawings

Fig. 1 is the structural representation of the applicable system of the present invention;

Fig. 2 is a working flow chart of a method for deleting container image data in a cloud environment disclosed by the present invention;

Fig. 3 is a schematic diagram of basic image sharing effect under local storage mode in the present invention;

FIG. 4 is a data flow diagram of a file access probe in the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Example

As shown in Figure 1, it is a schematic diagram of the system structure of a method for deleting container image data in a cloud environment, which is applied to the scale optimization of container images in a single-machine environment:

In this environment, the local image data pruning includes locally stored images and exported images. The purpose of the present invention is, in the local environment, by increasing the multiplexing rate of the image layer shared between the images, so that more upper-layer images can reuse the same basic image, thereby reducing the overall storage overhead of the image locally; In the export image environment, through the obtained file access information, the generated deleted image only contains the files that the image function depends on, thereby reducing the size of the exported image.

In order to clarify the application scenario of the present invention more clearly, the following details are combined with the system work flow diagram (Fig. 2), the schematic diagram of the basic image sharing effect in the local storage mode (Fig. 3), and the file access probe data flow diagram (Fig. 4). analyze.

As shown in Figure 2, a method for reducing container image data in a cloud environment, its application scenarios include local storage mode and image export mode.

The image deletion method in the local storage mode specifically includes the following steps:

T1. When the program is running, the local image storage is retrieved. If the image is not stored locally, step T2 is performed. If the image is already stored locally, step T3 is performed;

T2. At this time, only analyze the newly imported local image, and save the newly imported image size, the size of the basic image layer, and the SHA-256 digest value of each layer of the image to the local;

T3. If there are existing images locally, check the number of locally saved images. If there are more than 20 images, check the base images of all images, calculate the ratio of the shared base images to the total image size, and select the one with the largest ratio. As a shared basic image layer, and save the file name and file MD5 digest value contained in it to form a basic image file fingerprint library;

T4. After the shared base image layer is selected, for all new images added in the future, the SHA-256 digest value of the base image layer will be analyzed first, and compared with the SHA-256 digest value of the shared base image layer, if they are consistent Then it can be directly stored locally without modification; if it is inconsistent, step T5 is performed;

T5. The local storage module will analyze the newly added image, obtain the MD5 digest value of the file contained in it, and compare it with the digest value in the file fingerprint database to remove all duplicates;

T6. Regenerate a new image using the selected shared base image and the remainder after culling in step T5. Therefore, the basic image part of the image does not occupy additional local disk space, so as to reduce the local storage overhead.

The image deletion method in the image export mode specifically includes the following steps:

R1. When an image needs to be exported, execute the on-demand dynamic export method of the image, locate the file access information table of the image according to the name of the image to be exported, and execute step R3 if the target file access information table can be located, otherwise, execute step R2;

R2. Collect file access information in the image. When generating the container, import a file access probe to collect the files accessed by the container in real time during the running process, record it in text form, and make a file access information table, which is in step R3. Provide basis for image export;

R3. Read the file access information table, obtain the file access information table of the exported image, establish a prediction model of the image exported file, and then obtain the relevant files that are depended on when the image is running, and export these files to make a new image.

The local image analyzer is a process running on the local machine that communicates with Docker. Its main function is to obtain the size of each image currently stored in the local machine, the number of layers of each image, and the sharing of each layer between images. Happening. In the local storage mode, the image layer information obtained by the local image analyzer is used to share the computing method through the base image layer, so as to achieve the effect of sharing the base image layer.

则通过共享基础镜像层所带来的空间减少率则可定义为公式1：In the shared computing method of the basic image layer, the present invention reduces the total overhead of the global image storage by increasing the utilization rate of the basic image. To describe this problem, the definition is as follows: the virtual size of the mirror (without considering the space reduced by layer sharing) is V, then the virtual storage cost of the local n mirrors is

Then the space reduction rate brought by sharing the base image layer can be defined as Equation 1:

Among them, S represents the storage size of the shared layer, and L represents the number of times the layer is shared. A larger value of η indicates that the sharing of this layer has a more significant effect on improving the global storage cost. It can also be seen from Equation 1 that the larger the shared base image, the larger η. Generally, in order to ensure the versatility of the basic image, it is acceptable to select an image with a relatively large space for local storage. Therefore, the locally shared base image can be selected according to the value of η.

As shown in Figure 3, it is the basic image sharing effect diagram achieved in the local storage mode. In the figure, each circle is a layer of the mirror, and the circle from the far left to the shaded circle is the fully functional part of the mirror. In fact, the size of each layer of each image varies widely, with the largest layer in each image accounting for 67% of the total image size on average. As can be seen from the figure, by sharing the same image layer, the sharing ratio of the local base image layer can be increased, and the effect of multiple images sharing the same base image layer can be achieved, thereby reducing the disk space occupied by the image locally as a whole. size.

File fingerprint library, which contains the absolute path, size and MD5 digest value of all files contained in the base image layer selected to be shared. Its function is to eliminate the duplication of the shared base image layer in the newly imported image, thereby reducing the disk space occupied by local storage.

A file access probe is an independent executable file, that is, a container is run from the image to be modified, and the probe is executed in the container to capture the process executed in the container and the accessed data in real time. document. After the container runs, the probe will summarize the related files accessed during the container running, and write the file access information table in JSON format and save it in the shared data volume. The specific implementation is completed through the mutual communication between four concurrent processes and threads, namely User, Sensor, Monitor and Collector. User is a process that sends start or end commands to the probe, usually running in the host, and communicates with the Sensor through unix socket; Sensor is the probe process that captures file access information in the container, and its main function is to receive the capture of the Monitor. The report comes and reports these reports to the User process; the Monitor thread is responsible for summarizing the events (even) collected from the Collector thread, and organizes them into reports and submits them to the Sensor.

Figure 4 is the data flow diagram of the file access probe. There are three data flows in the whole system: the stop flow, which is sent by the User. The collection of the entire probe information is completed, and the probe work is stopped. After the Sensor receives the stop signal, it will be transmitted. For the Monitor, the Monitor performs the cleanup work (cleanup); the report stream, which is issued by the Monitor, is the Monitor execution event sorting function (ProcessEven), which converts the original access information collected from the Collector into a report structure, which is sent back to the Sensor, and then Sent by Sensor to User; even stream, sent by Collector, is when Collector executes the event capture function (GetEven) during container operation to collect original file access information in real time.

The file access information table is the file access information recorded in JSON format generated by the probe in the container. Each record takes the accessed file name as the primary key, and the corresponding attributes include the absolute path of the file, the size of the file, the MD5 digest value of the file, the type of the file and the process ID of the file that is accessed. If the file type is a symbolic link, try to solve the problem. reference, giving the actual path to the file.

Exporting the file prediction model, the present invention uses a full probability model to determine whether a certain directory is all exported. The basis of its design is that under the standard design, the files in each directory are a collection of functions that implement a certain function. When many files in a directory are exported, it is likely to indicate that the files in this directory are implemented. The core of the mirroring function, so all files in this directory should also be exported. The specific description is as follows: Define the set of dependent files required to export the mirroring as X, and its capacity is n. It is assumed that the influence of a certain file X _i , (i∈[1,n]) on whether the upper directory is exported is equally possible, that is, 1/n. For any directory in the file system, include zero or more files and subdirectories. Therefore, the derived probability for any directory Y can be described as:

为第i个文件或目录

的导出概率。由于集合X内的文件一定会被导出，因此对于任何导出文件的

公式(2)可进一步写成：where m is the total number of files and subdirectories under directory Y,

for the ith file or directory

The derived probability of . Since files in collection X are bound to be exported, for any exported file

Formula (2) can be further written as:

Where k is the number of files in directory Y, and l is the number of subdirectories. We give a threshold ε, 0≤ε≤1, if P(Y)>ε, export all files in the Y directory.

The above-mentioned embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited by the above-mentioned embodiments, and any other changes, modifications, substitutions, combinations, The simplification should be equivalent replacement manners, which are all included in the protection scope of the present invention.

Claims (7)

1. a method for pruning container image data under cloud environment, it is characterized in that, described method for pruning comprises local image storage mode and image export mode, wherein, The described local image storage mode includes the following steps: T1. Run the local image analyzer to retrieve the storage situation of the local image. If the image is not stored locally, then execute step T2, and if the image already exists locally, execute step T3; T2. At this time, only analyze the newly imported local image, and save the newly imported image size, the size of the basic image layer, and the SHA-256 digest value of each layer of the image to the local; T3. If there are existing images locally, check the number of locally saved images. If there are more than 20 images, the local image analyzer will check the base images of all images, and select the one with the largest ratio through the calculation method of sharing the base image layer. As a shared basic image layer, and save the absolute path, size and MD5 digest value of the files contained in it to form a basic image file fingerprint library; T4. After the shared base image layer is selected, for all new images added in the future, the SHA-256 digest value of the base image layer will be analyzed first, and compared with the SHA-256 digest value of the shared base image layer, if they are consistent Then it can be directly stored locally without modification; if it is inconsistent, step T5 is performed; T5. The local storage module will analyze the newly added image, obtain the MD5 digest value of the file contained in it, and compare it with the digest value in the file fingerprint database, remove all duplicate parts, and use the selected shared base image and the remainder to rebuild a new image; The image export mode includes the following steps: R1. When an image needs to be exported, execute the on-demand dynamic export method of the image, locate the file access information table of the image according to the name of the image to be exported, and execute step R3 if the target file access information table can be located, otherwise, execute step R2; R2. Collect file access information in the image. When generating the container, import a file access probe to collect the files accessed by the container in real time during the running process, record it in text form, and make a file access information table, which is in step R3. Provide basis for image export; R3. Read the file access information table, obtain the file access information table of the exported image, establish a prediction model of the image exported file, and then obtain the relevant files that are depended on when the image is running, and export these files to make a new image. 2. The method for deleting container image data in a cloud environment according to claim 1, wherein the local image analyzer will collect all locally stored image information, including the information of each image. size, the number of layers in each image, and the sharing of layers between images, and calculate the disk overhead reduction by sharing the underlying image layer. 3. The method for deleting container image data in a cloud environment according to claim 1, wherein the shared computing method for the basic image layer is that after computing each basic image of the shared local storage, the The ratio of the storage overhead reduced by the shared base image layer to the size of the total image, and the base image with the largest proportion is selected as the shared base image layer used for local storage. 4. the deletion method of container image data under a kind of cloud environment according to claim 1, is characterized in that, described file fingerprint library, comprises all files contained in the base image layer selected to be shared. Absolute path, size and MD5 digest values. 5. the deletion method of container image data under a kind of cloud environment according to claim 1, is characterized in that, the content of described file access information table comprises the name, MD5 digest value, size, type, absolute value of access file The path and the process accessing the file. 6. The method for deleting container image data in a cloud environment according to claim 1, wherein the file access probe is to collect and execute processes, related configuration files, The executable program depends on the file, and after the container runs, the absolute path, size, name, and MD5 digest value of the file accessed during the running are written into the file access information table. 7. The method for deleting container image data in a cloud environment according to claim 1, wherein the export file prediction model is used to obtain all files included in the export image, and the access information table is obtained by obtaining the file. The absolute path and access times of each file in each directory, the number and type of accessed files in each directory, and the access times of a certain file, calculate the dependency of the image on the file, and obtain the export probability value of the directory.

Images