CN112860335B - Private warehouse Docker mirror image information acquisition system and acquisition method thereof - Google Patents
Private warehouse Docker mirror image information acquisition system and acquisition method thereof Download PDFInfo
- Publication number
- CN112860335B CN112860335B CN202110099167.8A CN202110099167A CN112860335B CN 112860335 B CN112860335 B CN 112860335B CN 202110099167 A CN202110099167 A CN 202110099167A CN 112860335 B CN112860335 B CN 112860335B
- Authority
- CN
- China
- Prior art keywords
- mirror image
- acquisition
- docker
- task
- private
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 19
- 238000003860 storage Methods 0.000 claims abstract description 25
- 230000006835 compression Effects 0.000 claims abstract description 13
- 238000007906 compression Methods 0.000 claims abstract description 13
- 230000006870 function Effects 0.000 claims abstract description 13
- 238000004458 analytical method Methods 0.000 claims description 20
- 238000004891 communication Methods 0.000 claims description 15
- 238000004140 cleaning Methods 0.000 claims description 9
- 230000003993 interaction Effects 0.000 claims description 4
- 239000010410 layer Substances 0.000 description 75
- 239000003795 chemical substances by application Substances 0.000 description 33
- 230000004048 modification Effects 0.000 description 7
- 238000012986 modification Methods 0.000 description 7
- 230000008859 change Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 230000002776 aggregation Effects 0.000 description 4
- 238000004220 aggregation Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 239000002356 single layer Substances 0.000 description 3
- 238000010191 image analysis Methods 0.000 description 2
- 238000004806 packaging method and process Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 244000035744 Hura crepitans Species 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000002347 injection Methods 0.000 description 1
- 239000007924 injection Substances 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 239000000243 solution Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44505—Configuring for program initiating, e.g. using registry, configuration files
- G06F9/4451—User profiles; Roaming
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/10—Protecting distributed programs or content, e.g. vending or licensing of copyrighted material ; Digital rights management [DRM]
- G06F21/107—License processing; Key processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
- G06F21/45—Structures or tools for the administration of authentication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45562—Creating, deleting, cloning virtual machine instances
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45595—Network integration; Enabling network access in virtual machine instances
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/2141—Access rights, e.g. capability lists, access control lists, access tables, access matrices
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Multimedia (AREA)
- Technology Law (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The system comprises a Docker mirror image acquisition center and a private warehouse server; the Docker mirror image acquisition center comprises a configuration module, an acquisition task module, a storage module and a network module; the Docker mirror image acquisition center sets an acquisition agent on a private warehouse server for acquiring mirror images; the collection agent is used for creating a new image with collection function through compiling the file and creating a container based on the new image. The application also provides a method for collecting the image information of the private warehouse Docker, which comprises the following steps: the acquisition agent adds a mirror image layer for information acquisition to the mirror image to be acquired to construct a new mirror image: the added mirror image layer has the functions of compressing mirror image contents into an acquisition folder and sending a mirror image file compression packet in the acquisition folder to a Docker mirror image acquisition center; and establishing a container based on the new mirror image as a running basis according to the new mirror image after the mirror image layer is added. The method and the device improve the operation efficiency of the system.
Description
Technical Field
The application relates to the technical field of private warehouse Docker mirror image information vulnerability scanning, in particular to a private warehouse Docker mirror image information acquisition system and an acquisition method thereof.
Background
Along with the high-speed development of virtualization technology, container technology gradually becomes a mainstream deployment mode of various industries and businesses, but the packaging and overlapping layer storage modes of the container technology increase the difficulty of information acquisition and deep analysis of the container technology. Packaging injection of irregular use, open source software vulnerabilities, malicious software injection and the like of developers in the mirror image causes a plurality of potential safety hazards in a container environment and a business environment based on the mirror image. At the same time, the management and maintenance of container images by container image warehouse servers has become a de facto standard in the industry, and image collection and analysis for them has become an important direction in the industry.
In the face of such a situation, the information collection and analysis manner for the remote mirror on the remote mirror repository server is generally implemented by means of hierarchical parsing and merging analysis. If the open source software Clair analyzes the container mirror image in a layered mode, the mirror image layer address and the association relation between the mirror image layer address and the parent layer are required to be transmitted when the interface is called, so as to analyze and scan, and when the result is returned, the result of aggregation analysis of the open source software Clair and the parent layer chain is returned according to the layer. However, in such a manner, manual analysis is required during layer analysis, automation cannot be achieved, and coverage problems of the same results among layers cannot be eliminated during layer analysis result aggregation, namely, the problem of the parent layer is solved by the upper layer and cannot be reflected in the overall result, and Clair requires centralized storage and analysis mirroring, so that a large amount of calculation and storage performance are consumed in an analysis center.
Patent CN109918911a provides a mirror information collection method for a remote repository, in which a mirror is read from the remote repository and downloaded and scanned layer by layer, and such a method also scans in layer units and performs aggregate feedback on the relationship between layers, however, the invention does not mention the processing method of coverage content at the time of layer association. In general, the analysis results of the mirror image as a whole are more focused in the mirror image analysis, the invention also carries out aggregation analysis according to the single-layer analysis results, extra analysis and resource consumption are brought in the relation of the presentation of the scanning analysis results, and coverage detection is possible.
In summary, most of the existing mirror image scanning modes use layer scanning and provide the aggregation scanning result of the mirror image layer, so that extra calculation consumption and resource consumption are brought, a small number of scanning modes using mirror images as units need to be adapted to different superposition storage engines, and information mirror image information is restored by an external construction superposition storage mode, so that the mode brings extra adaptation and more fault points, possibly brings stability influence to the original system and service, and has certain defects.
Disclosure of Invention
The application provides a private warehouse Docker mirror image information acquisition system, which comprises a Docker mirror image acquisition center and a private warehouse server; the private warehouse server comprises a mirror image warehouse, and the mirror image warehouse consists of a Docker mirror image;
the Docker mirror image acquisition center comprises a configuration module, an acquisition task module, a storage module and a network module;
the configuration module comprises a network communication address for configuring the Docker mirror image acquisition center and authority authentication information for configuring a private warehouse server;
the acquisition task module is used for acquiring a mirror image list in the private warehouse server, creating an acquisition task according to the acquired mirror image list, issuing the acquisition task to the private warehouse server, and collecting an acquisition result after the private warehouse server finishes the acquisition task; the mirror image list at least comprises a mirror image warehouse name, a mirror image label and a mirror image ID;
the storage module is used for providing a system bottom layer storage to support the whole system;
the network module is used for providing network communication and realizing remote network interaction;
the Docker mirror image acquisition center sets an acquisition agent on a private warehouse server for acquiring mirror images; the acquisition agent can acquire information of a configuration module of the Docker mirror image acquisition center; acquiring authority of reading, writing and calling a local private warehouse server by an acquisition agent through acquiring authority authentication information of the private warehouse server; the acquisition agent can send acquired mirror image information to the Docker mirror image acquisition center through the obtained network communication address of the Docker mirror image acquisition center;
the acquisition agent comprises a local acquisition module and a remote task execution module; the local acquisition module is used for acquiring a mirror list to be acquired on the local private warehouse server; the remote task execution module is used for executing the acquisition task issued by the Docker mirror image acquisition center, creating a new mirror image with an acquisition function through compiling a file and creating a container based on the new mirror image.
The remote task execution module at least comprises a task analysis unit, a task execution unit and a resource cleaning unit; the task analysis unit is used for analyzing the acquisition tasks issued by the Docker mirror image acquisition center; the task execution unit is used for logically organizing and executing the analyzed acquisition task; the resource cleaning unit is used for cleaning various resources after the collection task is finished, and comprises the steps of collecting container operation results and container operation logs, deleting containers and mirror image files.
The network communication address of the Docker mirror image acquisition center comprises an IP address and a service port.
The authority authentication information of the private warehouse server comprises a private warehouse server type, a private warehouse server authentication type and private warehouse server authentication information; the private warehouse server authentication type comprises, but is not limited to, an account password mode and a Token mode, and the corresponding private warehouse server authentication information comprises a user name, a password and a Token.
The Docker mirror image acquisition center and the acquisition agent acquire a mirror image list in the private warehouse server through a container instruction interface or through a private warehouse server API.
The application also provides a Docker mirror image information acquisition method using the private warehouse Docker mirror image information acquisition system, which comprises the following steps:
s10, configuring the network communication address of the Docker mirror image acquisition center and the authority authentication information of the private warehouse server;
s20, a Docker mirror image acquisition center acquires a mirror image list in a remote private warehouse server; the mirror image list at least comprises a mirror image warehouse name, a mirror image label and a mirror image ID;
s30, the Docker mirror image acquisition center establishes an acquisition task according to the acquired mirror image list, and an acquisition agent is arranged on a private warehouse server where the mirror image to be acquired is located; sending the acquisition task to a corresponding acquisition agent; the acquisition task at least comprises an acquisition task number and a mirror image to be acquired;
s40, acquiring authority authentication information of a local private warehouse server of a Docker mirror image acquisition center by an acquisition agent, acquiring a mirror image list to be acquired on the local private warehouse server by the acquisition agent, and adding a mirror image layer for information acquisition for the mirror image to be acquired to construct a new mirror image: the added mirror image layer has the function of compressing mirror image contents into an acquisition folder and sending a mirror image file compression packet in the acquisition folder to a Docker mirror image acquisition center; according to the new mirror image added with the mirror image layer, establishing a container based on the new mirror image;
s50, starting a container in a safe environment, and simultaneously providing a network communication address of a Docker mirror image acquisition center, wherein the container automatically sends a mirror image file compression packet in an acquisition folder to the Docker mirror image acquisition center;
s60, the collection agent continuously manages the state of the container, and recovers resources when the mirror image collection task is finished; when all the acquisition tasks are completed, the acquisition agent feeds back the results of the acquisition tasks to the Docker mirror image acquisition center.
Wherein, in step S20, step S21 is further included, when the acquisition task is established, when the mirror image of the same ID is already in the acquisition task list, the acquisition task of the mirror image is skipped.
The Docker image acquisition center and the acquisition agent acquire a private warehouse Docker image list by using a Docker API or a container instruction.
In step S40, a new image layer is added by using a Dockerfile compiled file as a base image, and an acquisition folder and a compression tool folder are added under the same directory of the base image; the added mirror image layer has the functions of placing the compressed package formed by compressing the basic mirror image into the acquisition folder and sending the compressed package of the mirror image file in the acquisition folder to the Docker mirror image acquisition center.
The acquisition tasks sent by the Docker mirror image acquisition center to the acquisition agent and the acquisition tasks fed back by the acquisition agent to the Docker mirror image acquisition center are both expressed by JSON.
The beneficial effects realized by the application are as follows:
the invention performs information acquisition and scanning based on the mirror image as a unit, thereby reducing storage consumption caused by layered acquisition, and simultaneously, the repeated scanning consumption of files with the same file name in different layers can be removed by performing information acquisition based on the mirror image as a unit, thereby improving the scanning efficiency and reducing the calculation consumption in scanning. In addition, the invention constructs a new mirror image by adding a new mirror image layer with an acquisition function to the mirror image based on the characteristics of the container, and automatically completes the acquisition by constructing a container corresponding to the new mirror image. Meanwhile, the specific execution of the acquisition tasks is carried out through the acquisition agent, so that the mirror image pulling and mirror image analysis tasks which are uniformly executed by the original acquisition center can be distributed on each private warehouse server host, the calculation and storage pressure of the analysis center is reduced, and the overall efficiency and concurrent processing capacity are improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings may be obtained according to these drawings to those skilled in the art.
Fig. 1a and 1b are logical structure diagrams of mirror layer files when contents of a Docker mirror image are changed.
FIG. 1c is a logical block diagram of a container when it is running and when it reads the image as a whole.
Fig. 2 is a logic structure diagram of the private warehouse Docker mirror image information acquisition system of the present application.
Fig. 3 is a flowchart of steps of a method for collecting Docker mirror image information in a private warehouse of the present application.
Fig. 4 is a network distribution structure diagram of the private warehouse Docker mirror image information acquisition system of the present application.
Detailed Description
The following description of the embodiments of the present application will be made with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the present application. All other embodiments, based on the embodiments herein, which a person skilled in the art would obtain without making any inventive effort, are within the scope of the protection herein.
Docker is an open-source application container engine, which allows the developer to package applications and rely on packages to a portable container, then release them to any popular Linux or Windows machine, and also implement virtualization, where the containers are completely sandboxed without any interfaces between them.
A complete Docker consists of the following parts: client (Client), daemon (Daemon), mirror (Image), and Container (Container).
The Docker mirror image is similar to the mirror image in the virtual machine, is a read-only template facing the Docker engine and comprises a file system, any application program running needs an environment, the mirror image is used for providing the running environment, the Docker mirror image is a multi-layer file structure, each layer is called a mirror image layer (Layers), the multi-layer structure of the Docker mirror image can be regarded as a unified file system (joint file system), and the Docker process considers that the whole file system is mounted in a read-write mode. All Docker images start at one base image layer, the parent layer. When a mirror is modified or new content is added, a new mirror layer is created over all current mirror layers. In other words, when a change is generated, the original image file belonging to the lower layer will not change, all the changes will occur in the top layer, and if the contents of one image layer are modified and replaced, the contents will not occur in the image layer, but will occur in the upper image layer.
The Docker container is an application instance of the Docker mirror image, and can be created, started, stopped and deleted, and the containers are isolated from each other and are not affected by each other. The Docker container is similar to a lightweight sandbox and can be viewed as a very simple Linux system environment (including root rights, process space, user space, and web space, etc.), as well as applications running therein.
When a Docker container is started from the Docker mirror image, the Docker creates a writable layer on all mirror image layers, and the Docker mirror image is unchanged, so that if the container is to be deleted, the writable layer of the container is deleted directly.
The Docker image warehouse (reproducibility) is a place for storing the Docker images, and the registration server is a specific server for storing the image warehouse; the registration server may have a plurality of image repositories, each of which may have a plurality of images; the mirror repository corresponds to a specific item or directory having its own access address, e.g. mirror repository address d1. Dickerpal.
The Docker mirror warehouse is divided into a public warehouse and a private warehouse according to the access authority type. The public repository is open allowing all users to download images with hidden names. The private warehouse is invisible to other users, the mirror image can be downloaded only by logging in the authorized account, the server storing the private warehouse is the private server, and common private warehouse servers are generally Registry and Harbor.
As a set of images, an image repository (redundancy) is generally used to place the same image to which different versions are applied in the same repository, where the repository name is an image name, the version number is represented by a Tag number (Tag), and the image also has its own unique identification ID. Based on the warehouse name, tag and ID, a mirror image can be specifically located (separation is adopted between warehouse name and tag).
Dockerfile is a text file used to construct the image, and the text contains instructions and instructions needed to construct the image. For example, the Dockerfile starts with the FROM command, followed by various methods, commands and parameters, and may use the docker build instruction to create a mirror image, using the docker run launch container.
The API (Application Programming Interface ) is a number of predefined functions that provide the application and developer the ability to access a set of routines based on certain software or hardware without having to access source code or understand the details of the internal operating mechanisms. The Docker API is the interface that dockers call to an application.
When the Docker API is used, the system recognizes the local host container environment and adapts to different container versions to carry out API call to acquire information; when using container instructions, the information is obtained using the Docker system instructions.
The communication mode for acquiring the interaction information by the remote server adopts JSON (JavaScript Object Notation, JS object numbered musical notation) which is a lightweight data exchange format. Based on a subset of ECMAScript (js specification formulated by European computer Association), the method stores and represents data in a text format completely independent of programming language, has a simple and clear hierarchical structure, is easy to read and write by people, is easy to analyze and generate by machines, and effectively improves network transmission efficiency.
The concept of hierarchical storage used by Docker mirroring does not change each layer before the current layer, in other words, the result of any modification is only marking, adding, modifying at the current layer, and not changing other layers below the current layer, as shown in fig. 1a and 1b, the first layer mirror layer (Lay 1) of the mirror Image includes file 1, file 2 and file 3, the second layer (Lay 2) above the first layer includes file 4, file 5 and file 6, the third layer (Lay 3) above the second layer makes a change to file 5, and file 7 replaces file 5 with the layer file of the upper layer, and since the mirror is a modification to the layer file of the lower layer, file 5 in the second layer is not directly replaced on the second layer, but instead makes a change to replace on the third layer.
After the container corresponding to the dock mirror image is started, the content of the mirror image can be read in a single-layer mode with the help of the dock service, as shown in fig. 1c, after the container is started, although the layer file of the dock mirror image is not changed in a basic storage mode, after the container is operated, from the perspective of a user, the dock reads the mirror image as a whole, that is, the multi-layer structure of the mirror image is logically combined into a layer, the combined state on the logic structure is shown in fig. 1c, and the final state of the original mirror image file after multi-layer change is obtained.
According to the technology, the application provides a private warehouse Docker mirror image information acquisition system, which comprises a Docker mirror image acquisition center and a private warehouse server;
the Docker mirror image acquisition center is provided with an acquisition agent on a private warehouse server;
the Docker mirror image acquisition center comprises a configuration module, an acquisition task module, a storage module and a network module;
the configuration module is used for configuring the network address and the service port of the Docker mirror image acquisition center and configuring the basic information of the remote private warehouse server; the collection agent can send the compressed image file to the Docker image collection center through the network address and the service port of the Docker image collection center.
The basic information of the private warehouse server comprises a private warehouse server type, a private warehouse server authentication type and private warehouse server authentication information.
The acquisition task module is used for acquiring a mirror image list in the private warehouse server, creating an acquisition task according to the acquired mirror image list, issuing the acquisition task to the private warehouse server, and collecting an acquisition result after the private warehouse server finishes the acquisition task; the mirror image list at least comprises a mirror image warehouse name, a mirror image label and a mirror image ID;
the storage module is used for providing system bottom storage to support the whole system, comprises file storage, data storage and storage monitoring capability, and can be particularly used for mirror image information storage, acquisition task storage, acquisition result storage, other storage and other functions.
The network module is used for managing the system network, providing basic network communication capability and enabling the system to realize network interconnection and interaction for a remote place.
The Docker mirror image acquisition center sets an acquisition agent on a private warehouse server for acquiring mirror images; the acquisition agent can acquire information of a configuration module of the Docker mirror image acquisition center; acquiring authority of reading, writing and calling a local private warehouse server by an acquisition agent through acquiring authority authentication information of the private warehouse server; the acquisition agent can send acquired mirror image information to the Docker mirror image acquisition center through the obtained network communication address of the Docker mirror image acquisition center;
the acquisition agent comprises a local acquisition module and a remote task execution module; the local acquisition module is used for acquiring a mirror list of the local private warehouse server; the remote task execution module is used for executing the acquisition task issued by the Docker mirror image acquisition center, creating a new mirror image with an acquisition function through compiling a file and creating a container based on the new mirror image.
The remote task execution module at least comprises a task analysis unit, a task execution unit and a resource cleaning unit.
The task analysis unit is used for analyzing the acquisition task issued by the Docker mirror image acquisition center.
The task execution unit is used for logically organizing the analyzed acquisition task and executing the acquisition task.
The resource cleaning unit is used for cleaning various resources after the collection task is finished, and comprises the steps of collecting the operation result of the collection container, collecting the operation log of the container, deleting the container and deleting the mirror image file.
The application also provides a method for acquiring the Docker mirror image information of the private warehouse, which comprises the following steps:
configuring a network address and a service port of a Docker mirror image acquisition center, and configuring basic information of a remote private warehouse server;
the basic information of the private warehouse server comprises a private warehouse server type, a private warehouse server authentication type and private warehouse server authentication information.
The private warehouse server authentication type comprises, but is not limited to, an account password mode and a Token mode, and the corresponding private warehouse server authentication information comprises a user name, a password and a Token.
The Docker mirror image acquisition center acquires mirror image lists in each remote private warehouse server;
the mirror image list comprises basic information of all mirror images on the mirror image warehouse server, or basic information of specific mirror images is downloaded according to the requirement; the image basic information at least comprises an image ID, a warehouse (reproducibility) and a Tag (Tag);
the mode that the Docker mirror image acquisition center acquires the mirror image list in each remote private warehouse server can be acquired through a container instruction interface or through a private warehouse server API.
For example, in a specific operation, the Docker mirror image collection center obtains a mirror image repository on a Registry private repository server through an API interface of the private repository server, where the specific interface is as follows:
GET/v2/_catalog
the mirror image tag and the mirror image ID are obtained through the following interfaces:
GET/v2/<name>/tags/list
after receiving the instruction, the Registry private warehouse server sends the mirror list to the Docker mirror image acquisition center in a JSON mode.
After the Docker mirror image acquisition center obtains the mirror image list, an acquisition task is established, and an acquisition agent is deployed on a private warehouse server host machine for acquiring the mirror image; the Docker mirror image acquisition center transmits basic information of a local private warehouse server to an acquisition agent and transmits an acquisition task to a corresponding acquisition agent;
the acquisition tasks sent to the acquisition agent by the Docker mirror image acquisition center at least comprise an acquisition task number and a mirror image to be acquired;
for example, in a specific implementation, the acquisition tasks sent by the Docker mirror acquisition center to the acquisition agent are represented in JSON form as follows:
that is, the task number 231 sent by the Docker image collection center to the collection agent has two images to be collected, namely a mirror image with warehouse name mysql, label 5.3, ID e4247c08758ef42f3f7d1079d20718eea c414015a86950d748745a60ad73fd4, and a mirror image with warehouse name python, label 3.7.1-alpine, ID 020295c920c635bbb25e4c73e026834e1bbfc5225955d0ecd63016c5d78bc0 ca=mirror image.
Wherein, since the images can be uniquely distinguished by the image ID of the images themselves, the images with the same ID need only be collected once, and when the images with the same ID have been collected, the collection task of the images is not needed.
The acquisition agent adds a mirror image layer for information acquisition for the mirror image to be acquired in the acquisition task to construct a new mirror image:
the acquisition agent acquires the mirror image required to be acquired on the local private warehouse server according to the acquisition task, wherein the acquisition mode can use a mirror image warehouse API interface mode or can use a container instruction to acquire.
The image layer for information acquisition is added for the image to be acquired to construct a new image, and a basic image compiling file needs to be constructed, in this embodiment, taking mysql:5.3 image as an example, the basic Dockerfile is as follows:
FROM mysql:5.3
RUN mkdir-p/scan
RUN mkdir-p/scantools
COPY/tools//scantools
RUN/scantools/tar-zcvf../scan/scaninfo_mysql_5.3.tar.gz`find/-path /scantools-path'/scan'-o-path/tar-a-prune-maxdepth 1-o-print|sed 1d`scan scantools
CMD[“/scantools/push$CIP$CPORT../scan/scaninfo_mysql_5.3.tar.gz”]
docker build-t mysql:5.3-scan.
docker run-itd-e CIP＝10.10.10.10-e CPORT＝8080--name
mysql:2.175.3-scan-container mysql:2.175.3-scan
in the Dockerfile file, mysql:5.3 is used as a basic mirror image, and an acquisition folder scan and a compression tool folder scanols are added under the same directory of the basic mirror image; an image layer is added on a basic image by using an instruction, the added image layer has the function of compressing the file content of the basic image through a compression tool in a scanools to form an image file compression packet scaninfo_mysql_5.3.Tar.gz, the scaninfo_mysql_5.3.Tar.gz is put into an acquisition folder scan, the image file compression packet scaninfo_mysql_5.3.Tar.gz in the acquisition folder is sent to a Docker image acquisition center, and a new image after the image layer is added is compiled according to the Docker image, so that a mysql 5.3-scan container mysql is built on the basis of the new image; the container is started and simultaneously provides a network ip address 10.10.10.10 and a service port 8080 of the Docker mirror image acquisition center, and can automatically send a mirror image file compression packet scaninfo_mysql_5.3.Tar.gz in an acquisition file to the Docker mirror image acquisition center through the network address and the service port.
The new mirror container also executes all instruction operations of the original mirror container, so that the new mirror container needs to be operated in a safe environment in order to ensure operation safety.
When the new mirror image is successfully constructed, the docker returns the integral id of one mirror image, and when the container is operated, all the mirror images can be read and written as a whole in a mode of reading and writing the container by the id, and the bottom layer of the whole is pointed to multi-layer content.
In the Dockerfile command statement, the CMD statement is executed last, that is, after the container is run, the compressed image file is sent to the Docker image collection center. After the container is started, the multi-layer image file form of the image file compression package can be obtained and read in a single-layer structure, so that the image file after being combined in a logic structure is obtained, and vulnerability scanning is carried out on the image file, namely scanning of the images combined into a final state of one layer, wherein layering analysis is not needed, a plurality of engines are not needed, and vulnerability scanning is directly carried out.
The collection agent continuously manages the state of the container based on the new mirror image, and recovers resources when the mirror image collection task is finished;
after the container is constructed and started, the container is continuously monitored, when the container stops running, the container log and the stopping result are recorded, the container is deleted, the mirror image is deleted, and the resource is recovered.
When all the acquisition tasks are completed, the acquisition agent feeds back the results of the acquisition tasks to the Docker mirror image acquisition center,
the acquisition task results include an acquisition task ID and an acquisition result mirrored in each acquisition task, expressed in this embodiment as follows in JSON.
And the private warehouse container mirror image information acquisition center receives the acquisition task result and then ends the acquisition task.
While preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the application. It will be apparent to those skilled in the art that various modifications and variations can be made in the present application without departing from the spirit or scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims and the equivalents thereof, the present application is intended to cover such modifications and variations.
Claims (10)
1. The system comprises a Docker mirror image acquisition center and a private warehouse server; the private warehouse server comprises a mirror image warehouse, wherein the mirror image warehouse consists of a Docker mirror image;
the Docker mirror image acquisition center comprises a configuration module, an acquisition task module, a storage module and a network module;
the configuration module comprises a network communication address for configuring the Docker mirror image acquisition center and authority authentication information for configuring a private warehouse server;
the acquisition task module is used for acquiring a mirror image list in the private warehouse server, creating an acquisition task according to the acquired mirror image list, issuing the acquisition task to the private warehouse server, and collecting an acquisition result after the private warehouse server finishes the acquisition task; the mirror image list at least comprises a mirror image warehouse name, a mirror image label and a mirror image ID;
the storage module is used for providing a system bottom layer storage to support the whole system;
the network module is used for providing network communication and realizing remote network interaction;
the Docker mirror image acquisition center sets an acquisition agent on a private warehouse server for acquiring mirror images; the acquisition agent can acquire information of a configuration module of the Docker mirror image acquisition center; acquiring authority of reading, writing and calling a local private warehouse server by an acquisition agent through acquiring authority authentication information of the private warehouse server; the acquisition agent can send acquired mirror image information to the Docker mirror image acquisition center through the obtained network communication address of the Docker mirror image acquisition center;
the acquisition agent comprises a local acquisition module and a remote task execution module; the local acquisition module is used for acquiring a mirror list to be acquired on the local private warehouse server; the remote task execution module is used for executing the acquisition task issued by the Docker mirror image acquisition center, creating a new mirror image with an acquisition function through compiling a file and creating a container based on the new mirror image.
2. The private warehouse Docker mirror image information collection system as claimed in claim 1, wherein the remote task execution module comprises at least a task parsing unit, a task execution unit and a resource cleaning unit; the task analysis unit is used for analyzing the acquisition task issued by the Docker mirror image acquisition center; the task execution unit is used for logically organizing and executing the analyzed acquisition task; the resource cleaning unit is used for cleaning various resources after the collection task is finished, and comprises the steps of collecting container running results and container running logs, deleting containers and mirror image files.
3. The private warehouse Docker mirror image information collection system as claimed in claim 1, wherein the Docker mirror image collection center's network communication address includes an IP address and a service port.
4. The private repository Docker mirror image information collection system of claim 1, wherein the authority authentication information of the private repository server includes a private repository server type, a private repository server authentication type, and private repository server authentication information; the private warehouse server authentication types include, but are not limited to, an account password mode and a Token mode, and the corresponding private warehouse server authentication information includes a user name, a password and a Token.
5. The private repository Docker mirror image information collection system of claim 1, wherein the Docker mirror image collection center and collection agent obtain the mirror image list in the private repository server through a container instruction interface or through a private repository server API.
6. A dock mirror image information collection method using the private warehouse dock mirror image information collection system according to any one of claims 1 to 5, comprising the steps of:
s10, configuring the network communication address of the Docker mirror image acquisition center and the authority authentication information of the private warehouse server;
s20, a Docker mirror image acquisition center acquires a mirror image list in a remote private warehouse server; the mirror image list at least comprises a mirror image warehouse name, a mirror image label and a mirror image ID;
s30, the Docker mirror image acquisition center establishes an acquisition task according to the acquired mirror image list, and an acquisition agent is arranged on a private warehouse server where the mirror image to be acquired is located; sending the acquisition task to a corresponding acquisition agent; the acquisition task at least comprises an acquisition task number and a mirror image to be acquired;
s40, acquiring authority authentication information of a local private warehouse server of a Docker mirror image acquisition center by an acquisition agent, acquiring a mirror image list to be acquired on the local private warehouse server by the acquisition agent, and adding a mirror image layer for information acquisition for the mirror image to be acquired to construct a new mirror image: the added mirror image layer has the functions of compressing mirror image contents into an acquisition folder and sending a mirror image file compression packet in the acquisition folder to a Docker mirror image acquisition center; according to the new mirror image added with the mirror image layer, establishing a container based on the new mirror image;
s50, starting a container in a safe environment, and simultaneously providing a network communication address of a Docker mirror image acquisition center, wherein the container automatically sends a mirror image file compression packet in an acquisition folder to the Docker mirror image acquisition center;
s60, the collection agent continuously manages the state of the container, and recovers resources when the mirror image collection task is finished; when all the acquisition tasks are completed, the acquisition agent feeds back the results of the acquisition tasks to the Docker mirror image acquisition center.
7. The Docker image information collecting method according to claim 6, wherein in step S20, further comprising step S21, when the collecting task is established, when the image of the same ID is already in the collecting task list, skipping the collecting task of the image.
8. The Docker mirror image information collection method of claim 6, wherein the Docker mirror image collection center and the collection agent use a Docker API or container instructions to obtain a private repository Docker mirror image list.
9. The dock mirror image information collection method according to claim 6, wherein in step S40, a new mirror image layer is added using the dock mirror image compiled file as a base mirror image, and a collection folder and a compression tool folder are added under the same directory of the base mirror image; the added image layer has the functions of placing the compressed package formed by compressing the basic image into an acquisition folder and sending the image file compressed package in the acquisition folder to a Docker image acquisition center.
10. The method for acquiring the Docker mirror image information according to claim 6, wherein the acquisition task sent by the Docker mirror image acquisition center to the acquisition agent and the result of the acquisition task fed back by the acquisition agent to the Docker mirror image acquisition center are both represented by JSON.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110099167.8A CN112860335B (en) | 2021-01-25 | 2021-01-25 | Private warehouse Docker mirror image information acquisition system and acquisition method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110099167.8A CN112860335B (en) | 2021-01-25 | 2021-01-25 | Private warehouse Docker mirror image information acquisition system and acquisition method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112860335A CN112860335A (en) | 2021-05-28 |
| CN112860335B true CN112860335B (en) | 2024-02-20 |
Family
ID=76008737
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110099167.8A Active CN112860335B (en) | 2021-01-25 | 2021-01-25 | Private warehouse Docker mirror image information acquisition system and acquisition method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112860335B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11409787B2 (en) * | 2020-11-10 | 2022-08-09 | Nexcom International Co., Ltd. | Method for executing Docker image under protection |
| CN113918096B (en) * | 2021-10-21 | 2023-09-22 | 城云科技（中国）有限公司 | Method, device and application for uploading algorithm mirror image package |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107066310A (en) * | 2017-03-11 | 2017-08-18 | 郑州云海信息技术有限公司 | It is a kind of to build and using the method and device in the privately owned warehouses of safe Docker |
| CN107239688A (en) * | 2017-06-30 | 2017-10-10 | 平安科技（深圳）有限公司 | The purview certification method and system in Docker mirror images warehouse |
| CN108616419A (en) * | 2018-03-30 | 2018-10-02 | 武汉虹旭信息技术有限责任公司 | A kind of packet capture analysis system and its method based on Docker |
| CN109981351A (en) * | 2019-03-06 | 2019-07-05 | 浪潮通用软件有限公司 | A kind of private clound dispositions method |
-
2021
- 2021-01-25 CN CN202110099167.8A patent/CN112860335B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107066310A (en) * | 2017-03-11 | 2017-08-18 | 郑州云海信息技术有限公司 | It is a kind of to build and using the method and device in the privately owned warehouses of safe Docker |
| CN107239688A (en) * | 2017-06-30 | 2017-10-10 | 平安科技（深圳）有限公司 | The purview certification method and system in Docker mirror images warehouse |
| CN108616419A (en) * | 2018-03-30 | 2018-10-02 | 武汉虹旭信息技术有限责任公司 | A kind of packet capture analysis system and its method based on Docker |
| CN109981351A (en) * | 2019-03-06 | 2019-07-05 | 浪潮通用软件有限公司 | A kind of private clound dispositions method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112860335A (en) | 2021-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Taibi et al. | Architectural patterns for microservices: a systematic mapping study | |
| US20210311858A1 (en) | System and method for providing a test manager for use with a mainframe rehosting platform | |
| US11099823B2 (en) | Systems and methods for transformation of reporting schema | |
| US8108456B2 (en) | Method and apparatus for migrating the system environment on which the applications depend | |
| Murthy et al. | Apache Hadoop YARN: moving beyond MapReduce and batch processing with Apache Hadoop 2 | |
| Taibi et al. | Continuous architecting with microservices and devops: A systematic mapping study | |
| US8645326B2 (en) | System to plan, execute, store and query automation tests | |
| CN112860335B (en) | Private warehouse Docker mirror image information acquisition system and acquisition method thereof | |
| US8140578B2 (en) | Multilevel hierarchical associations between entities in a knowledge system | |
| US20030028579A1 (en) | Process for component-based application development | |
| CN1866214B (en) | Installation method and apparatus | |
| US8214809B2 (en) | Grid-enabled ANT compatible with both stand-alone and grid-based computing systems | |
| Lampa et al. | SciPipe: A workflow library for agile development of complex and dynamic bioinformatics pipelines | |
| US10585785B2 (en) | Preservation of modifications after overlay removal from a container | |
| Van Nieuwpoort et al. | User-friendly and reliable grid computing based on imperfect middleware | |
| JP7231518B2 (en) | Packaging support system and packaging support method | |
| JP5745932B2 (en) | Method, program, and system for reflecting operation on object which is image of mapping in graph data | |
| CN110991984A (en) | Digital operation analysis platform and method based on enterprise information heterogeneous system | |
| Zhao et al. | Notebook‐as‐a‐VRE (NaaVRE): From private notebooks to a collaborative cloud virtual research environment | |
| Hanjura | Heroku cloud application development | |
| WO2002069141A1 (en) | Method and apparatus creation and performance of service engagement modeling | |
| WO2010064317A2 (en) | Operation management support program, recording medium on which said program is recorded, operation management support device, and operation management support method | |
| CN112860481A (en) | Local Docker mirror image information acquisition system and acquisition method thereof | |
| Mattmann et al. | Revisiting the Anatomy and Physiology of the Grid | |
| Madushan | Cloud Native Applications with Ballerina: A guide for programmers interested in developing cloud native applications using Ballerina Swan Lake |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |