CN112860335A - Docker mirror image information acquisition system and method for private warehouse - Google Patents
Docker mirror image information acquisition system and method for private warehouse Download PDFInfo
- Publication number
- CN112860335A CN112860335A CN202110099167.8A CN202110099167A CN112860335A CN 112860335 A CN112860335 A CN 112860335A CN 202110099167 A CN202110099167 A CN 202110099167A CN 112860335 A CN112860335 A CN 112860335A
- Authority
- CN
- China
- Prior art keywords
- mirror image
- acquisition
- docker
- private warehouse
- collection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 20
- 238000003860 storage Methods 0.000 claims abstract description 26
- 230000006835 compression Effects 0.000 claims abstract description 13
- 238000007906 compression Methods 0.000 claims abstract description 13
- 230000006870 function Effects 0.000 claims abstract description 13
- 238000004458 analytical method Methods 0.000 claims description 18
- 238000004891 communication Methods 0.000 claims description 15
- 238000004140 cleaning Methods 0.000 claims description 9
- 238000012217 deletion Methods 0.000 claims description 4
- 230000037430 deletion Effects 0.000 claims description 4
- 230000003993 interaction Effects 0.000 claims description 3
- 239000010410 layer Substances 0.000 description 71
- 239000003795 chemical substances by application Substances 0.000 description 34
- 238000012986 modification Methods 0.000 description 8
- 230000004048 modification Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 239000002356 single layer Substances 0.000 description 3
- 244000035744 Hura crepitans Species 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 238000012856 packing Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000010076 replication Effects 0.000 description 2
- 239000000243 solution Substances 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000002347 injection Methods 0.000 description 1
- 239000007924 injection Substances 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000003245 working effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44505—Configuring for program initiating, e.g. using registry, configuration files
- G06F9/4451—User profiles; Roaming
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/10—Protecting distributed programs or content, e.g. vending or licensing of copyrighted material ; Digital rights management [DRM]
- G06F21/107—License processing; Key processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
- G06F21/45—Structures or tools for the administration of authentication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45562—Creating, deleting, cloning virtual machine instances
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45595—Network integration; Enabling network access in virtual machine instances
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/2141—Access rights, e.g. capability lists, access control lists, access tables, access matrices
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
A Docker mirror image information acquisition system of a private warehouse comprises a Docker mirror image acquisition center and a private warehouse server; the Docker mirror image acquisition center comprises a configuration module, an acquisition task module, a storage module and a network module; the Docker mirror image acquisition center sets an acquisition agent on a private warehouse server of a mirror image to be acquired; the collection agent is used for creating a new image with collection function by compiling the file and creating a container based on the new image as a running base. The application also provides a method for acquiring Docker mirror image information of a private warehouse, which comprises the following steps: the acquisition agent adds a mirror layer for information acquisition to the mirror to be acquired to construct a new mirror: the added mirror image layer has the functions of compressing mirror image contents into an acquisition folder and sending a mirror image file compression package in the acquisition folder to a Docker mirror image acquisition center; and establishing a container based on the new mirror image as the running basis according to the new mirror image after the mirror image layer is added. The application improves the operation efficiency of the system.
Description
Technical Field
The application relates to the technical field of private warehouse Docker mirror image information vulnerability scanning, in particular to a private warehouse Docker mirror image information acquisition system and an acquisition method thereof.
Background
With the rapid development of virtualization technology, container technology gradually becomes the mainstream deployment mode of various industries and services, but the packing and stacking storage modes of the container technology increase the difficulty of information acquisition and depth analysis. The packing injection of the unnormalized use of developers, the open source software bugs, the malicious software injection and the like in the mirror image leads the container environment and the service environment based on the mirror image to have a plurality of potential safety hazards. Meanwhile, the management and maintenance of container images through a container image warehouse server become the actual standard in the industry, and the image acquisition and analysis aiming at the container images become the key direction in the industry.
In the face of such a situation, the information collection and analysis manner for the remote mirror on the remote mirror repository server is usually realized by means of hierarchical parsing and merging analysis. If the open source software Clair analyzes the container mirror image in a layered mode, when an interface of the open source software Clair is called, mirror image layer addresses and the incidence relation between the mirror image layer addresses and the parent layer need to be transmitted to perform analysis and scanning, and when a result is returned, the aggregation analysis result between the mirror image layer addresses and the parent layer chain is returned according to the layers. However, in such a manner, manual parsing is required for parsing layers, automation cannot be achieved, and the problem of coverage of the same result between layers cannot be solved when layer analysis results are aggregated, that is, the problem that the upper layer solves the parent layer cannot be reflected in the overall result, Clair needs centralized storage and analysis mirror images, and a large amount of computation and storage performance needs to be consumed in an analysis center.
Patent CN109918911A provides a mirror image information collection method for remote warehouse, which reads the mirror image manifest from the remote warehouse and performs download scanning layer by layer, and this method also performs scanning in units of layers and performs aggregate feedback on the relationship between the layers, however, this invention does not mention the processing method of the coverage content when the layers are associated. The invention also carries out the aggregation analysis according to the single-layer analysis result, brings extra analysis and resource consumption in the relationship of the scanning analysis result, and can carry out coverage detection.
In summary, most of the existing mirror image scanning modes use layer scanning and provide an aggregate scanning result of a mirror image layer, which brings extra calculation consumption and resource consumption, while a few scans using a mirror image as a unit need to adapt to different overlay storage engines, and information mirror image information is restored by constructing an overlay storage mode externally, which brings extra adaptation and more fault points, possibly brings stability influence to the original system and service, and has certain defects.
Disclosure of Invention
The application provides a Docker mirror image information acquisition system of a private warehouse, which comprises a Docker mirror image acquisition center and a private warehouse server; the private warehouse server comprises a mirror image warehouse, and the mirror image warehouse consists of Docker mirror images;
the Docker mirror image acquisition center comprises a configuration module, an acquisition task module, a storage module and a network module;
the configuration module comprises a network communication address for configuring a Docker mirror image acquisition center and authority authentication information for configuring a private warehouse server;
the acquisition task module is used for acquiring a mirror image list in the private warehouse server, creating an acquisition task according to the acquired mirror image list, issuing the acquisition task to the private warehouse server, and collecting an acquisition result after the private warehouse server finishes the acquisition task; the mirror image list at least comprises a mirror image warehouse name, a mirror image label and a mirror image ID;
the storage module is used for providing system bottom storage to support the whole system;
the network module is used for providing network communication and realizing remote network interaction;
the Docker mirror image acquisition center sets an acquisition agent on a private warehouse server of a mirror image to be acquired; the acquisition agent can acquire information of a configuration module of the Docker mirror image acquisition center; acquiring authority for reading, writing and calling a local private warehouse server by an acquisition agent by acquiring authority authentication information of the private warehouse server; the acquisition agent can send the acquired mirror image information to the Docker mirror image acquisition center through the acquired network communication address of the Docker mirror image acquisition center;
the acquisition agent comprises a local acquisition module and a remote task execution module; the local acquisition module is used for acquiring a mirror image list to be acquired on the local private warehouse server; the remote task execution module is used for executing an acquisition task issued by the Docker mirror image acquisition center, creating a new mirror image with an acquisition function through compiling a file and creating a container based on the new mirror image as operation.
The remote task execution module at least comprises a task analysis unit, a task execution unit and a resource cleaning unit; the task analysis unit is used for analyzing an acquisition task issued by the Docker mirror image acquisition center; the task execution unit is used for logically organizing and executing the analyzed acquisition task; the resource cleaning unit is used for cleaning various resources after the collection task is finished, and comprises a collection container operation result, a container operation log, a deletion container and an image file.
The network communication address of the Docker mirror image acquisition center comprises an IP address and a service port.
The authority authentication information of the private warehouse server comprises a private warehouse server type, a private warehouse server authentication type and private warehouse server authentication information; the authentication type of the private warehouse server includes, but is not limited to, an account password mode and a Token mode, and the authentication information of the corresponding private warehouse server includes a user name, a password and a Token.
The Docker mirror image acquisition center and the acquisition agent acquire the mirror image list in the private warehouse server through a container instruction interface or through a private warehouse server API.
The application also provides a Docker mirror image information acquisition method using the private warehouse Docker mirror image information acquisition system, which specifically comprises the following steps:
s10, configuring a network communication address of a Docker mirror image acquisition center and authority authentication information of a private warehouse server;
s20, the Docker mirror image acquisition center acquires a mirror image list in the remote private warehouse server; wherein, the mirror image list at least comprises a mirror image warehouse name, a mirror image label and a mirror image ID;
s30, the Docker mirror image collection center establishes a collection task according to the obtained mirror image list, and sets a collection agent on a private warehouse server where the mirror image to be collected is located; sending the collection task to a corresponding collection agent; the acquisition task at least comprises an acquisition task number and a mirror image to be acquired;
s40, the acquisition agent obtains authority authentication information of a local private warehouse server of the Docker mirror image acquisition center, the acquisition agent obtains a mirror image list to be acquired on the local private warehouse server, and a mirror image layer for information acquisition is added to the mirror image to be acquired to construct a new mirror image: the added mirror image layer has the functions of compressing mirror image contents into an acquisition folder and sending a mirror image file compression package in the acquisition folder to a Docker mirror image acquisition center; according to the new mirror image after the mirror image layer is added, a container which takes the new mirror image as the operation basis is established;
s50, starting the container in a safe environment, providing a network communication address of a Docker mirror image acquisition center, and automatically sending the mirror image file compression packet in the acquisition folder to the Docker mirror image acquisition center by the container;
s60, the collection agent continuously manages the state of the container and recovers the resources when the mirror image collection task is finished; when all the collection tasks are completed, the collection agent feeds back the results of the collection tasks to the Docker mirror image collection center.
In step S20, the method further includes step S21, when the collection task is established, skipping the collection task of the mirror image when the mirror image with the same ID is already in the collection task list.
The Docker mirror image collection center and the collection agent use Docker API or container instruction to obtain the Docker mirror image list of the private warehouse.
In step S40, adding a new image layer to the base image using the Dockerfile compiled file, and adding a collection folder and a compression tool folder in the same directory of the base image; the added mirror image layer has the functions of putting a compressed packet obtained by compressing the basic mirror image into an acquisition folder and sending the compressed packet of the mirror image file in the acquisition folder to a Docker mirror image acquisition center.
And the results of the acquisition tasks sent by the Docker mirror image acquisition center to the acquisition agents and the acquisition tasks fed back by the acquisition agents to the Docker mirror image acquisition center are expressed by JSON.
The beneficial effect that this application realized is as follows:
the invention carries out information acquisition and scanning based on the mirror image as a unit, thereby reducing the storage consumption brought by layered acquisition, and meanwhile, the repeated scanning consumption of files with the same file name at different layers can be removed by carrying out information acquisition by taking the mirror image as a unit, thereby improving the scanning efficiency and reducing the calculation consumption in scanning. In addition, based on the characteristics of the container, the new mirror image is constructed in a mode of adding a new mirror image layer with an acquisition function to the mirror image, and the acquisition is automatically completed by constructing the container corresponding to the new mirror image. Meanwhile, the acquisition agent is used for specifically executing the acquisition tasks, so that the mirror image pulling and mirror image analysis tasks which are uniformly executed by the original acquisition center can be dispersed on the host machines of the private warehouse servers, the calculation and storage pressure of the analysis center is reduced, and the overall efficiency and the concurrent processing capacity are improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art according to the drawings.
Fig. 1a and 1b are logical structure diagrams of mirror layer files when the content of a Docker mirror is changed.
FIG. 1c is a logical block diagram of a container runtime reading a mirror as a whole.
Fig. 2 is a logical structure diagram of the private warehouse Docker mirror image information acquisition system of the present application.
Fig. 3 is a flowchart illustrating steps of a method for acquiring Docker mirror image information of a private warehouse according to the present application.
Fig. 4 is a network distribution structure diagram of the private warehouse Docker mirror image information acquisition system of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below clearly and completely with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any inventive effort, shall fall within the scope of protection of the present application.
Docker is an open source application container engine, so that developers can pack applications and dependency packages into a portable container and then distribute the container to any popular Linux or Windows machine, and virtualization can be realized, the container completely uses a sandbox mechanism, and no interface exists among the containers.
An entire Docker consists of the following parts: client (Client), Daemon (Daemon), Image (Image), and Container (Container).
The Docker mirror image is similar to a mirror image in a virtual machine and is a read-only template which comprises a file system and faces a Docker engine, any application program needs an environment for running, the mirror image is used for providing the running environment, the Docker mirror image is a multi-layer file structure, each layer is called a mirror layer (Layers), the multi-layer structure of the Docker mirror image can be regarded as a unified file system (combined file system), and a Docker process considers that the whole file system is mounted in a read-write mode. All Docker images start at a base image layer, i.e. the parent layer. When a mirror is modified or new content is added, a new mirror layer is created on top of all the current mirror layers. In time, when a change occurs, the original image file belonging to the lower layer does not change, all changes occur in the top layer, and if the content of one image layer is modified and replaced, the content does not occur on the image layer, but is performed in the upper image layer.
The Docker container is an application example of a Docker mirror image, the container can be created, started, stopped and deleted, and the containers are mutually isolated and do not influence each other. The Docker container is similar to a lightweight sandbox, which may be viewed as a very simple Linux system environment (including root rights, process space, user space, and web space, etc.), and applications running therein.
When the Docker container is started from the Docker mirror image, the Docker creates a writable layer on all the mirror layers, and the Docker mirror image is not changed, so that if the container is to be deleted, the writable layer of the container can be directly deleted.
A Docker mirror image warehouse (replication) is a place for storing Docker mirror images, and a registration server is a specific server for storing the mirror image warehouse; the registration server may have a plurality of mirror image repositories, each repository having a plurality of mirror images; the mirror repository corresponds to a specific item or directory and has its own access address, for example, the mirror repository address is d1. dickerplane.com/ubuntu, where d1. dickerplane.com is the registration server address and ubuntu is the repository name.
The Docker mirror image warehouse is divided into a public warehouse and a private warehouse according to the access authority type. The public repository is open, allowing all users to download the image anonymously. The private warehouse is invisible to other users, the mirror image can be downloaded only when an account with authority logs in, a server for storing the private warehouse is a private server, and commonly used private warehouse servers are generally Registry and Harbor.
The mirror repository (replication) is used as a set of mirrors, generally, the same mirror with different versions is placed in the same repository, the name of the repository is the name of the mirror, the version number is represented by a Tag number (Tag), and the mirror also has its own unique identification ID. Based on the repository name, tag and ID, a mirror can be specifically located (with ": separation" between repository name and tag).
Dockerfile is a text file used to construct an image, the text content containing instructions and descriptions necessary to construct the image. For example, Dockerfile starts with FROM command, followed by various methods, commands and parameters, and may use a docker build instruction to create a mirror, and an instruction docker run to start a container.
An API (Application Programming Interface) is a predefined function that provides applications and developers the ability to access a set of routines based on certain software or hardware without accessing source code or understanding the details of the internal workings. The Docker API is the call interface that Docker gives to applications.
When a Docker API is used, the system identifies the container environment of the local host and adapts to different container versions to call the API to acquire information; when the container instruction is used, the information is acquired by using a Docker system instruction.
JSON (JavaScript Object Notation) is adopted as a communication mode for acquiring the interactive information by the remote server, and is a lightweight data exchange format. The method is based on a subset of ECMAScript (js specification established by European computer Association), adopts a text format completely independent of a programming language to store and represent data, has a simple and clear hierarchical structure, is easy to read and write by people, is easy to analyze and generate by a machine, and effectively improves the network transmission efficiency.
The concept of hierarchical storage used by Docker Image, except the current layer, each of the previous layers is unchanged, in other words, any modification only results in marking, adding and modifying at the current layer without changing other layers below the current layer, as shown in fig. 1a and 1b, the first layer of mirror Image (Lay1) of mirror Image includes file 1, file 2 and file 3, the second layer above the first layer (Lay2) includes file 4, file 5 and file 6, the third layer above the second layer (Lay3) makes changes to file 5, and file 5 is replaced with file 7, since the mirror Image is a modification to the layer file of the lower layer with the layer file of the upper layer, so the file 5 in the second layer is not directly replaced on the second layer, but a change of replacement is made on the third layer.
On the basis of the Docker image, after the container corresponding to the image is started, with the help of Docker service, the content of the image can be read in a single-layer manner, as shown in fig. 1c, after the container is started, although the layer file of the Docker image is not changed in the basic storage manner, after the container is operated, from the perspective of a user, the Docker reads the image as a whole, that is, the multi-layer structure of the image is logically combined into one layer, the combination state in the logical structure is shown in fig. 1c, and the final state of the original image file after multi-layer modification is obtained.
According to the above technology, the application provides a private warehouse Docker image information acquisition system, which comprises a Docker image acquisition center and a private warehouse server;
the Docker mirror image acquisition center is provided with an acquisition agent on a private warehouse server;
the Docker mirror image acquisition center comprises a configuration module, an acquisition task module, a storage module and a network module;
the configuration module is used for configuring a network address and a service port of the Docker mirror image acquisition center and configuring basic information of a remote private warehouse server; through the network address and the service port of the Docker mirror image acquisition center, the acquisition agent can send the compressed mirror image file to the Docker mirror image acquisition center.
The basic information of the private warehouse server comprises a private warehouse server type, a private warehouse server authentication type and private warehouse server authentication information.
The acquisition task module is used for acquiring a mirror image list in the private warehouse server, creating an acquisition task according to the acquired mirror image list, issuing the acquisition task to the private warehouse server, and collecting an acquisition result after the private warehouse server finishes the acquisition task; the mirror image list at least comprises a mirror image warehouse name, a mirror image label and a mirror image ID;
the storage module is used for providing system bottom storage to support the whole system, and comprises file storage, data storage and storage monitoring capacity, and the storage module can be used for mirror image information storage, acquisition task storage, acquisition result storage and other storage functions.
The network module is used for managing a system network, providing basic network communication capability and enabling the system to realize network interconnection and interaction for remote.
The Docker mirror image acquisition center sets an acquisition agent on a private warehouse server of a mirror image to be acquired; the acquisition agent can acquire information of a configuration module of the Docker mirror image acquisition center; acquiring authority for reading, writing and calling a local private warehouse server by an acquisition agent by acquiring authority authentication information of the private warehouse server; the acquisition agent can send the acquired mirror image information to the Docker mirror image acquisition center through the acquired network communication address of the Docker mirror image acquisition center;
the acquisition agent comprises a local acquisition module and a remote task execution module; the local acquisition module is used for acquiring a mirror image list of a local private warehouse server; the remote task execution module is used for executing an acquisition task issued by the Docker mirror image acquisition center, creating a new mirror image with an acquisition function through compiling a file and creating a container based on the new mirror image as operation.
The remote task execution module at least comprises a task analysis unit, a task execution unit and a resource cleaning unit.
The task analysis unit is used for analyzing an acquisition task issued by the Docker mirror image acquisition center.
The task execution unit is used for logically organizing the analyzed acquisition task and executing the acquisition task.
The resource cleaning unit is used for cleaning various resources after the collection task is finished, and comprises a collection container operation result, a collection container operation log, a container deletion unit and a mirror image file deletion unit.
The application also provides a method for acquiring Docker mirror image information of the private warehouse, which comprises the following steps:
configuring a network address and a service port of a Docker mirror image acquisition center, and configuring basic information of a remote private warehouse server;
the basic information of the private repository server includes a private repository server type, a private repository server authentication type, and private repository server authentication information.
The authentication type of the private warehouse server includes, but is not limited to, an account password mode and a Token mode, and the authentication information of the corresponding private warehouse server includes a user name, a password and a Token.
The method comprises the following steps that a Docker mirror image collection center obtains mirror image lists in remote private warehouse servers;
the mirror image list comprises basic information of all mirror images on the mirror image warehouse server, or the basic information of a specific mirror image is downloaded according to the requirement; the mirror image basic information at least comprises a mirror image ID, a warehouse (repeatability) and a label (Tag);
the manner in which the Docker image acquisition center acquires the image list in each remote private warehouse server may be acquired through a container instruction interface or may be acquired through a private warehouse server API.
For example, in a specific operation, the Docker image acquisition center acquires an image warehouse on the private warehouse server through an API interface of the Registry private warehouse server, where the specific interface is as follows:
GET/v2/_catalog
acquiring a mirror label and a mirror ID through the following interfaces:
GET/v2/<name>/tags/list
after receiving the instruction, the Registry private warehouse server sends the mirror image list to the Docker mirror image acquisition center in a JSON form.
After the Docker mirror image acquisition center obtains a mirror image list, establishing an acquisition task, and deploying an acquisition agent on a private warehouse server host machine of a mirror image to be acquired; the Docker mirror image acquisition center sends basic information of a local private warehouse server to an acquisition agent, and simultaneously sends an acquisition task to a corresponding acquisition agent;
the system comprises a Docker mirror image acquisition center, an acquisition agent and a management center, wherein an acquisition task sent by the Docker mirror image acquisition center to the acquisition agent at least comprises an acquisition task number and a mirror image to be acquired;
for example, in a specific implementation, the collection task sent by the Docker image collection center to the collection agent is represented in JSON form as follows:
that is, the task number sent by the Docker image acquisition center to the acquisition agent is 231, there are two images to be acquired, which are the image of the warehouse name mysql, the label is 5.3, the ID is e4247c08758ef42f3f7d1079d20718eea6c414015a86950d748745a60ad73fd4, and the image of the warehouse name python, the label is 3.7.1-alpine, the ID is 020295c920c635bbb25e4c73e026834e1bbfc5225955d0ecd63016c5d bc0 ca.
The images can be uniquely distinguished by the image ID of the images, so that the images with the same ID only need to be collected once, and when the images with the same ID are collected, the collection task of the images is not needed.
The acquisition agent adds a mirror image layer for information acquisition to a mirror image to be acquired in an acquisition task to construct a new mirror image:
the acquisition agent acquires the mirror image to be acquired on the local private warehouse server according to the acquisition task, and the acquisition mode can be a mode of an API (application program interface) of the mirror image warehouse or a mode of a container instruction.
In the embodiment, a mysql:5.3 mirror image is taken as an example, and the basic Dockerfile is as follows:
FROM mysql:5.3
RUN mkdir-p/scan
RUN mkdir-p/scantools
COPY/tools//scantools
RUN/scantools/tar-zcvf../scan/scaninfo_mysql_5.3.tar.gz`find/-path /scantools-path'/scan'-o-path/tar-a-prune-maxdepth 1-o-print|sed 1d`scan scantools
CMD[“/scantools/push$CIP$CPORT../scan/scaninfo_mysql_5.3.tar.gz”]
docker build-t mysql:5.3-scan.
docker run-itd-e CIP＝10.10.10.10-e CPORT＝8080--name
mysql:2.175.3-scan-container mysql:2.175.3-scan
in the Dockerfile file, mysql:5.3 is used as a basic mirror image, and a collection folder scan and a compression tool folder scantools are added under the same directory of the basic mirror image; adding a mirror image layer on a basic mirror image by using an instruction, wherein the added mirror image layer has the functions of compressing the file content of the basic mirror image by a compression tool in scantools to form a mirror image file compression packet scaninfo _ mysql _5.3.tar.gz, and sending the mirror image file compression packet scaninfo _ mysql _5.3.tar.gz in a collection folder to a Docker mirror image collection center, compiling a new mirror image after adding the mirror image layer according to Docker file, and establishing a container mysql of 5.3-scan, 2.175.3-scan-container, which takes the new mirror image as a running base; the container is started, a network ip address 10.10.10.10 and a service port 8080 of the Docker mirror image acquisition center are provided, and the container can automatically send a mirror image file compression packet scaninfo _ mysql _5.3.tar.gz in an acquired file to the Docker mirror image acquisition center through the network address and the service port.
Because the new mirror image container also executes all instruction operations of the original mirror image container, the new mirror image container needs to be operated in a safe environment in order to ensure the operation safety.
When the new mirror image is successfully constructed, the docker returns the whole id of one mirror image, when the container is operated, all the mirror images can be read and written as a whole in a mode of reading and writing the container by the id, and the bottom layer of the whole points to the multilayer content.
In the Dockerfile command statement, the CMD statement is executed last, that is, after the container runs, the compressed image file is sent to the Docker image collection center. After the container is started, the multi-layer image files of the image file compression package can be obtained and read in a single-layer structure, so that the image files combined on the logic structure are obtained and subjected to vulnerability scanning, namely, the images in the final state combined into one layer are scanned, and at the moment, the vulnerability scanning can be directly carried out without layered analysis or a plurality of engines.
The collection agent continuously manages the state of the container based on the new mirror image as the operation basis and recovers the resources when the mirror image collection task is finished;
after the container is constructed and started, the container is continuously monitored, when the container stops running, container logs and stopping results are recorded, the container is deleted, the mirror image is deleted, and resources are recycled.
When all the collection tasks are completed, the collection agent feeds back the results of the collection tasks to the Docker mirror image collection center,
the collection task result includes a collection task ID and a collection result mirrored in each collection task, and is expressed as JSON in the present embodiment.
And the private warehouse container mirror image information acquisition center finishes the acquisition task after receiving the acquisition task result.
While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all such alterations and modifications as fall within the scope of the application. It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (10)
1. A Docker mirror image information acquisition system of a private warehouse comprises a Docker mirror image acquisition center and a private warehouse server; the private warehouse server comprises a mirror image warehouse, and the mirror image warehouse consists of Docker mirror images;
the Docker mirror image acquisition center comprises a configuration module, an acquisition task module, a storage module and a network module;
the configuration module comprises a network communication address for configuring a Docker mirror image acquisition center and authority authentication information for configuring a private warehouse server;
the acquisition task module is used for acquiring a mirror image list in the private warehouse server, creating an acquisition task according to the acquired mirror image list, issuing the acquisition task to the private warehouse server, and collecting an acquisition result after the private warehouse server finishes the acquisition task; the mirror image list at least comprises a mirror image warehouse name, a mirror image label and a mirror image ID;
the storage module is used for providing system bottom storage to support the whole system;
the network module is used for providing network communication and realizing remote network interaction;
the Docker mirror image acquisition center sets an acquisition agent on a private warehouse server of a mirror image to be acquired; the acquisition agent can acquire information of a configuration module of the Docker mirror image acquisition center; acquiring authority for reading, writing and calling a local private warehouse server by an acquisition agent by acquiring authority authentication information of the private warehouse server; the acquisition agent can send the acquired mirror image information to the Docker mirror image acquisition center through the acquired network communication address of the Docker mirror image acquisition center;
the acquisition agent comprises a local acquisition module and a remote task execution module; the local acquisition module is used for acquiring a mirror image list to be acquired on the local private warehouse server; the remote task execution module is used for executing an acquisition task issued by the Docker mirror image acquisition center, creating a new mirror image with an acquisition function through compiling a file and creating a container based on the new mirror image as operation.
2. The private warehouse Docker mirror image information acquisition system of claim 1, wherein the remote task execution module comprises at least a task parsing unit, a task execution unit, and a resource cleaning unit; the task analysis unit is used for analyzing an acquisition task issued by a Docker mirror image acquisition center; the task execution unit is used for logically organizing and executing the analyzed acquisition task; the resource cleaning unit is used for cleaning various resources after the collection task is finished, and comprises a collection container operation result and a container operation log, a deletion container and an image file.
3. The private warehouse Docker image information collection system of claim 1, wherein the network communication address of the Docker image collection center comprises an IP address and a service port.
4. The private warehouse Docker mirror image information collection system of claim 1, wherein the authority authentication information of the private warehouse server comprises a private warehouse server type, a private warehouse server authentication type, and private warehouse server authentication information; the authentication type of the private warehouse server includes, but is not limited to, an account password mode and a Token mode, and the authentication information of the corresponding private warehouse server includes a user name, a password and a Token.
5. The private warehouse Docker image information collection system of claim 1, wherein the Docker image collection center and the collection agent obtain the list of images in the private warehouse server through a container instruction interface or through a private warehouse server API.
6. A Docker image information acquisition method using the private warehouse Docker image information acquisition system of any of claims 1-5, comprising the steps of:
s10, configuring a network communication address of a Docker mirror image acquisition center and authority authentication information of a private warehouse server;
s20, the Docker mirror image acquisition center acquires a mirror image list in the remote private warehouse server; the mirror list at least comprises a mirror warehouse name, a mirror label and a mirror ID;
s30, the Docker mirror image collection center establishes a collection task according to the obtained mirror image list, and sets a collection agent on the private warehouse server where the mirror image to be collected is located; sending the collection task to a corresponding collection agent; the acquisition task at least comprises an acquisition task number and a mirror image to be acquired;
s40, the acquisition agent obtains authority authentication information of a local private warehouse server of the Docker mirror image acquisition center, the acquisition agent obtains a mirror image list to be acquired on the local private warehouse server, and a mirror image layer for information acquisition is added to the mirror image to be acquired to construct a new mirror image: the added mirror image layer has the functions of compressing mirror image contents into an acquisition folder and sending a mirror image file compression package in the acquisition folder to a Docker mirror image acquisition center; according to the new mirror image after the mirror image layer is added, a container which takes the new mirror image as the operation basis is established;
s50, starting the container in a safe environment, providing a network communication address of a Docker mirror image acquisition center, and automatically sending the mirror image file compression packet in the acquisition folder to the Docker mirror image acquisition center by the container;
s60, collecting agent continuously managing container state and recovering resource when mirror image collection task is finished; when all the collection tasks are completed, the collection agent feeds back the results of the collection tasks to the Docker mirror image collection center.
7. The private warehouse Docker image information gathering method of claim 6, wherein in the step S20, further comprising a step S21 of skipping an image of the same ID when the image is already in the acquisition task list when the acquisition task is established.
8. The private warehouse Docker image information collection method of claim 6, wherein the Docker image collection center and the collection agent use a Docker API or a container instruction to obtain the private warehouse Docker image list.
9. The private warehouse Docker image information collection method of claim 6, wherein, in step S40, a new image layer is added to the base image using Docker file compilation files, and a collection folder and a compression tool folder are added under the same directory of the base image; the added mirror image layer has the functions of putting a compressed package obtained by compressing the basic mirror image into the acquisition folder and sending the compressed package of the mirror image file in the acquisition folder to the Docker mirror image acquisition center.
10. The private warehouse Docker mirror image information acquisition method of claim 6, wherein the results of acquisition tasks sent by the Docker mirror image acquisition center to the acquisition agents and the acquisition tasks fed back by the acquisition agents to the Docker mirror image acquisition center are both expressed in JSON.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110099167.8A CN112860335B (en) | 2021-01-25 | 2021-01-25 | Private warehouse Docker mirror image information acquisition system and acquisition method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110099167.8A CN112860335B (en) | 2021-01-25 | 2021-01-25 | Private warehouse Docker mirror image information acquisition system and acquisition method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112860335A true CN112860335A (en) | 2021-05-28 |
| CN112860335B CN112860335B (en) | 2024-02-20 |
Family
ID=76008737
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110099167.8A Active CN112860335B (en) | 2021-01-25 | 2021-01-25 | Private warehouse Docker mirror image information acquisition system and acquisition method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112860335B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113918096A (en) * | 2021-10-21 | 2022-01-11 | 城云科技（中国）有限公司 | Method and device for uploading algorithm mirror image packet and application |
| US11409787B2 (en) * | 2020-11-10 | 2022-08-09 | Nexcom International Co., Ltd. | Method for executing Docker image under protection |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107066310A (en) * | 2017-03-11 | 2017-08-18 | 郑州云海信息技术有限公司 | It is a kind of to build and using the method and device in the privately owned warehouses of safe Docker |
| CN107239688A (en) * | 2017-06-30 | 2017-10-10 | 平安科技（深圳）有限公司 | The purview certification method and system in Docker mirror images warehouse |
| CN108616419A (en) * | 2018-03-30 | 2018-10-02 | 武汉虹旭信息技术有限责任公司 | A kind of packet capture analysis system and its method based on Docker |
| CN109981351A (en) * | 2019-03-06 | 2019-07-05 | 浪潮通用软件有限公司 | A kind of private clound dispositions method |
-
2021
- 2021-01-25 CN CN202110099167.8A patent/CN112860335B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107066310A (en) * | 2017-03-11 | 2017-08-18 | 郑州云海信息技术有限公司 | It is a kind of to build and using the method and device in the privately owned warehouses of safe Docker |
| CN107239688A (en) * | 2017-06-30 | 2017-10-10 | 平安科技（深圳）有限公司 | The purview certification method and system in Docker mirror images warehouse |
| CN108616419A (en) * | 2018-03-30 | 2018-10-02 | 武汉虹旭信息技术有限责任公司 | A kind of packet capture analysis system and its method based on Docker |
| CN109981351A (en) * | 2019-03-06 | 2019-07-05 | 浪潮通用软件有限公司 | A kind of private clound dispositions method |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11409787B2 (en) * | 2020-11-10 | 2022-08-09 | Nexcom International Co., Ltd. | Method for executing Docker image under protection |
| CN113918096A (en) * | 2021-10-21 | 2022-01-11 | 城云科技（中国）有限公司 | Method and device for uploading algorithm mirror image packet and application |
| CN113918096B (en) * | 2021-10-21 | 2023-09-22 | 城云科技（中国）有限公司 | Method, device and application for uploading algorithm mirror image package |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112860335B (en) | 2024-02-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11789715B2 (en) | Systems and methods for transformation of reporting schema | |
| US8645326B2 (en) | System to plan, execute, store and query automation tests | |
| US8108456B2 (en) | Method and apparatus for migrating the system environment on which the applications depend | |
| US20210311858A1 (en) | System and method for providing a test manager for use with a mainframe rehosting platform | |
| Taibi et al. | Continuous architecting with microservices and devops: A systematic mapping study | |
| JP3912895B2 (en) | Structured data management system, computer-readable recording medium on which structured data management program is recorded, and structured data management method | |
| US8214809B2 (en) | Grid-enabled ANT compatible with both stand-alone and grid-based computing systems | |
| US20030028579A1 (en) | Process for component-based application development | |
| CN112860335B (en) | Private warehouse Docker mirror image information acquisition system and acquisition method thereof | |
| US10585785B2 (en) | Preservation of modifications after overlay removal from a container | |
| US20140074849A1 (en) | Remote artifact repository | |
| JP5745932B2 (en) | Method, program, and system for reflecting operation on object which is image of mapping in graph data | |
| CN113778500A (en) | Software development life cycle management platform based on DevOps | |
| CN110991984A (en) | Digital operation analysis platform and method based on enterprise information heterogeneous system | |
| CN116860223A (en) | Cloud-protogenesis-based low-code development and delivery method | |
| WO2002069141A1 (en) | Method and apparatus creation and performance of service engagement modeling | |
| US20210176138A1 (en) | Data center cartography bootstrapping from process table data | |
| US20110041119A1 (en) | Storing z/os product tag information within z/os load module datasets | |
| US20220398077A1 (en) | Application to container conversion and migration | |
| CN112860481A (en) | Local Docker mirror image information acquisition system and acquisition method thereof | |
| CN113656183A (en) | Task processing method, device, equipment and storage medium | |
| Mattmann et al. | Revisiting the Anatomy and Physiology of the Grid | |
| CN100468332C (en) | Program running method | |
| Hicks et al. | Integration and implementation (int) cs 5604 f2020 | |
| CN116382596B (en) | Space-time big data storage method and system based on distributed technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |