CN112714163A - Data transmission method, data transmission device, electronic equipment, medium and computer program product - Google Patents
Data transmission method, data transmission device, electronic equipment, medium and computer program product Download PDFInfo
- Publication number
- CN112714163A CN112714163A CN202011525520.6A CN202011525520A CN112714163A CN 112714163 A CN112714163 A CN 112714163A CN 202011525520 A CN202011525520 A CN 202011525520A CN 112714163 A CN112714163 A CN 112714163A
- Authority
- CN
- China
- Prior art keywords
- mirror
- layer
- uploaded
- data
- mirror image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Abstract
The present disclosure discloses a data transmission method, an apparatus, an electronic device, a medium, and a computer program product, which relate to the technical field of artificial intelligence, and in particular, to the technical field of container technology, cloud computing, and cloud service. The specific implementation scheme is as follows: acquiring a data uploading instruction from mirror image equipment; the data uploading instruction comprises a mirror image layer identifier of a mirror image layer to be uploaded in a mirror image file to be uploaded and mirror image layer data to be uploaded; storing the mirror layer identification of the mirror layer to be uploaded and the mirror layer data to be uploaded into a mirror storage system in an associated manner through an object storage form; acquiring a data downloading instruction from mirror equipment, determining mirror layer data to be downloaded in the mirror storage system according to the data downloading instruction, and sending the mirror layer data to be downloaded to the mirror equipment. The method and the device have the advantages that the metadata storage pressure of the mirror image warehouse is reduced on the premise that mirror image distribution is not influenced, and the mirror image distribution efficiency of the mirror image warehouse is improved.
Description
Technical Field
The present disclosure relates to the field of artificial intelligence technologies, and in particular, to the field of container technologies, cloud computing, and cloud service technologies, and in particular, to a data transmission method, an apparatus, an electronic device, a medium, and a computer program product.
Background
Compared with a virtual machine, the Docker container technology greatly reduces the resource occupancy rate and facilitates the operation and maintenance of virtualization. However, for thousands of Docker hosts, how to efficiently manage and distribute Docker images is a key factor in determining whether a Docker container can be deployed quickly.
In the prior art, a backend storage system performs file-type storage on a Docker image, a host sends a request instruction to an image warehouse, and the image warehouse acquires a target Docker image from the backend storage system and sends the target Docker image to the host.
Disclosure of Invention
The present disclosure provides a method, apparatus, electronic device, medium, and computer program product for reducing mirror inventory storage pressure.
According to an aspect of the present disclosure, there is provided a data transmission method, including:
acquiring a data uploading instruction from mirror image equipment; the data uploading instruction comprises a mirror image layer identifier of a mirror image layer to be uploaded in a mirror image file to be uploaded and mirror image layer data to be uploaded;
storing the mirror layer identification of the mirror layer to be uploaded and the mirror layer data to be uploaded into a mirror storage system in an associated manner through an object storage form;
acquiring a data downloading instruction from mirror equipment, determining mirror layer data to be downloaded in the mirror storage system according to the data downloading instruction, and sending the mirror layer data to be downloaded to the mirror equipment.
According to another aspect of the present disclosure, there is provided a data transmission apparatus including:
the data uploading instruction acquisition module is used for acquiring a data uploading instruction from the mirror image equipment; the data uploading instruction comprises a mirror image layer identifier of a mirror image layer to be uploaded in a mirror image file to be uploaded and mirror image layer data to be uploaded;
the object storage module is used for storing the mirror image layer identification of the mirror image layer to be uploaded and the mirror image layer data to be uploaded into the mirror image storage system in an associated manner through an object storage form;
and the data distribution module is used for acquiring a data downloading instruction from the mirror image equipment, determining mirror image layer data to be downloaded in the mirror image storage system according to the data downloading instruction, and sending the mirror image layer data to be downloaded to the mirror image equipment.
According to another aspect of the present disclosure, there is provided an electronic device including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any of the present disclosure.
According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of the present disclosure.
According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method according to any one of the present disclosure.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.
Drawings
The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:
FIG. 1 is a flow chart of a data transmission method disclosed in accordance with an embodiment of the present disclosure;
FIG. 2 is a flow chart of a data transmission method disclosed in accordance with an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of a data transmission device according to an embodiment of the disclosure;
fig. 4 is a block diagram of an electronic device for implementing the data transmission method disclosed in the embodiment of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
In the research and development process of the applicant, it is found that in the existing mirror image distribution method, the mirror image layer data of each mirror image layer in the mirror image file is usually stored in the mirror image storage system in a file storage form, and the metadata corresponding to each mirror image file is maintained in the mirror image warehouse, so that the normal operation of the mirror image distribution operation can be ensured based on the maintained metadata. However, as the number of image files increases, the data volume of metadata in the image repository increases, which undoubtedly increases the storage pressure of the image repository, and accordingly reduces the image distribution efficiency of the image repository.
Fig. 1 is a flowchart of a data transmission method disclosed in an embodiment of the present disclosure, and this embodiment may be applied to a case where a mirror warehouse stores mirror layer data uploaded by a mirror device. The method of the present embodiment may be performed by a data transmission method apparatus, which may be implemented by software and/or hardware, and may be integrated on any electronic device with computing capability.
As shown in fig. 1, the data transmission method disclosed in this embodiment may include:
s101, acquiring a data uploading instruction from mirror image equipment; the data uploading instruction comprises a mirror image layer identifier of a mirror image layer to be uploaded in the mirror image file to be uploaded and mirror image layer data to be uploaded.
The mirror device refers to an electronic device capable of running an image file, for example, a Docker mirror device is a mirror device capable of running a Docker image file to start a Docker container. The image file contains basic information for container starting work, such as programs, libraries, resources, configuration files, configuration parameters and the like, namely, the corresponding container starting work can be started at the local end of the image equipment based on the image file. The mirror image file is composed of at least one layer of loosely coupled mirror image layers, each layer of mirror image layer comprises respective corresponding data content, only one layer of basic mirror image layer is included in the initial stage of establishing the mirror image file, and a new mirror image layer is established on the basic mirror image layer along with the modification or the increase of the content of the mirror image file. The image file always keeps the combination of all the current image layers, namely the sum of all the image layer data is the image data of the image file.
In one embodiment, any mirror image device generates a data uploading instruction according to a mirror image layer identifier of a mirror image layer to be uploaded and a mirror image layer data to be uploaded in a mirror image file to be uploaded, and a mirror image warehouse acquires the data uploading instruction from the mirror image device. The mirror image warehouse is an electronic device for storing mirror image files, and the mirror image device can initiate a downloading request to the mirror image warehouse so as to download the required mirror image files from the mirror image warehouse; the data uploading instruction comprises a push instruction.
Optionally, the mirror layer identifier of the mirror layer to be uploaded is obtained by performing hash calculation on the mirror layer data of the mirror layer to be uploaded.
The mirror image device performs hash calculation on the mirror image layer data to be uploaded of any mirror image layer to be uploaded, and the obtained hash value is used as the mirror image layer identification of the mirror image layer to be uploaded. Due to the uniqueness of the hash value, different mirror layer identifications to be uploaded correspond to different mirror layer identifications.
The image layer identification of the image layer to be uploaded is obtained by carrying out Hash calculation on the image layer data to be uploaded of the image layer to be uploaded, so that the effect of accurately distinguishing the image layers to be uploaded is realized, and a foundation is laid for subsequently storing the image layer identification of the image layer to be uploaded and the image layer data to be uploaded.
And S102, storing the mirror image layer identification of the mirror image layer to be uploaded and the mirror image layer data to be uploaded in a mirror image storage system in an associated manner through an object storage form.
The mirror image storage system is an electronic device with data storage capacity, and is in communication connection with the mirror image warehouse through an interface. Object storage is an object-based storage in which data is broken down into units called "objects" and stored in a single repository where the data, i.e., "objects," metadata describing the data, and a unique identifier that allows the "objects" to be found on a mirrored storage system, i.e., "object names," are stored. The object storage mode allows the mirror image storage system to store data, identifiers and metadata at the same time, and the problem of overlarge storage pressure of a mirror image warehouse is well solved.
In one embodiment, the mirror repository includes an object store interface in addition to a file store interface, such as a posix interface. The mirror image warehouse takes the mirror image layer identification of the mirror image layer to be uploaded as an object name in the object storage, takes the mirror image layer data to be uploaded of the mirror image layer to be uploaded as an object in the object storage, and stores the mirror image layer identification of the mirror image layer to be uploaded and the mirror image layer data to be uploaded into the mirror image storage system in an associated mode through an object storage mode based on a preset object storage interface. Due to the characteristics of the object storage mode, metadata can be stored in the mirror image storage system, and at the local end of the mirror image warehouse, only the identifier of the object which can be found, namely the mirror image layer identifier of the mirror image layer, needs to be stored, and the corresponding mirror image layer data can be quickly positioned in the mirror image storage system according to the mirror image layer identifier, so that mirror image distribution is realized.
The mirror image layer identification of the mirror image layer to be uploaded and the mirror image layer data to be uploaded are stored in the mirror image storage system in an associated mode through an object storage mode, and therefore metadata storage pressure of the mirror image warehouse is reduced.
S103, acquiring a data downloading instruction from the mirror image equipment, determining mirror image layer data to be downloaded in the mirror image storage system according to the data downloading instruction, and sending the mirror image layer data to be downloaded to the mirror image equipment.
In one embodiment, the mirror warehouse receives a data download instruction sent by any mirror device, determines a mirror layer identifier of a mirror layer to be downloaded according to the data download instruction and a mirror layer identifier stored in the mirror warehouse, further obtains the mirror layer data to be downloaded of the mirror layer to be downloaded from the mirror storage system according to an association relationship between the mirror layer identifier and the mirror layer data established in the mirror storage system, and sends the mirror layer data to be downloaded to the mirror device.
According to the technical scheme of the embodiment, the data uploading instruction is acquired from the mirror image equipment; the data uploading instruction comprises a mirror layer identifier of a mirror layer to be uploaded and a mirror layer data to be uploaded in a mirror file to be uploaded, and the mirror layer identifier of the mirror layer to be uploaded and the mirror layer data to be uploaded are stored in a mirror storage system in an object storage mode in a related mode.
On the basis of the above embodiment, after S101, the method includes:
and rejecting the data uploading instruction under the condition that the mirror image layer identification of the mirror image layer to be uploaded is stored in the mirror image warehouse.
In one embodiment, after the mirror warehouse acquires the data uploading instruction, the mirror layer identifier of the mirror layer to be uploaded is obtained through analysis, and the mirror layer identifier of the mirror layer to be uploaded is compared with the mirror layer identifier stored at the local end of the mirror warehouse. And if the mirror layer identifier of the mirror layer to be uploaded is the same as the stored mirror layer identifier of any mirror warehouse, indicating that the mirror layer data of the mirror layer to be uploaded is uploaded to the mirror warehouse, rejecting the data uploading instruction associated with the mirror layer to be uploaded.
Under the condition that the mirror image layer identification of the mirror image layer to be uploaded is stored in the mirror image warehouse, the data uploading instruction is rejected, the effect of limiting repeated uploading of the same-name object is achieved, the data multiplexing of the mirror image layer is achieved, and the storage pressure of the mirror image warehouse is reduced.
Fig. 2 is a flowchart of a data transmission method disclosed according to an embodiment of the present disclosure, which is further optimized and expanded based on the above technical solution, and can be combined with the above optional embodiments.
As shown in fig. 2, the method may include:
s201, acquiring a data uploading instruction from mirror image equipment; the data uploading instruction comprises a file identifier of the mirror image file to be uploaded, a mirror image layer identifier of the mirror image layer to be uploaded in the mirror image file to be uploaded and mirror image layer data to be uploaded.
Optionally, the file identifier of the image file to be uploaded is obtained by performing hash calculation on total data of an image layer to be uploaded in the image file to be uploaded.
The mirror image device performs hash calculation on the sum of all mirror image layer data in the mirror image file to be uploaded, and the obtained hash value is used as a file identifier of the mirror image file to be uploaded. Due to the uniqueness of the hash value, different image files to be uploaded correspond to different file identifications.
The total data of the image layer to be uploaded in the image file to be uploaded is subjected to Hash calculation to obtain the file identification of the image file to be uploaded, and a foundation is laid for subsequently distinguishing image layer data of different image files recorded in an image warehouse.
S202, recording the mirror image layer identification of the mirror image layer to be uploaded in the mirror image warehouse through a first mode.
In one embodiment, the first mode is recorded as a permanent record, that is, the mirror repository permanently records the mirror layer identifier of the mirror layer to be uploaded at the local end of the mirror repository. The means for recording optionally includes storing the image layer identifier of the image layer to be uploaded in an image warehouse development file, such as a manifest file of the image warehouse.
Optionally, S202 includes:
and recording the mirror image layer identification of the mirror image layer to be uploaded in a mirror image warehouse in a first mode, and taking the file identification of the mirror image file to be uploaded as a recording label.
In one embodiment, the mirror repository permanently records the mirror layer identifier of the mirror layer to be uploaded at the local end of the mirror repository, and uses the file identifier of the mirror file to be uploaded as a record label, thereby distinguishing the mirror layer identifiers of different mirror files. Optionally, a development file is newly created in the mirror image warehouse, the mirror image layer identifier of the mirror image layer to be uploaded is stored in the development file, and the file identifier of the mirror image file to be uploaded is used as the tag of the development file. Namely, one to-be-uploaded image file corresponds to one development file, and only the image layer identifier of the to-be-uploaded image layer in the to-be-uploaded image file is stored in the development file.
The mirror image layer identification of the mirror image layer to be uploaded is recorded in the mirror image warehouse in a first mode, and the file identification of the mirror image file to be uploaded is used as the recording label, so that the effect of distinguishing different mirror image layer identifications of the mirror image file to be uploaded is achieved, management of the mirror image warehouse is facilitated, the corresponding mirror image layer identification is determined according to the obtained file identification for the subsequent mirror image warehouse, and a foundation is laid.
S203, storing the mirror image layer identification of the mirror image layer to be uploaded and the mirror image layer data to be uploaded into a mirror image storage system in a key value pair associated manner; in the key value pair, the mirror layer identifier of the mirror layer to be uploaded is a keyword, and the data of the mirror layer to be uploaded is a data value.
In one embodiment, the mirror warehouse establishes a key value pair according to the mirror layer identifier of the mirror layer to be uploaded and the mirror layer data to be uploaded, the mirror layer identifier of the mirror layer to be uploaded in the key value pair is a key word, namely an object name in the corresponding object storage, and the mirror layer data to be uploaded is a data value, namely an object in the corresponding object storage. And then storing the mirror layer identification of the mirror layer to be uploaded and the data of the mirror layer to be uploaded into a mirror storage system in a key value pair form in an associated manner.
For example, assuming that the image layer identifier of the image layer to be uploaded is "00001" and the image layer data to be uploaded is "QWERSDFZXCV", the Key Value pair is established by using "00001" as the Key and "QWERSDFZXCV" as the data Value.
And S204, under the condition that the data of the mirror layer to be uploaded of any mirror layer to be uploaded is successfully stored, recording the mirror layer identification of the mirror layer to be uploaded in a mirror warehouse through a second mode.
In one embodiment, the second mode is recorded as a temporary record, when any mirror layer data to be uploaded of the mirror layer to be uploaded is successfully stored in the mirror storage system, the mirror storage system feeds back successful storage information to the mirror warehouse, and the mirror warehouse responds to the successful storage information and temporarily records the successfully stored mirror layer identifier of the mirror layer to be uploaded at the local end of the mirror warehouse. The recording means optionally includes storing the image layer identifier of the successfully stored image layer to be uploaded in a local temporary file, such as a manifest.
S205, comparing the mirror layer identifier recorded in the first mode with the mirror layer identifier recorded in the second mode, and determining whether the mirror layer data to be uploaded of the mirror layer to be uploaded is uploaded successfully according to the comparison result.
In one embodiment, the mirror warehouse compares the mirror layer identifier recorded in the first mode with the mirror layer identifier recorded in the second mode, if the mirror layer identifier recorded in the first mode is the same as the mirror layer identifier recorded in the second mode, it is determined that the data of the mirror layer to be uploaded on the mirror layer to be uploaded has been uploaded successfully, and the mirror warehouse is disconnected from the mirror storage system in communication, so that the resource loss of the system is reduced; if the mirror layer identification recorded in the first mode further comprises the mirror layer identification except the mirror layer identification recorded in the second mode, determining that the mirror layer data to be uploaded of the mirror layer to be uploaded is not successfully uploaded, and continuing to maintain the communication connection with the mirror storage system so as to continue to upload the mirror layer data.
In an optional implementation manner, the image warehouse compares the image layer identifier permanently recorded in the development file with the image layer identifier temporarily recorded in the local temporary file, if the image layer identifiers are the same, it is determined that the image layer data to be uploaded of the image layer to be uploaded is successfully uploaded, the image warehouse disconnects communication with the image storage system, and deletes the local temporary file, so as to avoid occupying local resources of the image warehouse.
S206, acquiring a data downloading instruction from the mirror image equipment, determining mirror image layer data to be downloaded in the mirror image storage system according to the data downloading instruction, and sending the mirror image layer data to be downloaded to the mirror image equipment.
According to the technical scheme of the embodiment, the mirror layer identification of the mirror layer to be uploaded is recorded in the mirror warehouse in the first mode, so that the effect of recording the mirror layer identification of all the mirror layers to be uploaded in the mirror file to be uploaded is achieved, the smooth image distribution of the subsequent mirror warehouse is ensured, and a foundation is laid for determining whether all the mirror layer data to be uploaded are successfully uploaded or not subsequently; the mirror layer identification of the mirror layer to be uploaded and the mirror layer data to be uploaded are stored in the mirror storage system in a key value pair correlation mode, so that the effect of storing objects of the mirror layer identification of the mirror layer to be uploaded and the mirror layer data to be uploaded is achieved, the metadata storage pressure of a mirror warehouse is reduced on the premise that mirror distribution is not influenced, and the effect of improving the mirror distribution efficiency of the mirror warehouse is achieved; under the condition that the data of the mirror layer to be uploaded of any mirror layer to be uploaded is successfully stored, the mirror layer identifier of the mirror layer to be uploaded is recorded in the mirror warehouse in a second mode, so that the effect of recording the mirror layer identifier of the successfully stored mirror layer to be uploaded is realized, and a foundation is laid for determining whether all the data of the mirror layer to be uploaded are successfully uploaded or not; and comparing the mirror layer identifier recorded in the first mode with the mirror layer identifier recorded in the second mode, and determining whether the mirror layer data to be uploaded of the mirror layer to be uploaded is uploaded successfully according to the comparison result, so that whether all the mirror layer data to be uploaded are successfully stored is verified, and all the mirror layer data to be uploaded of the mirror layer to be uploaded in the mirror file to be uploaded are successfully stored in the mirror storage system.
On the basis of the above embodiment, S206 optionally includes the following A, B, C and D steps:
A. acquiring a data downloading instruction from mirror equipment; and the data downloading instruction comprises a file identifier of the mirror image file to be downloaded.
In one implementation, any image device generates a data download instruction according to a file identifier of an image file to be downloaded, and an image warehouse acquires the data download instruction from the image device. The data downloading instruction comprises a pull instruction.
B. And determining the image layer identification of the image layer to be downloaded in the image file to be downloaded according to the file identification of the image file to be downloaded.
In one embodiment, the mirror repository performs conditional search according to a file identifier of a mirror file to be downloaded, and determines a mirror layer identifier of a mirror layer to be downloaded in the mirror file to be downloaded in the local storage of the mirror repository.
Optionally, step B includes:
determining a record label matched with the file identification of the mirror image file to be downloaded in the mirror image warehouse; and determining the mirror image layer identification of the mirror image layer to be downloaded in the mirror image file to be downloaded according to the record label.
In one implementation mode, the file identifier of the image file to be downloaded is used as a search condition, traversal search is performed in each record label, a record label which is the same as the file identifier of the image file to be downloaded is determined, and the record label is associated with the image layer identifier in the record content and used as the image layer identifier of the image layer to be downloaded in the image file to be downloaded.
In an optional implementation manner, a file identifier of an image file to be downloaded is used as a search condition, traversal search is performed in a development file tag, a development file tag identical to the file identifier of the image file to be downloaded is determined, the development file is analyzed, and an image layer identifier obtained through analysis is used as an image layer identifier of an image layer to be downloaded in the image file to be downloaded.
Determining a record label matched with a file identifier of an image file to be downloaded in an image warehouse; and determining the image layer identifier of the image layer to be downloaded in the image file to be downloaded according to the record label, thereby realizing the effect of determining the image layer identifier of the image layer to be downloaded in the image file to be downloaded, and laying a foundation for subsequently determining the image layer data to be downloaded.
C. And searching in the mirror image storage system by taking the mirror image layer identification of the mirror image layer to be downloaded as a search keyword, and determining a target keyword matched with the search keyword.
In one implementation, the mirror repository uses the mirror layer identifier of the mirror layer to be downloaded as a search key, searches among key value pairs stored in the mirror storage system, and determines a target key matched with the search key.
D. And taking the data value corresponding to the target keyword as the image layer data to be downloaded of the image layer to be downloaded, and sending the image layer data to be downloaded to the image device.
In one embodiment, the mirror warehouse uses the data value corresponding to the target keyword as the to-be-downloaded mirror layer data of the to-be-downloaded mirror layer according to the incidence relation between the target keyword and the corresponding data value, acquires the to-be-downloaded mirror layer data from the mirror storage system, and sends the acquired to-be-downloaded mirror layer data to the mirror device to complete the downloading of the mirror layer data.
The data downloading instruction is obtained from the mirror image equipment, the mirror image layer identification of the mirror image layer to be downloaded in the mirror image file to be downloaded is determined according to the file identification of the mirror image file to be downloaded, then the mirror image layer identification of the mirror image layer to be downloaded is used as a search keyword, searching is carried out in the mirror image storage system, a target keyword matched with the search keyword is determined, finally, a data value corresponding to the target keyword is used as the mirror image layer data to be downloaded of the mirror image layer to be downloaded, and the mirror image layer data to be downloaded is sent to the mirror image equipment.
Fig. 3 is a schematic structural diagram of a data transmission apparatus according to an embodiment of the present disclosure, which may be applied to a situation where a mirror warehouse stores mirror layer data uploaded by a mirror device. The device of the embodiment can be implemented by software and/or hardware, and can be integrated on any electronic equipment with computing capability.
As shown in fig. 3, the data transmission device 30 disclosed in this embodiment may include a data upload instruction acquisition module 31, an object storage module 32, and a data distribution module 33, where:
a data upload instruction obtaining module 31, configured to obtain a data upload instruction from the mirroring device; the data uploading instruction comprises a mirror image layer identifier of a mirror image layer to be uploaded in a mirror image file to be uploaded and mirror image layer data to be uploaded;
the object storage module 32 is used for storing the mirror layer identifier of the mirror layer to be uploaded and the mirror layer data to be uploaded into the mirror storage system in an object storage form in an associated manner;
the data distribution module 33 is configured to obtain a data download instruction from the mirroring device, determine, according to the data download instruction, mirroring layer data to be downloaded in the mirroring storage system, and send the mirroring layer data to be downloaded to the mirroring device.
Optionally, the object storage module 32 is specifically configured to:
storing the mirror layer identification of the mirror layer to be uploaded and the mirror layer data to be uploaded into a mirror storage system in a key value pair form in an associated manner; in the key value pair, the mirror layer identifier of the mirror layer to be uploaded is a keyword, and the data of the mirror layer to be uploaded is a data value.
Optionally, the apparatus further includes a first mode recording module, specifically configured to:
recording a mirror image layer identifier of a mirror image layer to be uploaded in a mirror image warehouse through a first mode;
the device also comprises a data uploading verification module, which is specifically used for:
under the condition that the data of the mirror layer to be uploaded of any mirror layer to be uploaded is successfully stored, recording the mirror layer identification of the mirror layer to be uploaded in a mirror warehouse through a second mode;
and comparing the mirror layer identifier recorded in the first mode with the mirror layer identifier recorded in the second mode, and determining whether the mirror layer data to be uploaded of the mirror layer to be uploaded is uploaded successfully according to the comparison result.
Optionally, the data uploading instruction further includes a file identifier of the mirror image file to be uploaded;
correspondingly, the first mode recording module is further specifically configured to:
and recording the mirror image layer identification of the mirror image layer to be uploaded in a mirror image warehouse in a first mode, and taking the file identification of the mirror image file to be uploaded as a recording label.
Optionally, the apparatus further includes a mirror layer identifier checking module, specifically configured to:
and rejecting the data uploading instruction under the condition that the mirror image layer identification of the mirror image layer to be uploaded is stored in the mirror image warehouse.
Optionally, the file identifier of the image file to be uploaded is obtained by performing hash calculation on total data of an image layer to be uploaded in the image file to be uploaded; and the mirror layer identification of the mirror layer to be uploaded is obtained by performing hash calculation on the mirror layer data of the mirror layer to be uploaded.
Optionally, the data distribution module 33 is specifically configured to:
acquiring a data downloading instruction from mirror equipment; the data downloading instruction comprises a file identifier of a mirror image file to be downloaded;
determining the mirror image layer identification of the mirror image layer to be downloaded in the mirror image file to be downloaded according to the file identification of the mirror image file to be downloaded;
taking the image layer identification of the image layer to be downloaded as a search keyword, searching in the image storage system, and determining a target keyword matched with the search keyword;
and taking the data value corresponding to the target keyword as the image layer data to be downloaded of the image layer to be downloaded, and sending the image layer data to be downloaded to the image device.
Optionally, the data distribution module 33 is further specifically configured to:
determining a record label matched with the file identification of the mirror image file to be downloaded in the mirror image warehouse;
and determining the mirror image layer identification of the mirror image layer to be downloaded in the mirror image file to be downloaded according to the record label.
The data transmission device 30 disclosed in the embodiment of the present disclosure can execute the data transmission method disclosed in the embodiment of the present disclosure, and has functional modules and beneficial effects corresponding to the execution method. Reference may be made to the description of any method embodiment of the disclosure for a matter not explicitly described in this embodiment.
The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.
FIG. 4 shows a schematic block diagram of an example electronic device 400 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 4, the apparatus 400 includes a computing unit 401 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)402 or a computer program loaded from a storage unit 408 into a Random Access Memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.
A number of components in device 400 are connected to I/O interface 405, including: an input unit 406 such as a keyboard, a mouse, or the like; an output unit 407 such as various types of displays, speakers, and the like; a storage unit 408 such as a magnetic disk, optical disk, or the like; and a communication unit 409 such as a network card, modem, wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), blockchain networks, and the internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.
The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.
Claims (15)
1. A method of data transmission, comprising:
acquiring a data uploading instruction from mirror image equipment; the data uploading instruction comprises a mirror image layer identifier of a mirror image layer to be uploaded in a mirror image file to be uploaded and mirror image layer data to be uploaded;
storing the mirror layer identification of the mirror layer to be uploaded and the mirror layer data to be uploaded into a mirror storage system in an associated manner through an object storage form;
acquiring a data downloading instruction from mirror equipment, determining mirror layer data to be downloaded in the mirror storage system according to the data downloading instruction, and sending the mirror layer data to be downloaded to the mirror equipment.
2. The method of claim 1, wherein the step of storing the mirror layer identifier of the mirror layer to be uploaded and the mirror layer data to be uploaded in a mirror storage system in an associated manner through an object storage form comprises the following steps:
storing the mirror layer identification of the mirror layer to be uploaded and the mirror layer data to be uploaded into a mirror storage system in a key value pair form in an associated manner; in the key value pair, the mirror layer identifier of the mirror layer to be uploaded is a keyword, and the data of the mirror layer to be uploaded is a data value.
3. The method according to claim 2, wherein before storing the mirror layer identifier of the mirror layer to be uploaded and the mirror layer data to be uploaded in the mirror storage system in an associated manner through an object storage form, the method further comprises:
recording a mirror image layer identifier of a mirror image layer to be uploaded in a mirror image warehouse through a first mode;
after the mirror image layer identifier of the mirror image layer to be uploaded and the mirror image layer data to be uploaded are stored in the mirror image storage system in an associated manner in an object storage form, the method further comprises the following steps:
under the condition that the data of the mirror layer to be uploaded of any mirror layer to be uploaded is successfully stored, recording the mirror layer identification of the mirror layer to be uploaded in a mirror warehouse through a second mode;
and comparing the mirror layer identifier recorded in the first mode with the mirror layer identifier recorded in the second mode, and determining whether the mirror layer data to be uploaded of the mirror layer to be uploaded is uploaded successfully according to the comparison result.
4. The method of claim 3, wherein the data uploading instruction further comprises a file identifier of an image file to be uploaded;
correspondingly, the mirror image layer identifier of the mirror image layer to be uploaded is recorded in the mirror image warehouse in a first mode, and the method comprises the following steps:
and recording the mirror image layer identification of the mirror image layer to be uploaded in a mirror image warehouse in a first mode, and taking the file identification of the mirror image file to be uploaded as a recording label.
5. The method of claim 1, wherein after the obtaining the data upload instruction from the mirroring device, further comprising:
and rejecting the data uploading instruction under the condition that the mirror image layer identification of the mirror image layer to be uploaded is stored in the mirror image warehouse.
6. The method according to claim 4, wherein the file identifier of the image file to be uploaded is obtained by performing hash calculation on total data of an image layer to be uploaded in the image file to be uploaded; and the mirror layer identification of the mirror layer to be uploaded is obtained by performing hash calculation on the mirror layer data of the mirror layer to be uploaded.
7. The method of claim 4, wherein obtaining a data download instruction from a mirroring device, determining mirroring layer data to be downloaded in the mirroring storage system according to the data download instruction, and sending the mirroring layer data to be downloaded to the mirroring device comprises:
acquiring a data downloading instruction from mirror equipment; the data downloading instruction comprises a file identifier of a mirror image file to be downloaded;
determining the mirror image layer identification of the mirror image layer to be downloaded in the mirror image file to be downloaded according to the file identification of the mirror image file to be downloaded;
taking the image layer identification of the image layer to be downloaded as a search keyword, searching in the image storage system, and determining a target keyword matched with the search keyword;
and taking the data value corresponding to the target keyword as the image layer data to be downloaded of the image layer to be downloaded, and sending the image layer data to be downloaded to the image device.
8. The method of claim 7, wherein determining the image layer identifier of the image layer to be downloaded in the image file to be downloaded according to the file identifier of the image file to be downloaded comprises:
determining a record label matched with the file identification of the mirror image file to be downloaded in the mirror image warehouse;
and determining the mirror image layer identification of the mirror image layer to be downloaded in the mirror image file to be downloaded according to the record label.
9. A data transmission apparatus comprising:
the data uploading instruction acquisition module is used for acquiring a data uploading instruction from the mirror image equipment; the data uploading instruction comprises a mirror image layer identifier of a mirror image layer to be uploaded in a mirror image file to be uploaded and mirror image layer data to be uploaded;
the object storage module is used for storing the mirror image layer identification of the mirror image layer to be uploaded and the mirror image layer data to be uploaded into the mirror image storage system in an associated manner through an object storage form;
and the data distribution module is used for acquiring a data downloading instruction from the mirror image equipment, determining mirror image layer data to be downloaded in the mirror image storage system according to the data downloading instruction, and sending the mirror image layer data to be downloaded to the mirror image equipment.
10. The apparatus of claim 9, wherein the object storage module is specifically configured to:
storing the mirror layer identification of the mirror layer to be uploaded and the mirror layer data to be uploaded into a mirror storage system in a key value pair form in an associated manner; in the key value pair, the mirror layer identifier of the mirror layer to be uploaded is a keyword, and the data of the mirror layer to be uploaded is a data value.
11. The apparatus according to claim 10, wherein the apparatus further comprises a first mode recording module, specifically configured to:
recording a mirror image layer identifier of a mirror image layer to be uploaded in a mirror image warehouse through a first mode;
the device also comprises a data uploading verification module, which is specifically used for:
under the condition that the data of the mirror layer to be uploaded of any mirror layer to be uploaded is successfully stored, recording the mirror layer identification of the mirror layer to be uploaded in a mirror warehouse through a second mode;
and comparing the mirror layer identifier recorded in the first mode with the mirror layer identifier recorded in the second mode, and determining whether the mirror layer data to be uploaded of the mirror layer to be uploaded is uploaded successfully according to the comparison result.
12. The apparatus of claim 11, wherein the data upload instruction further comprises a file identifier of an image file to be uploaded;
correspondingly, the first mode recording module is further specifically configured to:
and recording the mirror image layer identification of the mirror image layer to be uploaded in a mirror image warehouse in a first mode, and taking the file identification of the mirror image file to be uploaded as a recording label.
13. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.
14. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-8.
15. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011525520.6A CN112714163B (en) | 2020-12-22 | 2020-12-22 | Data transmission method, device, electronic equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011525520.6A CN112714163B (en) | 2020-12-22 | 2020-12-22 | Data transmission method, device, electronic equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112714163A true CN112714163A (en) | 2021-04-27 |
| CN112714163B CN112714163B (en) | 2022-06-10 |
Family
ID=75545029
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011525520.6A Active CN112714163B (en) | 2020-12-22 | 2020-12-22 | Data transmission method, device, electronic equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112714163B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113918096A (en) * | 2021-10-21 | 2022-01-11 | 城云科技（中国）有限公司 | Method and device for uploading algorithm mirror image packet and application |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106445473A (en) * | 2015-08-06 | 2017-02-22 | 华为技术有限公司 | Container deployment method and apparatus |
| CN106487850A (en) * | 2015-08-29 | 2017-03-08 | 华为技术有限公司 | The methods, devices and systems of mirror image are obtained under a kind of cloud environment |
| CN107431720A (en) * | 2015-12-31 | 2017-12-01 | 华为技术有限公司 | Mirror image dispositions method and device |
| CN107426258A (en) * | 2016-05-23 | 2017-12-01 | 华为技术有限公司 | A kind of upload of image file and method for down loading and device |
| US20180309807A1 (en) * | 2012-05-31 | 2018-10-25 | Leading Market Technologies, Inc. | Apparatus and Method for Acquiring, Managing, Sharing, Monitoring, Analyzing and Publishing Web-Based Time Series Data |
| US10146788B1 (en) * | 2014-10-10 | 2018-12-04 | Google Llc | Combined mirroring and caching network file system |
| CN109358857A (en) * | 2018-08-22 | 2019-02-19 | 华为技术有限公司 | Mirror image building, mirrored storage, mirror image distribution method and device |
| CN110912955A (en) * | 2018-09-17 | 2020-03-24 | 阿里巴巴集团控股有限公司 | Container mirror image downloading and uploading method and device |
| CN112084006A (en) * | 2020-09-10 | 2020-12-15 | 杭州海康威视系统技术有限公司 | Mirror image packet processing method and device and electronic equipment |
| CN112084165A (en) * | 2019-06-12 | 2020-12-15 | 阿里巴巴集团控股有限公司 | Method and device for managing mirror image warehouse, electronic equipment and readable storage medium |
-
2020
- 2020-12-22 CN CN202011525520.6A patent/CN112714163B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180309807A1 (en) * | 2012-05-31 | 2018-10-25 | Leading Market Technologies, Inc. | Apparatus and Method for Acquiring, Managing, Sharing, Monitoring, Analyzing and Publishing Web-Based Time Series Data |
| US10146788B1 (en) * | 2014-10-10 | 2018-12-04 | Google Llc | Combined mirroring and caching network file system |
| CN106445473A (en) * | 2015-08-06 | 2017-02-22 | 华为技术有限公司 | Container deployment method and apparatus |
| CN106487850A (en) * | 2015-08-29 | 2017-03-08 | 华为技术有限公司 | The methods, devices and systems of mirror image are obtained under a kind of cloud environment |
| CN107431720A (en) * | 2015-12-31 | 2017-12-01 | 华为技术有限公司 | Mirror image dispositions method and device |
| CN107426258A (en) * | 2016-05-23 | 2017-12-01 | 华为技术有限公司 | A kind of upload of image file and method for down loading and device |
| CN109358857A (en) * | 2018-08-22 | 2019-02-19 | 华为技术有限公司 | Mirror image building, mirrored storage, mirror image distribution method and device |
| CN110912955A (en) * | 2018-09-17 | 2020-03-24 | 阿里巴巴集团控股有限公司 | Container mirror image downloading and uploading method and device |
| CN112084165A (en) * | 2019-06-12 | 2020-12-15 | 阿里巴巴集团控股有限公司 | Method and device for managing mirror image warehouse, electronic equipment and readable storage medium |
| CN112084006A (en) * | 2020-09-10 | 2020-12-15 | 杭州海康威视系统技术有限公司 | Mirror image packet processing method and device and electronic equipment |
Non-Patent Citations (3)
| Title |
|---|
| B. HALEVY等: "Parallel NFS (pNFS) Flexible File Layout draft-ietf-nfsv4-flex-files-19.txt", 《IETF 》, 3 May 2018 (2018-05-03) * |
| 陆志刚等: "基于分片复用的多版本容器镜像加载方法", 《软件学报》 * |
| 陆志刚等: "基于分片复用的多版本容器镜像加载方法", 《软件学报》, no. 06, 8 June 2020 (2020-06-08) * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113918096A (en) * | 2021-10-21 | 2022-01-11 | 城云科技（中国）有限公司 | Method and device for uploading algorithm mirror image packet and application |
| CN113918096B (en) * | 2021-10-21 | 2023-09-22 | 城云科技（中国）有限公司 | Method, device and application for uploading algorithm mirror image package |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112714163B (en) | 2022-06-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112905537B (en) | File processing method and device, electronic equipment and storage medium | |
| CN103108033A (en) | File uploading method and system | |
| CN115242731A (en) | Message processing method, device, equipment and storage medium | |
| CN112714163B (en) | Data transmission method, device, electronic equipment and medium | |
| CN112671892B (en) | Data transmission method, device, electronic equipment and medium | |
| CN112433757A (en) | Method and device for determining interface calling relationship | |
| EP4092544A1 (en) | Method, apparatus and storage medium for deduplicating entity nodes in graph database | |
| CN115454971A (en) | Data migration method and device, electronic equipment and storage medium | |
| CN113326038B (en) | Method, apparatus, device, storage medium and program product for providing service | |
| CN115328736A (en) | Probe deployment method, device, equipment and storage medium | |
| CN115617800A (en) | Data reading method and device, electronic equipment and storage medium | |
| CN114912111A (en) | Cloud host virus detection method, device and system | |
| CN114897666A (en) | Graph data storage, access, processing method, training method, device and medium | |
| CN114172998A (en) | File transmission method, file transmission device, electronic equipment, medium and product | |
| CN114579520B (en) | Mirror image data processing method, mirror image data loading method and device | |
| CN112784596A (en) | Method and device for identifying sensitive words | |
| CN113360689B (en) | Image retrieval system, method, related device and computer program product | |
| CN112527802B (en) | Soft link method and device based on key value database | |
| CN113434508B (en) | Method and apparatus for storing information | |
| CN113268987B (en) | Entity name recognition method and device, electronic equipment and storage medium | |
| CN112783507B (en) | Data stream guiding playback method and device, electronic equipment and readable storage medium | |
| CN113946702A (en) | Image feature storage method and device, storage medium and electronic equipment | |
| CN115801763A (en) | File transmission method and device, electronic equipment and storage medium | |
| CN116303258A (en) | Method and device for constructing hash index of distributed storage system | |
| CN115858472A (en) | Data processing method, device, server and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |