CN115510163A - Mirror image file synchronization method and device, storage medium and electronic equipment - Google Patents

Mirror image file synchronization method and device, storage medium and electronic equipment Download PDF

Info

Publication number
CN115510163A
CN115510163A CN202211202855.3A CN202211202855A CN115510163A CN 115510163 A CN115510163 A CN 115510163A CN 202211202855 A CN202211202855 A CN 202211202855A CN 115510163 A CN115510163 A CN 115510163A
Authority
CN
China
Prior art keywords
image file
mirror image
target
file
container
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211202855.3A
Other languages
Chinese (zh)
Inventor
宋小妹
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Inspur Intelligent Technology Co Ltd
Original Assignee
Suzhou Inspur Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Inspur Intelligent Technology Co Ltd filed Critical Suzhou Inspur Intelligent Technology Co Ltd
Priority to CN202211202855.3A priority Critical patent/CN115510163A/en
Publication of CN115510163A publication Critical patent/CN115510163A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application provides a method, a device, a storage medium and an electronic device for synchronizing mirror image files, wherein the method comprises the following steps: responding to a synchronization instruction, and synchronizing the mirror image file to be synchronized from a first container mirror image warehouse to a second container mirror image warehouse, wherein the first container mirror image warehouse and the second container mirror image warehouse belong to the same type of mirror image warehouse; traversing the image file of the second container image warehouse and the image file in the target database of the image file which can be identified by the storage inference platform, and determining the image file with difference between the second container image warehouse and the target database according to the traversal result to obtain the target image file; and synchronizing the image file of the second container image warehouse to the target database based on the target image file so that the inference platform provides service to the target object by using the image file of the target database. Through the method and the device, the problem of low synchronization efficiency of the mirror image file in the related technology is solved, and the effect of improving the synchronization efficiency of the mirror image file is achieved.

Description

Method and device for synchronizing mirror image files, storage medium and electronic equipment
Technical Field
The embodiment of the application relates to the field of computers, in particular to a method and a device for synchronizing an image file, a storage medium and electronic equipment.
Background
Currently, a mirror repository may be used to store mirror files, and there may be a need for mirror synchronization between multiple mirror repositories. In the related art, the following two schemes are generally adopted to implement synchronization of image files between different image repositories:
in the first scheme, a data mirror backup tool RSYNC provided by Linux is used for realizing mirror image file synchronization between two mirror image warehouses.
In a second scheme, for a scenario in which an ifas (Infrastructure as a Service) Service is used for mirror storage, synchronization of a mirror file is configured by using a configuration tool of the ifas.
As can be seen from the above, the two ways need to adopt different tool strategies according to the storage environment where the mirror repository is located. In an interaction scenario between the image repository of the inference platform and the image repository of another platform or environment, as shown in fig. 1, it is generally necessary to store the image files of the other image repository locally from the server side, and upload the locally stored image files to the image repository of the inference platform (i.e., the image transfer in fig. 1). Then, the mirror image file in the mirror image warehouse of the inference platform is synchronized to the mirror image database of the inference platform, so that the inference platform can read and use the mirror image file from the mirror image database.
Generally, the image files trained by the training platform may reach 10G or more, the synchronization process of the image files will be long, and the synchronization efficiency of the image files is low. Moreover, in the process of synchronizing the mirror image file to the local or uploading the mirror image file to the reasoning platform, if any error occurs, resynchronization is needed, and the synchronization efficiency of the mirror image file is further reduced.
Disclosure of Invention
The embodiment of the application provides a method and a device for synchronizing mirror image files, a storage medium and electronic equipment, which are used for at least solving the problem of low efficiency of synchronizing the mirror image files in the related art.
According to an embodiment of the present application, there is provided a method for synchronizing an image file, including: responding to a synchronization instruction, and synchronizing the mirror image file to be synchronized from a first container mirror image warehouse to a second container mirror image warehouse, wherein the first container mirror image warehouse is used for storing the mirror image file generated by the training platform for a preset training scene, the second container mirror image warehouse is used for storing the mirror image file required by the reasoning platform for providing service for the target object, and the first container mirror image warehouse and the second container mirror image warehouse belong to the same type of mirror image warehouse; traversing the image files in the second container image warehouse and traversing the image files in a target database to obtain a traversal result, wherein the target database is used for storing the image files which can be identified by the inference platform; determining an image file with a difference between a second container image warehouse and a target database according to the traversal result to obtain a target image file; and synchronizing the image file in the second container image warehouse to the target database based on the target image file so that the inference platform provides service to the target object by using the image file in the target database.
In an exemplary embodiment, the method for synchronizing the image file further includes: analyzing the synchronous instruction, and determining the item identification of the source item, the item identification of the target item and the user information of a target user, wherein the target user is a user sending the synchronous instruction; determining a first container mirror image warehouse according to the project identification of the source project, and determining a second container mirror image warehouse according to the project identification of the target project; establishing connection between a first container mirror image warehouse and a second container mirror image warehouse according to user information of a target user, and determining synchronization authority of the target user, wherein the synchronization authority represents whether the target user can access a mirror image file in the first container mirror image warehouse; and when the synchronization authority of the target user reaches the target authority, synchronizing the mirror image file to be synchronized from the first container mirror image warehouse to the second container mirror image warehouse.
In an exemplary embodiment, the method for synchronizing the image file further includes: after the mirror image file to be synchronized is synchronized from the first container mirror image warehouse to the second container mirror image warehouse, when the mirror image file in the first container mirror image warehouse is updated, the updated mirror image file is synchronized from the first container mirror image warehouse to the second container mirror image warehouse.
In an exemplary embodiment, the method for synchronizing the image file further includes: acquiring a target file identifier and a target file size of a first image file in the second container image warehouse according to the traversal result; when the target database does not have a mirror image file with the file identifier which is the same as the target file identifier of the first mirror image file and the file size which is the same as the target file size of the first mirror image file, determining that the first mirror image file is a first target mirror image file; acquiring a target file identifier and a target file size of a second mirror image file in the target database according to the traversal result; and when the second container mirror image warehouse does not have a mirror image file with the same file identifier as the target file identifier of the second mirror image file and the same file size as the target file of the second mirror image file, determining that the second mirror image file is the second target mirror image file.
In an exemplary embodiment, the method for synchronizing the image file further includes: when the target image file is the first target image file, acquiring service information of the service provided by the reasoning platform, a file identifier of the first target image file and image synchronization information of the first target image file; generating target data based on the service information, the file identification of the first target image file and the image synchronization information; and synchronizing the target data and the first target image file to the target database.
In an exemplary embodiment, the method for synchronizing the image file further includes: and when the target image file is the second target image file, deleting the second target image file from the target database.
In an exemplary embodiment, the method for synchronizing the image file further includes: when detecting that the mirror image file in the second container mirror image warehouse is updated, acquiring the updated mirror image file; and synchronizing the updated image file to the first container image warehouse.
In an exemplary embodiment, the method for synchronizing the image file further includes: after the mirror image file in the second container mirror image warehouse is synchronized to the target database based on the target mirror image file, responding to a platform synchronization instruction, and synchronizing the mirror image file corresponding to the first reasoning platform to the container mirror image warehouse corresponding to the second reasoning platform, wherein the container mirror image warehouse of the first reasoning platform is the same as the container mirror image warehouse of the second reasoning platform.
According to another embodiment of the present application, there is provided a synchronization apparatus for an image file, including: the system comprises a first synchronization module, a second synchronization module and a third synchronization module, wherein the first synchronization module is used for responding to a synchronization instruction and synchronizing a mirror image file to be synchronized from a first container mirror image warehouse to a second container mirror image warehouse, the first container mirror image warehouse is used for storing a mirror image file generated by a training platform for a preset training scene, the second container mirror image warehouse is used for storing a mirror image file required by an inference platform for providing service for a target object, and the first container mirror image warehouse and the second container mirror image warehouse belong to the same type of mirror image warehouse; the traversal module is used for traversing the image file in the second container image warehouse and traversing the image file in the target database to obtain a traversal result, wherein the target database is used for storing the image file which can be identified by the inference platform; the difference determining module is used for determining the image file with the difference between the second container image warehouse and the target database according to the traversal result to obtain a target image file; and the second synchronization module is used for synchronizing the mirror image file in the second container mirror image warehouse to the target database based on the target mirror image file so that the inference platform provides service for the target object by using the mirror image file in the target database.
According to yet another embodiment of the present application, there is also provided a computer-readable storage medium having a computer program stored therein, wherein the computer program is configured to execute the above-mentioned synchronization method for image files when running.
According to yet another embodiment of the present application, there is also provided an electronic device including a memory and a processor, the memory storing a computer program, and the processor being configured to execute the computer program to perform the synchronization method of the image file described above.
Through the method and the device, because the container mirror image warehouse has the characteristic of copying, in the process of synchronizing the mirror image files of different container mirror image warehouses, the synchronization among the different container mirror image warehouses can be realized directly on the basis of the characteristic of copying of the container mirror image warehouse, the mirror image files do not need to be stored to the local, the mirror image files do not need to be uploaded to an inference platform, when the mirror image files are large, the synchronization among the different container mirror image warehouses can be quickly realized, and therefore the synchronization efficiency of the mirror image files is improved. In addition, in the application, in the process of synchronizing the mirror image files in the container mirror image warehouse of the inference platform to the mirror image database of the inference platform, the mirror image database is synchronized based on the difference of the mirror image files between the container mirror image warehouse and the mirror image database, and all the mirror image files in the container mirror image warehouse are not copied, so that in the process of synchronizing the container mirror image warehouse of the inference platform and the mirror image database of the inference platform, only the mirror image files with the difference need to be synchronized, and the synchronization efficiency of the mirror image files is further improved. Moreover, even if any error occurs in the synchronization process, the resynchronization does not greatly reduce the synchronization efficiency.
Therefore, the scheme provided by the application can solve the problem of low synchronization efficiency of the mirror image file in the related technology, and achieves the effect of improving the synchronization efficiency of the mirror image file.
Drawings
Fig. 1 is a flowchart of a mirror transfer method in the related art;
fig. 2 is a block diagram of a hardware structure of a mobile terminal according to an embodiment of the present application;
FIG. 3 is a flow chart of a method for synchronizing an image file according to an embodiment of the present application;
FIG. 4 is a flow diagram of mirror synchronization between a container mirror repository and a mirror database according to an embodiment of the present application;
FIG. 5 is a flow chart of mirror synchronization between a container mirror repository and a mirror database according to an embodiment of the present application;
FIG. 6 is a flow diagram of mirror synchronization between a container mirror repository and a mirror database according to an embodiment of the present application;
FIG. 7 is a block diagram of a synchronization apparatus for an image file according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
Embodiments of the present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the accompanying drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
The method embodiments provided in the embodiments of the present application may be executed in a mobile terminal, a computer terminal, or a similar computing device. Taking the example of running on a mobile terminal, fig. 2 is a block diagram of a hardware structure of the mobile terminal of a method for synchronizing an image file according to an embodiment of the present application. As shown in fig. 2, the mobile terminal may comprise one or more processors 202 (only one is shown in fig. 2) (the processor 202 may comprise, but is not limited to, a processing means such as a microprocessor MCU or a programmable logic device FPGA), and a memory 204 for storing data, wherein the mobile terminal may further comprise a transmission device 206 for communication functions and an input-output device 208. It will be understood by those skilled in the art that the structure shown in fig. 2 is only an illustration, and does not limit the structure of the mobile terminal. For example, the mobile terminal may also include more or fewer components than shown in FIG. 2, or have a different configuration than shown in FIG. 2.
The memory 204 can be used for storing computer programs, for example, software programs and modules of application software, such as a computer program corresponding to the synchronization method of the image file in the embodiment of the present application, and the processor 202 executes various functional applications and data processing by running the computer programs stored in the memory 204, so as to implement the above method. Memory 204 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 204 may further include memory located remotely from the processor 202, which may be connected to the mobile terminal through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 206 is used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 206 includes a Network adapter (NIC) that can be connected to other Network devices via a base station to communicate with the internet. In one example, the transmitting device 206 may be a Radio Frequency (RF) module, which is used to communicate with the internet via wireless.
In this embodiment, a method for synchronizing an image file running in the mobile terminal or the network architecture is provided, and fig. 3 is a flowchart of a method for synchronizing an image file according to an embodiment of the present application, and as shown in fig. 3, the flowchart includes the following steps:
step S302, responding to the synchronization instruction, and synchronizing the mirror image file to be synchronized from the first container mirror image warehouse to the second container mirror image warehouse.
In step S302, the first container mirror warehouse is configured to store mirror images generated by the training platform for a preset training scenario, the second container mirror warehouse is configured to store mirror images required by the inference platform to provide services for the target object, and the first container mirror warehouse and the second container mirror warehouse belong to mirror image warehouses of the same type, but the first container mirror warehouse is different from the second container mirror warehouse. The image file represents the logic of the service provided by the reasoning platform, for example, the image file can provide the reasoning platform with the logic of the shopping service. The target object may be an individual or an enterprise, and is not particularly limited in this application.
Optionally, in step S302, the container mirror warehouse may be, but is not limited to, a Harbor. Wherein, harbor is an open-source container mirror warehouse, harbor performs corresponding enterprise-level expansion on the basis of Docker Registry, so that Harbor has wider application, and these new enterprise-level characteristics include but are not limited to: management user interface, role-based Access control, AD (Active Directory)/LDAP (Lightweight Directory Access Protocol) integration, audit logs, and the like.
The training platform is a resource platform for Artificial Intelligence development facing to Artificial Intelligence enterprise training scenes, can realize functions of containerized deployment, visual development, centralized management and the like, can provide high-performance AI (Artificial Intelligence) computing resources for users, realizes efficient computing support, accurate resource management and scheduling, agile data integration and acceleration, and streamlined AI scene and service integration, effectively communicates development environments, computing resources and data resources, and improves development efficiency.
The inference platform can be but not limited to an artificial intelligence inference server platform, is mainly oriented to enterprise AI application deployment and online service management scenes, provides a reliable, easy-to-use and flexible inference service deployment and computing resource management platform for enterprises through the full stack AI capabilities of unified application interfaces, computational elastic expansion, A/B tests, rolling release, multi-model weighted evaluation and the like, helps users to quickly bring on-line AI services, improves the utilization efficiency of AI computing resources, and realizes quick landing of AI industries.
In an alternative embodiment, after receiving a synchronization instruction for synchronizing the image file sent by the user, the inference platform may determine the source image repository (i.e., the first container image repository) and the target image repository (i.e., the second container image repository) according to the synchronization instruction. Then, the inference platform can use the cross-data-warehouse mirror image remote copy function supported by the Harbor to realize the synchronization of the mirror image file between the source mirror image warehouse and the target mirror image warehouse.
It should be noted that, in the process of generating the image file by the training platform, a developer needs to frequently modify the image file, and once the training is completed, a stable image file is generated. At this time, the developer can put the trained environment into the actual production environment, i.e., the inference platform. At this time, the synchronization of the image file needs to go through the following steps: and storing the image file to the local from the server side, and uploading the image file to the reasoning platform from the local for use by the reasoning platform. However, in general, the image files trained by the training platform may reach 10G or more, the synchronization process of the image files will be very long, and if any error occurs in the process of copying to the local or uploading to the inference platform, the image files need to be re-synchronized, which increases the synchronization time of the image files.
In the application, since the hardor supports the cross-data-warehouse mirror image remote copy function, in the process of synchronizing the mirror image files across the mirror image warehouse, the mirror image files do not need to be stored locally, the local mirror images do not need to be copied to the inference platform, and when the mirror image files are large, synchronization among mirror image warehouses of different containers can be quickly realized, so that the synchronization efficiency of the mirror image files is improved. Even if any error occurs in the synchronization process, the resynchronization does not greatly reduce the synchronization efficiency.
And S304, traversing the image file in the second container image warehouse, and traversing the image file in a target database to obtain a traversal result, wherein the target database is used for storing the image file which can be identified by the inference platform.
In step S304, after the cross-repository synchronization of the container mirror repository is realized, the inference platform needs to synchronize the mirror files in the container mirror repository of the inference platform to the mirror database of the inference platform (i.e. the above target database), and then the inference platform can use the mirror files to provide services for the user or the enterprise.
In the process of synchronizing the mirror image files in the second container mirror image warehouse to the target database, the inference platform traverses the second container mirror image warehouse and the target database without copying all the mirror image files in the second container mirror image warehouse to the target database, determines the difference between the mirror image files in the second container mirror image warehouse and the mirror image files in the target database according to the traversal result, and then can realize the synchronization of the mirror image files in the second container mirror image warehouse according to the difference.
And S306, determining the mirror image file with the difference between the second container mirror image warehouse and the target database according to the traversal result to obtain the target mirror image file.
In step S306, the target image file may be an image file that the second container image repository has but the target database does not have, or an image file that the second container image repository does not have but the target database has.
It is easy to note that the number of the mirror image files having a difference between the second container mirror image warehouse and the target database is usually smaller than the number of the mirror image files of the second container mirror image warehouse, so that in the process of synchronizing the container mirror image warehouse of the inference platform and the mirror image database of the inference platform, the mirror image files having a difference are synchronized, and the synchronization efficiency of the mirror image files can be greatly reduced.
And step S308, synchronizing the mirror image file in the second container mirror image warehouse to the target database based on the target mirror image file, so that the inference platform provides service for the target object by using the mirror image file in the target database.
In step S308, the inference platform may determine a synchronization manner of the image file according to a file type corresponding to the target image file in a process of synchronizing the image file, where the file type corresponding to the target image file may include, but is not limited to, a first file type and a second file type, and the first file type is an image file that the second container image repository has but the target database does not have; the second file type is an image file that the second container image repository does not have, but the target database does. That is, the synchronization modes corresponding to different file types are different, for example, for the first file type, the corresponding synchronization mode is to copy the image file that the second container image warehouse has but the target database does not have into the target database; for the second file type, the corresponding synchronization mode is to delete the image file which the second container image warehouse does not have but the target database has from the target database.
In addition, in the process of synchronizing the mirror image files in the second container mirror image warehouse to the target database, the target database is synchronized based on the difference of the mirror image files between the second container mirror image warehouse and the target database, and all the mirror image files in the second container mirror image warehouse are not copied, so that in the process of synchronizing the second container mirror image warehouse and the target database, only the mirror image files with the difference need to be synchronized, and the synchronization efficiency of the mirror image files is further improved. Moreover, even if any error occurs in the synchronization process, the resynchronization does not greatly reduce the synchronization efficiency.
Based on the content defined in the above steps S302 to S308, since the container mirror image warehouse has a "copy" characteristic, in the process of synchronizing the mirror image files of different container mirror image warehouses, synchronization between different container mirror image warehouses can be realized directly based on the "copy" characteristic of the container mirror image warehouse without saving the mirror image files to the local or uploading the mirror image files to the inference platform, and when the mirror image files are large, synchronization between different container mirror image warehouses can be quickly realized, thereby improving synchronization efficiency of the mirror image files. In addition, in the application, in the process of synchronizing the mirror image files in the container mirror image warehouse of the inference platform to the mirror image database of the inference platform, the mirror image database is synchronized based on the difference of the mirror image files between the container mirror image warehouse and the mirror image database, and all the mirror image files in the container mirror image warehouse are not copied, so that in the process of synchronizing the container mirror image warehouse of the inference platform and the mirror image database of the inference platform, only the mirror image files with the difference need to be synchronized, and the synchronization efficiency of the mirror image files is further improved. Moreover, even if any error occurs in the synchronization process, the resynchronization does not greatly reduce the synchronization efficiency.
Therefore, the scheme provided by the application can solve the problem of low synchronization efficiency of the mirror image file in the related technology, and achieves the effect of improving the synchronization efficiency of the mirror image file.
It should be noted that, the execution subject of the above steps may be an inference platform, but is not limited thereto.
Note that, the following describes steps S302 to S308 in detail.
In an optional embodiment, the synchronization of the mirror image file includes two stages, namely a mirror image transfer stage and a database synchronization stage, wherein the mirror image transfer stage mainly implements synchronization of the mirror image file between the container mirror image warehouse of the inference platform and the container mirror image warehouse of another platform, and the other platform may be another inference platform or a training platform; and the database synchronization stage mainly realizes the synchronization of the mirror image file between the container mirror image warehouse (namely, the second container mirror image warehouse) of the inference platform and the mirror image database (namely, the target database) of the inference platform.
In the mirror image transfer stage, the reasoning platform responds to the synchronization instruction and synchronizes the mirror image file to be synchronized from the first container mirror image warehouse to the second container mirror image warehouse.
Specifically, the inference platform analyzes the synchronization instruction, determines a project identifier of the source project, a project identifier of the target project and user information of the target user, determines a first container mirror image warehouse according to the project identifier of the source project, and determines a second container mirror image warehouse according to the project identifier of the target project. And then, establishing the connection between the first container mirror image warehouse and the second container mirror image warehouse according to the user information of the target user, and determining the synchronization authority of the target user. And finally, when the synchronization authority of the target user reaches the target authority, the inference platform synchronizes the mirror image file to be synchronized from the first container mirror image warehouse to the second container mirror image warehouse.
In the foregoing process, the target user is a user who sends a synchronization instruction, and the synchronization permission indicates whether the target user can access the mirror image file in the first container mirror image warehouse, where the synchronization permission indicates whether the copy policy of Harbor is activated, that is, in this embodiment, only after a user with a certain permission sends a synchronization instruction, the copy policy of Harbor can be activated, so as to implement mirror image synchronization across container mirror image warehouses.
Optionally, the target user may determine, through the terminal device, an item identifier of the source item, an item identifier of the target item, and information (i.e., the user information) such as a user name and a password required for connection between the container mirror repository corresponding to the source item and the container mirror repository corresponding to the target item, and an address of a container mirror image of the container repository. After the target user sets the information, the terminal device can generate a synchronization instruction based on the information, and send the synchronization instruction to the reasoning platform, and then the reasoning platform analyzes the synchronization instruction to obtain the item identifier of the source item, the item identifier of the target item and the user information of the target user. The inference platform can determine the addresses of the container mirror image warehouse of the source item and the container mirror image warehouse of the target item according to the item identifier of the source item and the item identifier of the target item, and establish the connection between the two container mirror image warehouses based on the addresses of the two container mirror image warehouses.
After the connection between the two container mirror repositories, the inference platform makes an authority determination according to the user information of the target user, for example, the inference platform may determine whether the user can access the mirror file in the first container mirror repository according to the user name of the target user. When the user is determined to be able to access the mirror image file in the first container mirror image warehouse, the inference platform can determine that the copy strategy of the Harbor is activated, and at the moment, the inference platform copies all mirror image files under the source item of the Harbor to the container mirror image warehouse corresponding to the target item of the Harbor, so that cross-warehouse synchronization of the container mirror image files is realized.
It should be noted that since the Harbor supports the cross-data-warehouse mirror image remote copy function, in the process of synchronizing mirror image files across the mirror image warehouse, it is not necessary to store the mirror image files to the local, and it is also not necessary to copy the local mirror image to the inference platform, and when the mirror image files are large, synchronization between mirror image warehouses of different containers can be quickly realized, thereby improving the synchronization efficiency of the mirror image files. Even if any error occurs in the synchronization process, the resynchronization does not greatly reduce the synchronization efficiency.
Further, after the mirror image file to be synchronized is synchronized from the first container mirror image warehouse to the second container mirror image warehouse, when the mirror image file in the first container mirror image warehouse is updated, the inference platform synchronizes the updated mirror image file from the first container mirror image warehouse to the second container mirror image warehouse.
It should be noted that, since the Harbor has a function of mirroring remote copy across data warehouses, when an image file under a source item of the Harbor is added or deleted, as long as a copy policy is still in an active state, changes of the image file in the first container image warehouse can be automatically synchronized into the second container image warehouse, and manual operation by a user is not needed, so that updating of the image file across image warehouses is realized, and synchronization efficiency of the image file is improved.
In an alternative embodiment, during the database synchronization phase, the inference platform performs steps S304-S308 to implement mirror synchronization between the container mirror repository and the mirror database of the inference platform. At this stage, the inference platform can determine the synchronization mode of the image file according to the file type corresponding to the target image file.
When the file type is the first file type (namely the image file which the second container image warehouse has but the target database does not have), the inference platform traverses the image file in the second container image warehouse and the image file in the target database to obtain a traversal result, and obtains the target file identifier of the first image file in the second container image warehouse and the target file size of the first image file according to the traversal result. And when the target database does not have the mirror image file with the file identifier which is the same as the target file identifier of the first mirror image file and the file size which is the same as the target file size of the first mirror image file, determining that the first mirror image file is the first target mirror image file.
After the target mirror image file is determined to be the first target mirror image file, the reasoning platform acquires service information of service provided by the reasoning platform, a file identification of the first target mirror image file and mirror image synchronization information of the first target mirror image file, generates target data based on the service information, the file identification of the first target mirror image file and the mirror image synchronization information, and then synchronizes the target data and the first target mirror image file to a target database.
Optionally, fig. 4 shows a mirror synchronization process between the container mirror repository and the mirror database when the file type is the first file type (i.e. the mirror file that the second container mirror repository has but the target database does not have). As shown in fig. 4, the inference platform first traverses the image file in the second container image repository, traverses the image file in the target database, and detects whether there is an image file in the target database whose file identifier is the same as the target file identifier of the first image file and whose file size is the same as the target file size of the first image file. If the image file exists, the reasoning platform continues to compare the next image file in the second container image warehouse with the image file in the target database until the image file in the second container image warehouse is completely the same as the image file in the target database, or the target database does not have an image file with a file identifier which is the same as the target file identifier of the first image file, or the file size is the same as the target file size of the first image file.
In addition, as shown in fig. 4, when there exists a mirror image file in which the file identifier is the same as the target file identifier of the first mirror image file and the file size is the same as the target file size of the first mirror image file in the target database, the inference platform combines the file identifier of the target mirror image file (for example, the name of the mirror image file), the service information of the service provided by the inference platform, and the mirror image synchronization information of the target mirror image file according to a data format that can be recognized by the inference platform to obtain target data, inserts the target data into the target database of the inference platform, and synchronizes the first mirror image file to the target database of the inference platform, so that not only the first mirror image file but also relevant information of the first mirror image file are stored in the target database of the inference platform.
It should be noted that the service information of the service provided by the inference platform may include, but is not limited to, object information of an object receiving the service, such as an object name, a user group to which the object belongs, and the like, where the object information characterizes rights possessed by the object, and the rights characterize the service provided by the inference platform that the object can enjoy, for example, the inference platform provides services a and B to the object 1, that is, the object 1 can access the image files corresponding to the services a and B; the inference platform provides services a and C to object 2, i.e., object 2 can access the image files corresponding to services a and C.
In addition, the mirror synchronization information may include, but is not limited to, information such as a time when the mirror file is synchronized between the second container mirror repository and the target database, and a size of the target mirror file.
When the file type is a second file type (namely the second container mirror image warehouse does not have the mirror image file, but the target database has the mirror image file), the inference platform traverses the mirror image file in the second container mirror image warehouse and the mirror image file in the target database to obtain a traversal result, and obtains a target file identifier of the second mirror image file in the target database and a target file size of the second mirror image file according to the traversal result. And when the mirror image file with the file identifier same as the target file identifier of the second mirror image file does not exist in the second container mirror image warehouse, and the size of the mirror image file is the same as that of the target file of the second mirror image file, the reasoning platform determines that the second mirror image file is the second target mirror image file.
After determining the target image file, the inference platform deletes the second target image file from the target database.
Optionally, fig. 5 illustrates a mirror synchronization process between the container mirror repository and the mirror database when the file type is the second file type (i.e., the mirror file that the second container mirror repository does not have, but the target database does). As shown in fig. 5, the inference platform first traverses the image file in the second container image repository, traverses the image file in the target database, and detects whether there is an image file in the second container image repository whose file identifier is the same as the target file identifier of the second image file, and whose file size is the same as the target file size of the second image file. If the mirror image file exists, the reasoning platform continues to compare the next mirror image file in the target database with the mirror image file in the second container mirror image warehouse until the mirror image file in the target database is completely the same as the mirror image file in the second container mirror image warehouse, or the mirror image file which has the same file identifier as the target file identifier of the second mirror image file and the same file size as the target file of the second mirror image file does not exist in the second container mirror image warehouse.
In addition, as shown in fig. 5, when there is no image file in the second container image repository whose file identifier is the same as the target file identifier of the second image file and whose file size is the same as the target file size of the second image file, the inference platform deletes the second target image file and the related data corresponding to the second target image file from the target database.
It should be noted that, the method can ensure that the data of the user in the inference platform is not deleted and the mirror images of other items can be synchronized, thereby realizing the mirror image synchronization between the container mirror image warehouse and the mirror image database of the inference platform.
In addition, it should be noted that, in practical applications, two processing modes of the first file type and the second file type are required to be performed in a process that the inference platform passes through the mirror image file between the container mirror image warehouse and the mirror image database. For example, fig. 6 schematically shows a synchronization flow of a mirror file in which the processing method of the first file type is executed first and the processing method of the second file type is executed.
In an alternative embodiment, the user may slightly modify the scheme provided in this embodiment, and mirror synchronization between the inference platform and other training platforms may be implemented.
Specifically, when it is detected that the image file in the second container image warehouse is updated, the updated image file is obtained, and the updated image file is synchronized to the first container image warehouse. For example, in practical applications, the inference platform updates (e.g., modifies, deletes, or adds) the image file in the second container image repository according to actual needs, and at this time, the inference platform may synchronize the updated image file to the container image repository of the training platform (i.e., the first container image repository), so as to implement image synchronization between the first container image repository and the second container image repository.
It should be noted that the process of synchronizing the image from the inference platform to the other training platforms is similar to the process of synchronizing the image from the training platforms to the inference platform, and is not described herein again.
In another alternative embodiment, the scheme provided by the embodiment can also be applied to mirror image transfer between inference platforms. Specifically, after synchronizing the image file in the second container image warehouse to the target database based on the target image file, the inference platform responds to the platform synchronization instruction to synchronize the image file corresponding to the first inference platform to the container image warehouse corresponding to the second inference platform, wherein the container image warehouse of the first inference platform is the same as the container image warehouse of the second inference platform.
Optionally, after receiving the synchronization instruction, the source inference platform (i.e., the first inference platform) parses the synchronization instruction, so as to determine relevant information of the target inference platform (i.e., the second inference platform), for example, an address, a name, a mirror access permission, and the like of the target inference platform, and then the source inference platform determines the mirror image file to be synchronized according to the mirror access permission of the target inference platform, and synchronizes the mirror image file to be synchronized to the container mirror image repository of the target inference platform according to the address, the name, and the like of the target inference platform.
According to the scheme, the mirror image file can be synchronized to the reasoning platform in the training environment or the user testing environment, the mirror image file between the reasoning platforms can be synchronized, the operation is simple, the synchronization speed of the mirror image file is greatly improved, the scheme can be applied to the mirror image synchronization of the reasoning platform to other training platforms by slight modification, and the portability is high.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present application.
In this embodiment, a synchronization apparatus for an image file is further provided, and the apparatus is used to implement the foregoing embodiments and preferred embodiments, and details of which have been already described are not described again. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware or a combination of software and hardware is also possible and contemplated.
Fig. 7 is a block diagram of a synchronization apparatus for an image file according to an embodiment of the present application, where as shown in fig. 7, the synchronization apparatus includes: a first synchronization module 701, a traversal module 703, a difference determination module 705, and a second synchronization module 707.
The first synchronization module 701 is configured to respond to a synchronization instruction and synchronize a mirror image file to be synchronized from a first container mirror image warehouse to a second container mirror image warehouse, where the first container mirror image warehouse is configured to store a mirror image file generated by a training platform for a preset training scene, the second container mirror image warehouse is configured to store a mirror image file required by an inference platform to provide service for a target object, and the first container mirror image warehouse and the second container mirror image warehouse belong to the same type of mirror image warehouse; the traversal module 703 is configured to traverse the image file in the second container image warehouse and traverse the image file in the target database to obtain a traversal result, where the target database is used to store the image file that can be identified by the inference platform; the difference determining module 705 is configured to determine, according to the traversal result, an image file that is different between the second container image warehouse and the target database, to obtain a target image file; a second synchronization module 707, configured to synchronize the image file in the second container image repository to the target database based on the target image file, so that the inference platform provides a service to the target object by using the image file in the target database.
Optionally, the first synchronization module includes: the device comprises a first determination module, a second determination module, a third determination module and a third synchronization module. The first determining module is used for analyzing the synchronization instruction, and determining the item identifier of the source item, the item identifier of the target item and the user information of the target user, wherein the target user is a user sending the synchronization instruction; the second determining module is used for determining the first container mirror image warehouse according to the item identifier of the source item and determining the second container mirror image warehouse according to the item identifier of the target item; the third determining module is used for establishing the connection between the first container mirror image warehouse and the second container mirror image warehouse according to the user information of the target user and determining the synchronization authority of the target user, wherein the synchronization authority represents whether the target user can access the mirror image file in the first container mirror image warehouse or not; and the third synchronization module is used for synchronizing the mirror image file to be synchronized from the first container mirror image warehouse to the second container mirror image warehouse when the synchronization authority of the target user reaches the target authority.
Optionally, the synchronization apparatus for the image file further includes: and the fourth synchronization module is used for synchronizing the updated image file from the first container image warehouse to the second container image warehouse when the image file in the first container image warehouse is updated after the image file to be synchronized is synchronized from the first container image warehouse to the second container image warehouse.
Optionally, the difference determining module includes: the device comprises a first obtaining module, a fourth determining module, a third obtaining module and a fifth determining module. The first acquisition module is used for acquiring the target file identifier and the target file size of the first image file in the second container image warehouse according to the traversal result; the fourth determining module is used for determining the first mirror image file as a first target mirror image file when the target database does not have a mirror image file with the same file identifier as the target file identifier and the same file size as the target file; a third obtaining module and a fifth determining module; the third acquisition module is used for acquiring a target file identifier and a target file size of a second mirror image file in the target database according to the traversal result; and the fifth determining module is used for determining that the second mirror image file is the second target mirror image file when the second container mirror image warehouse does not have the mirror image file with the same file identifier as the target file identifier and the same file size as the target file.
Optionally, the second synchronization module includes: the device comprises a second acquisition module, a generation module and a fifth synchronization module. The second acquisition module is used for acquiring service information of the service provided by the reasoning platform, a file identifier of the target image file and image synchronization information of the target image file when the target image file is the first target image file; the generating module is used for generating target data based on the service information, the file identification of the target mirror image file and the mirror image synchronization information; and the fifth synchronization module is used for synchronizing the target data and the target mirror image file to the target database.
Optionally, the second synchronization module includes: and the file deleting module is used for deleting the target image file from the target database when the target image file is a second target image file.
Optionally, the synchronization apparatus for the image file further includes: a fourth obtaining module and a sixth synchronizing module. The fourth obtaining module is used for obtaining the updated mirror image file when detecting that the mirror image file in the second container mirror image warehouse is updated; and the sixth synchronization module is used for synchronizing the updated mirror image file to the first container mirror image warehouse.
Optionally, the synchronization apparatus for the image file further includes: and the seventh synchronization module is used for responding to the platform synchronization instruction after synchronizing the image file in the second container image warehouse to the target database based on the target image file, and synchronizing the image file corresponding to the first inference platform to the container image warehouse corresponding to the second inference platform, wherein the container image warehouse of the first inference platform is the same as the container image warehouse of the second inference platform.
It should be noted that the above modules may be implemented by software or hardware, and for the latter, the following may be implemented, but not limited to: the modules are all positioned in the same processor; alternatively, the modules are respectively located in different processors in any combination.
Embodiments of the present application further provide a computer-readable storage medium having a computer program stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.
In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to: various media capable of storing computer programs, such as a usb disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.
An embodiment of the present application further provides an electronic device, where fig. 8 is a schematic diagram of an alternative electronic device according to an embodiment of the present application, and as shown in fig. 8, includes a memory and a processor, where the memory stores a computer program, and the processor is configured to execute the computer program to perform the steps in any of the method embodiments described above.
In an exemplary embodiment, the electronic device may further include a transmission device and an input/output device (not shown in fig. 8), wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
For specific examples in this embodiment, reference may be made to the examples described in the foregoing embodiments and exemplary implementations, and details of this embodiment are not repeated herein.
It will be apparent to those skilled in the art that the various modules or steps of the present application described above may be implemented using a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and they may be implemented using program code executable by the computing devices, such that they may be stored in a memory device and executed by the computing devices, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into separate integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the principle of the present application shall be included in the protection scope of the present application.

Claims (11)

1. A method for synchronizing an image file, comprising:
responding to a synchronization instruction, and synchronizing a mirror image file to be synchronized from a first container mirror image warehouse to a second container mirror image warehouse, wherein the first container mirror image warehouse is used for storing a mirror image file generated by a training platform for a preset training scene, the second container mirror image warehouse is used for storing a mirror image file required by an inference platform for providing service for a target object, and the first container mirror image warehouse and the second container mirror image warehouse belong to the same type of mirror image warehouse;
traversing the image file in the second container image warehouse, and traversing the image file in a target database to obtain a traversal result, wherein the target database is used for storing the image file which can be identified by the inference platform;
determining a mirror image file with a difference between the second container mirror image warehouse and the target database according to the traversal result to obtain a target mirror image file;
and synchronizing the mirror image file in the second container mirror repository to the target database based on the target mirror image file, so that the inference platform provides services to the target object by using the mirror image file in the target database.
2. The method of claim 1, wherein synchronizing the image file to be synchronized from the first container mirror repository to the second container mirror repository in response to the synchronization instruction comprises:
analyzing the synchronous instruction, and determining the item identification of a source item, the item identification of a target item and the user information of a target user, wherein the target user is the user who sends the synchronous instruction;
determining the first container mirror image warehouse according to the project identifier of the source project, and determining the second container mirror image warehouse according to the project identifier of the target project;
establishing connection between the first container mirror image warehouse and the second container mirror image warehouse according to the user information of the target user, and determining the synchronization authority of the target user, wherein the synchronization authority represents whether the target user can access the mirror image file in the first container mirror image warehouse;
and when the synchronization authority of the target user reaches the target authority, synchronizing the mirror image file to be synchronized from the first container mirror image warehouse to the second container mirror image warehouse.
3. The method of claim 2, wherein after synchronizing the image file to be synchronized from the first container mirror repository to the second container mirror repository, the method further comprises:
and when the image file in the first container image warehouse is updated, synchronizing the updated image file from the first container image warehouse to the second container image warehouse.
4. The method of claim 1, wherein determining an image file that is different between the second container image repository and the target database according to the traversal result, and obtaining a target image file comprises:
acquiring a target file identifier of a first image file in the second container image warehouse and a target file size of the first image file according to the traversal result; when the target database does not have a mirror image file with the file identifier which is the same as the target file identifier of the first mirror image file and the file size which is the same as the target file size of the first mirror image file, determining that the first mirror image file is a first target mirror image file;
acquiring a target file identifier of a second image file in the target database and the size of the target file of the second image file according to the traversal result; and when the mirror image file with the file identifier which is the same as the target file identifier of the second mirror image file does not exist in the second container mirror image warehouse and the file size which is the same as the target file size of the second mirror image file, determining that the second mirror image file is the second target mirror image file.
5. The method of claim 4, wherein synchronizing the image file in the second container image repository into the target database based on the target image file comprises:
when the target image file is the first target image file, acquiring service information of the service provided by the reasoning platform, a file identifier of the first target image file and image synchronization information of the first target image file;
generating target data based on the service information, the file identification of the first target image file and the image synchronization information;
and synchronizing the target data and the first target image file to the target database.
6. The method of claim 4, wherein synchronizing the image file in the second container image repository into the target database based on the target image file comprises:
and deleting the second target image file from the target database when the target image file is the second target image file.
7. The method of claim 1, further comprising:
when detecting that the image file in the second container image warehouse is updated, acquiring the updated image file;
and synchronizing the updated image file to the first container image warehouse.
8. The method of claim 1, wherein after synchronizing the image file in the second container image repository into the target database based on the target image file, the method further comprises:
and responding to a platform synchronization instruction, and synchronizing the mirror image file corresponding to the first reasoning platform to the container mirror image warehouse corresponding to the second reasoning platform, wherein the container mirror image warehouse of the first reasoning platform is the same as the container mirror image warehouse of the second reasoning platform.
9. An apparatus for synchronizing an image file, comprising:
the system comprises a first synchronization module, a second synchronization module and a third synchronization module, wherein the first synchronization module is used for responding to a synchronization instruction and synchronizing a mirror image file to be synchronized from a first container mirror image warehouse to a second container mirror image warehouse, the first container mirror image warehouse is used for storing a mirror image file generated by a training platform for a preset training scene, the second container mirror image warehouse is used for storing a mirror image file required by an inference platform for providing service for a target object, and the first container mirror image warehouse and the second container mirror image warehouse belong to the same type of mirror image warehouse;
the traversal module is used for traversing the image file in the second container image warehouse and traversing the image file in a target database to obtain a traversal result, wherein the target database is used for storing the image file which can be identified by the inference platform;
the difference determining module is used for determining the image file with the difference between the second container image warehouse and the target database according to the traversal result to obtain a target image file;
and the second synchronization module is used for synchronizing the mirror image file in the second container mirror image warehouse to the target database based on the target mirror image file so that the inference platform provides service for the target object by using the mirror image file in the target database.
10. A computer-readable storage medium, in which a computer program is stored, wherein the computer program, when executed by a processor, implements the method for synchronizing the image file according to any one of claims 1 to 8.
11. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method for synchronizing an image file according to any one of claims 1 to 8 when executing the computer program.
CN202211202855.3A 2022-09-29 2022-09-29 Mirror image file synchronization method and device, storage medium and electronic equipment Pending CN115510163A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211202855.3A CN115510163A (en) 2022-09-29 2022-09-29 Mirror image file synchronization method and device, storage medium and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211202855.3A CN115510163A (en) 2022-09-29 2022-09-29 Mirror image file synchronization method and device, storage medium and electronic equipment

Publications (1)

Publication Number Publication Date
CN115510163A true CN115510163A (en) 2022-12-23

Family

ID=84508318

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211202855.3A Pending CN115510163A (en) 2022-09-29 2022-09-29 Mirror image file synchronization method and device, storage medium and electronic equipment

Country Status (1)

Country Link
CN (1) CN115510163A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116302210A (en) * 2023-05-17 2023-06-23 阿里云计算有限公司 Image file importing method and device, electronic equipment and storage medium
CN117077218A (en) * 2023-10-07 2023-11-17 之江实验室 Synchronization method, device, medium and equipment for mirror images of multi-center medical institution

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116302210A (en) * 2023-05-17 2023-06-23 阿里云计算有限公司 Image file importing method and device, electronic equipment and storage medium
CN116302210B (en) * 2023-05-17 2023-08-04 阿里云计算有限公司 Image file importing method and device, electronic equipment and storage medium
CN117077218A (en) * 2023-10-07 2023-11-17 之江实验室 Synchronization method, device, medium and equipment for mirror images of multi-center medical institution
CN117077218B (en) * 2023-10-07 2024-01-26 之江实验室 Synchronization method, device, medium and equipment for mirror images of multi-center medical institution

Similar Documents

Publication Publication Date Title
US11556366B2 (en) Container login method, apparatus, and storage medium
CN1988535B (en) Synchronous method, system for file storage and customer terminal
CN115510163A (en) Mirror image file synchronization method and device, storage medium and electronic equipment
US20130283267A1 (en) Virtual machine construction
CN102255866A (en) Method and device for downloading data
CN111756818B (en) File transfer method, device, equipment and storage medium
WO2017097181A1 (en) Data pushing method and apparatus
CN107704550A (en) File migration method, apparatus and computer-readable recording medium
CN111176706A (en) Terminal upgrading method and device, electronic equipment and storage medium
CN110018875B (en) Desktop synchronization implementation method based on IDV
CN114422538B (en) Multi-cloud storage system, multi-cloud data reading and writing method and electronic equipment
CN103685497A (en) On-line storing and sharing method and system
CN114895701B (en) Unmanned aerial vehicle inspection method and system
CN115617768A (en) Log management method, system, electronic device and storage medium
CN112738153B (en) Gateway selection method, system, device, server and medium in service system
EP3770761A1 (en) Software distribution in a wireless ad hoc network for ad-hoc data processing on a source node
CN106407320B (en) File processing method, device and system
CN113852919B (en) Method and device for generating early warning message, storage medium and electronic device
CN111404979B (en) Method and device for processing service request and computer readable storage medium
US20200401993A1 (en) Implementation of rolling key to identify systems inventories
CN116501552B (en) Data backup method, device, system and storage medium
CN117033325B (en) Mirror image file preheating and pulling method and device
CN116126785A (en) File acquisition method, device, system, storage medium and electronic equipment
WO2024129083A1 (en) Security scan with backup
CN111176716A (en) Upgrade package generation method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination