CN115510163A - Synchronization method, device, storage medium and electronic equipment for mirror image files - Google Patents

Abstract

The present application provides a method, device, storage medium and electronic equipment for synchronizing mirror files. The method includes: responding to a synchronization command, synchronizing the mirror files to be synchronized from the first container mirror warehouse to the second container mirror warehouse, the first The container image repository and the second container image repository belong to the same type of image repository; traverse the image files of the second container image repository and the image files in the target database that store the image files that can be recognized by the reasoning platform, and determine the second container image according to the traversal results There are differences between the image file between the warehouse and the target database to obtain the target image file; based on the target image file, the image file of the second container image warehouse is synchronized to the target database, so that the reasoning platform uses the image file of the target database to provide services to the target object . Through this application, the problem of low synchronization efficiency of mirrored files in related technologies is solved, and the effect of improving the synchronization efficiency of mirrored files is achieved.

Description

Synchronization method, device, storage medium and electronic equipment for mirror image files

technical field

The embodiments of the present application relate to the computer field, and in particular, relate to a method, device, storage medium, and electronic device for synchronizing image files.

Background technique

Currently, mirror warehouses can be used to store mirror files, and there may be a need for mirror synchronization among multiple mirror warehouses. In related technologies, the following two solutions are usually adopted to realize the synchronization of mirror files between different mirror warehouses:

The first solution is to use the data mirror backup tool RSYNC provided by Linux to realize the synchronization of mirror files between two mirror warehouses.

In the second solution, for the scenario of using IaaS (Infrastructure as a Service, infrastructure as a service) service for image storage, use the IaaS configuration tool to configure the synchronization of the image file.

From the above content, we can see that the above two methods need to adopt different tool strategies for the storage environment where the mirror warehouse is located. In the interaction scenario between the mirror warehouse of the inference platform and the mirror warehouse of other platforms or environments, as shown in Figure 1, it is usually necessary to save the mirror files of other mirror warehouses from the server to the local, and upload the locally saved mirror files to the image warehouse of the inference platform (that is, the image transfer in Figure 1). Then, the mirror file in the mirror warehouse of the reasoning platform is synchronized to the mirror database of the reasoning platform, so that the reasoning platform can read and use the mirror file from the mirror database.

Usually, the image files trained by the training platform may reach 10G or more, so the process of image file synchronization will be very long, and the synchronization efficiency of image files is relatively low. Moreover, in the process of synchronizing the image file locally or uploading it to the inference platform, if any error occurs, it needs to be re-synchronized, which further reduces the synchronization efficiency of the image file.

Contents of the invention

Embodiments of the present application provide a method, device, storage medium, and electronic device for synchronizing mirror files, so as to at least solve the problem of low synchronization efficiency of mirror files in related technologies.

According to an embodiment of the present application, a method for synchronizing image files is provided, including: responding to a synchronization instruction, synchronizing the image files to be synchronized from the first container image repository to the second container image repository, wherein the first container image The warehouse is used to store the image files generated by the training platform for preset training scenarios. The second container image warehouse is used to store the image files required by the inference platform to provide services for the target objects. The first container image warehouse and the second container image warehouse belong to Mirror warehouses of the same type; traverse the mirror files in the second container mirror warehouse, and traverse the mirror files in the target database to obtain the traverse results, wherein the target database is used to store mirror files that can be recognized by the reasoning platform; determine the first The mirror file in the second container mirror warehouse and the target database are different, and the target mirror file is obtained; based on the target mirror file, the mirror file in the second container mirror warehouse is synchronized to the target database, so that the reasoning platform uses the mirror image in the target database Documentation provides services to target audiences.

In an exemplary embodiment, the method for synchronizing mirror files further includes: parsing the synchronization instruction, determining the project identifier of the source project, the project identifier of the target project, and the user information of the target user, wherein the target user is the user who issued the synchronization instruction User; determine the first container mirror warehouse according to the project identification of the source project, determine the second container mirror warehouse according to the project identification of the target project; establish the connection between the first container mirror warehouse and the second container mirror warehouse according to the user information of the target user, and Determine the synchronization authority of the target user, wherein the synchronization authority represents whether the target user can access the image file in the first container image warehouse; when the synchronization authority of the target user reaches the target authority, the image file to be synchronized is transferred from the first container image warehouse Synchronize to the second container mirror repository.

In an exemplary embodiment, the method for synchronizing image files further includes: after synchronizing the image files to be synchronized from the first container image repository to the second container image repository, updating the image files in the first container image repository , the updated image file is synchronized from the first container image repository to the second container image repository.

In an exemplary embodiment, the method for synchronizing image files further includes: obtaining the target file identifier and target file size of the first image file in the second container mirror repository according to the traversal result; When there is no image file whose file ID is the same as the target file ID of the first image file and whose file size is the same as the target file size of the first image file, it is determined that the first image file is the first target image file; obtain the target file identifier and target file size of the second image file in the target database according to the traversal result; there is no file identifier and the target file of the second image file in the second container mirror warehouse When identifying the image files that are identical and have the same file size as the target file size of the second image file, it is determined that the second image file is the second target image file.

In an exemplary embodiment, the method for synchronizing image files further includes: when the target image file is the first target image file, acquiring service information of services provided by the reasoning platform, the first target image The file identifier of the file, the mirror synchronization information of the first target mirror file; generate target data based on the service information, the file identifier of the first target mirror file and the mirror synchronization information; The first target image file is synchronized to the target database.

In an exemplary embodiment, the method for synchronizing an image file further includes: when the target image file is the second target image file, deleting the second target image file from the target database.

In an exemplary embodiment, the method for synchronizing the image file further includes: when it is detected that there is an update of the image file in the second container mirror repository, obtaining the updated image file; synchronizing the updated image file to the first container in the mirror repository.

In an exemplary embodiment, the method for synchronizing image files further includes: after synchronizing the image files in the second container mirror warehouse to the target database based on the target image files, responding to the platform synchronization instruction, and synchronizing the The image file of the image is synchronized to the container image repository corresponding to the second inference platform, wherein the container image repository of the first inference platform is the same as the container image repository of the second inference platform.

According to another embodiment of the present application, a device for synchronizing image files is provided, including: a first synchronization module, configured to respond to a synchronization instruction and synchronize the image files to be synchronized from the first container image warehouse to the second container image Warehouse, wherein, the first container image warehouse is used to store the image files generated by the training platform for preset training scenarios, the second container image warehouse is used to store the image files required by the reasoning platform to provide services for target objects, and the first container image The warehouse and the second container mirror warehouse belong to the same type of mirror warehouse; the traversal module is used to traverse the image files in the second container mirror warehouse, and traverse the mirror files in the target database to obtain the traverse results, wherein the target database is used to store The image file that can be identified by the inference platform; the difference determination module is used to determine the image file that is different between the second container image warehouse and the target database according to the traversal result, and obtain the target image file; the second synchronization module is used to obtain the target image file based on the target image file Synchronize the image file in the second container image warehouse to the target database, so that the reasoning platform uses the image file in the target database to provide services to the target object.

According to yet another embodiment of the present application, a computer-readable storage medium is also provided, and a computer program is stored in the computer-readable storage medium, wherein the computer program is configured to execute the above-mentioned image file when running. synchronous method.

According to yet another embodiment of the present application, there is also provided an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the above-mentioned image file synchronization method.

Through this application, due to the "copy" feature of the container mirror warehouse, in the process of synchronizing the image files of different container mirror warehouses, the synchronization between different container mirror warehouses can be realized directly based on the "copy" feature of the container mirror warehouse. There is no need to save the image file locally or upload the image file to the inference platform. When the image file is relatively large, it can also quickly realize the synchronization between different container image warehouses, thereby improving the synchronization efficiency of the image file. In addition, in this application, in the process of synchronizing the image files in the container mirror warehouse of the inference platform to the mirror database of the inference platform, the mirror database is updated based on the difference between the mirror files in the container mirror warehouse and the mirror database. Synchronized, but will not copy all the image files in the container mirror warehouse. Therefore, in the process of synchronizing the container mirror warehouse of the inference platform and the mirror database of the inference platform, only the mirror files with differences need to be synchronized. Further improved the synchronization efficiency of image files. Moreover, even if any errors occur during the synchronization process, resynchronization will not significantly reduce the synchronization efficiency.

It can be seen that the solution provided by the present application can solve the problem of low mirror file synchronization efficiency in related technologies, and achieve the effect of improving the mirror file synchronization efficiency.

Description of drawings

Fig. 1 is the flowchart of the image transfer method in the related art;

Fig. 2 is the block diagram of the hardware structure of the mobile terminal according to the synchronization method of the image file of the embodiment of the application;

Fig. 3 is the flow chart of the method for synchronizing the image file according to the embodiment of the application;

FIG. 4 is a flow chart of mirror synchronization between a container mirror warehouse and a mirror database according to an embodiment of the present application;

FIG. 5 is a flow chart of mirror synchronization between a container mirror warehouse and a mirror database according to an embodiment of the present application;

FIG. 6 is a flow chart of mirror synchronization between a container mirror warehouse and a mirror database according to an embodiment of the present application;

Fig. 7 is the structural block diagram of the synchronization device of mirror image file according to the embodiment of the application;

Fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

detailed description

Embodiments of the present application will be described in detail below with reference to the drawings and in combination with the embodiments.

It should be noted that the terms "first" and "second" in the description and claims of the present application and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence.

The method embodiments provided in the embodiments of the present application may be executed in mobile terminals, computer terminals or similar computing devices. Taking running on a mobile terminal as an example, FIG. 2 is a block diagram of a hardware structure of a mobile terminal according to a method for synchronizing mirror image files according to an embodiment of the present application. As shown in Figure 2, the mobile terminal may include one or more (only one is shown in Figure 2) processors 202 (processor 202 may include but not limited to processing devices such as microprocessor MCU or programmable logic device FPGA, etc.) and a memory 204 for storing data, wherein the above-mentioned mobile terminal may also include a transmission device 206 and an input and output device 208 for communication functions. Those of ordinary skill in the art can understand that the structure shown in FIG. 2 is only for illustration, and it does not limit the structure of the above mobile terminal. For example, the mobile terminal may also include more or fewer components than those shown in FIG. 2, or have a different configuration from that shown in FIG.

The memory 204 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the mirror file synchronization method in the embodiment of the present application, and the processor 202 runs the computer program stored in the memory 204, thereby Executing various functional applications and data processing is to realize the above-mentioned method. The memory 204 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 204 may further include a memory that is remotely located relative to the processor 202, and these remote memories may be connected to the mobile terminal through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

Transmission device 206 is used to receive or transmit data via a network. The specific example of the above network may include a wireless network provided by the communication provider of the mobile terminal. In one example, the transmission device 206 includes a network interface controller (NIC for short), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 206 may be a radio frequency (Radio Frequency, RF for short) module, which is used to communicate with the Internet in a wireless manner.

In this embodiment, a method for synchronizing a mirror image file running on the above-mentioned mobile terminal or network architecture is provided. FIG. 3 is a flowchart of a method for synchronizing a mirror image file according to an embodiment of the present application. As shown in FIG. 3 , the process Including the following steps:

Step S302, responding to the synchronization command, synchronizing the image file to be synchronized from the first container mirror repository to the second container mirror repository.

In step S302, the first container image warehouse is used to store the image files generated by the training platform for preset training scenarios, the second container image warehouse is used to store the image files required by the inference platform to provide services for the target objects, and the first container The mirror warehouse and the second container mirror warehouse belong to the same type of mirror warehouse, but the first container mirror warehouse is different from the second container mirror warehouse. Wherein, the image file represents the logic of the service provided by the reasoning platform, for example, the image file may be the logic of the shopping service provided by the reasoning platform to the shopping platform. The above-mentioned target object may be an individual or an enterprise, which is not specifically limited in this application.

Optionally, in step S302, the container mirror repository may be, but not limited to, Harbor. Among them, Harbor is an open source container mirror warehouse. Harbor has carried out corresponding enterprise-level extensions on the basis of Docker Registry, so that Harbor has a wider range of applications. These new enterprise-level features include but are not limited to: management user interface , role-based access control, AD (Active Directory, Active Directory)/LDAP (Lightweight Directory Access Protocol, Lightweight Directory Access Protocol) integration and audit logs, etc.

The above-mentioned training platform is a resource platform for artificial intelligence development for artificial intelligence enterprise training scenarios. It can realize functions such as containerized deployment, visual development, and centralized management, and can provide users with extremely high-performance AI (Artificial Intelligence, Artificial intelligence) computing resources to achieve efficient computing power support, precise resource management and scheduling, agile data integration and acceleration, process-based AI scenarios and business integration, effectively open up the development environment, computing resources and data resources, and improve development efficiency .

The reasoning platform mentioned above can be, but not limited to, an artificial intelligence reasoning server platform, which is mainly oriented to enterprise AI application deployment and online service management scenarios, through unified application interface, elastic scaling of computing power, A/B testing, rolling release, and multi-model weighted evaluation And other full-stack AI capabilities, providing enterprises with a reliable, easy-to-use, and flexible inference service deployment and computing resource management platform, helping users quickly launch AI services, improving the utilization efficiency of AI computing resources, and realizing the rapid implementation of the AI industry.

In an optional embodiment, after the inference platform receives the synchronization instruction for synchronizing the image file sent by the user, it can determine the source mirror warehouse (that is, the first container mirror warehouse) and the target mirror warehouse (that is, the first container mirror warehouse) according to the synchronization instruction. Two container mirror warehouses). Then, the inference platform can use the cross-data warehouse mirror remote replication function supported by Harbor to realize the synchronization of mirror files between the source mirror warehouse and the target mirror warehouse.

It should be noted that during the process of generating image files on the training platform, developers need to frequently modify the image files. Once the training is completed, a stable image file will be generated. At this point, developers can put the trained environment into the actual production environment, which is the inference platform. At this time, the synchronization of the image file needs to go through the following steps: save the image file from the server to the local, and upload the image file from the local to the inference platform for use by the inference platform. Under normal circumstances, the image file trained by the training platform may reach 10G or more, so the synchronization process of the image file will be very long, and if any error occurs during the process of copying to the local or uploading to the inference platform, you need to restart Synchronize mirror files, increase the synchronization time of mirror files.

In this application, since Harbor supports the remote copy function of mirroring across data warehouses, during the process of synchronizing mirroring files across mirroring warehouses, there is no need to save the mirroring files locally, nor to copy the local mirroring to the inference platform. When the file is relatively large, it can also quickly realize the synchronization between different container image warehouses, thereby improving the synchronization efficiency of the image file. Even if there are any errors in the synchronization process, resynchronization will not significantly reduce the synchronization efficiency.

Step S304, traversing the image files in the second container image repository and traversing the image files in the target database to obtain a traversal result, wherein the target database is used to store image files that can be recognized by the reasoning platform.

In step S304, after realizing the cross-warehouse synchronization of the container mirror warehouse, the reasoning platform needs to synchronize the image files in the container mirror warehouse of the reasoning platform to the mirror database of the reasoning platform (that is, the above-mentioned target database), and then the reasoning platform can Use these image files to provide services to users or businesses.

In the process of synchronizing the image files in the second container mirror warehouse to the target database, the reasoning platform does not need to copy all the mirror files in the second container mirror warehouse to the target database, but performs a process on the second container mirror warehouse and the target database Traversing, determining the difference between the image file in the second container mirror repository and the mirror file in the target database according to the traverse result, and then realizing the synchronization of the mirror file in the second container mirror repository according to the difference.

Step S306, according to the traversal result, determine the image file that has a difference between the second container image repository and the target database, and obtain the target image file.

In step S306, the above-mentioned target image file may be an image file that the second container image repository has but that the target database does not have, or an image file that the second container image repository does not have but that the target database has.

It is easy to notice that the number of image files with differences between the second container image warehouse and the target database is usually smaller than the number of image files in the second container image warehouse. Therefore, the container image warehouse of the synchronous reasoning platform In the process of mirroring the database with the inference platform, synchronizing the mirror files with differences can greatly reduce the synchronization efficiency of the mirror files.

Step S308, based on the target image file, the image file in the second container image repository is synchronized to the target database, so that the reasoning platform uses the image file in the target database to provide services to the target object.

In step S308, in the process of synchronizing the image file, the reasoning platform can determine the synchronization method of the image file according to the file type corresponding to the target image file, wherein the file type corresponding to the target image file can include but not limited to the first The file type and the second file type, the first file type is an image file that the second container image repository has but the target database does not have; the second file type is an image file that the second container image repository does not have but the target database has . That is, different file types correspond to different synchronization methods. For example, for the first file type, the corresponding synchronization method is to copy the image files that the second container mirror warehouse has but not the target database to the target database ; For the second file type, the corresponding synchronization method is to delete from the target database the image file that the second container mirror repository does not have but that the target database has.

In addition, in the process of synchronizing the image file in the second container mirror warehouse to the target database, the target database is synchronized based on the difference between the mirror file in the second container mirror warehouse and the target database, and does not All the image files in the second container mirror warehouse are copied. Therefore, in the process of synchronizing the second container mirror warehouse with the target database, only the mirror files with differences need to be synchronized, which further improves the synchronization efficiency of mirror files. Moreover, even if any errors occur during the synchronization process, resynchronization will not significantly reduce the synchronization efficiency.

Based on the content defined in the above steps S302 to S308, it can be seen that since the container mirror warehouse has the "copy" feature, in the process of synchronizing the image files of different container mirror warehouses, it can be realized directly based on the "copy" feature of the container mirror warehouse. Synchronization between different container mirror repositories does not need to save the mirror file locally, nor upload the mirror file to the inference platform. When the mirror file is relatively large, it can also quickly realize the synchronization between different container mirror warehouses, thereby improving the image quality. File synchronization efficiency. In addition, in this application, in the process of synchronizing the image files in the container mirror warehouse of the inference platform to the mirror database of the inference platform, the mirror database is updated based on the difference between the mirror files in the container mirror warehouse and the mirror database. Synchronized, but will not copy all the image files in the container mirror warehouse. Therefore, in the process of synchronizing the container mirror warehouse of the inference platform and the mirror database of the inference platform, only the mirror files with differences need to be synchronized. Further improved the synchronization efficiency of image files. Moreover, even if any errors occur during the synchronization process, resynchronization will not significantly reduce the synchronization efficiency.

It can be seen that the solution provided by the present application can solve the problem of low mirror file synchronization efficiency in related technologies, and achieve the effect of improving the mirror file synchronization efficiency.

It should be noted that the execution subject of the above steps may be an inference platform, but is not limited thereto.

In addition, it should be noted that the above step S302 to step S308 will be described in detail below.

In an optional embodiment, the synchronization of image files includes two stages, namely the image transfer stage and the database synchronization stage, wherein the image transfer stage mainly realizes the connection between the container mirror warehouse of the inference platform and the container mirror warehouse of other platforms The other platforms can be other inference platforms or training platforms; the database synchronization stage mainly implements the container mirror warehouse of the inference platform (ie, the second container mirror warehouse) and the mirror database of the inference platform (ie, the target database). Synchronization of mirror files between.

In the image transfer phase, the reasoning platform responds to the synchronization command and synchronizes the image files to be synchronized from the first container image repository to the second container image repository.

Specifically, the reasoning platform analyzes the synchronization command, determines the project identifier of the source project, the project identifier of the target project, and the user information of the target user, and determines the first container mirror warehouse according to the project identifier of the source project, and determines the first container mirror warehouse according to the project identifier of the target project. The ID identifies the second container registry. Then, establish a connection between the first container mirror repository and the second container mirror repository according to the user information of the target user, and determine the synchronization permission of the target user. Finally, the reasoning platform synchronizes the image file to be synchronized from the first container mirror warehouse to the second container mirror warehouse when the synchronization authority of the target user reaches the target authority.

In the above process, the target user is the user who issued the synchronization command, and the synchronization permission indicates whether the target user can access the image file in the first container mirror repository, wherein the synchronization permission indicates whether Harbor's replication strategy is activated, that is, in this embodiment In , only users with certain permissions can activate Harbor's replication strategy after issuing a synchronization command to achieve mirror synchronization across container mirror warehouses.

Optionally, the target user can determine the project ID of the source project, the project ID of the target project, and the user name and password required for the connection between the container mirror warehouse corresponding to the source project and the container mirror warehouse corresponding to the target project through the terminal device And information such as the address of the container warehouse image (that is, the above-mentioned user information). After the target user sets the above information, the terminal device can generate a synchronization command based on the above information, and send the above synchronization command to the reasoning platform, and then the reasoning platform parses the synchronization command to obtain the project ID of the source project and the project of the target project Identification and user information of the target user. The inference platform can determine the address of the source project's container mirror repository and the address of the target project's container mirror repository according to the project identifier of the source project and the project identifier of the target project, and establish two container images based on the addresses of the two container mirror warehouses Links between warehouses.

After the connection between the two container mirror repositories, the reasoning platform determines the authority according to the user information of the target user. For example, the reasoning platform can determine whether the user can access the image in the first container mirror warehouse based on the user name of the target user. document. When it is determined that the user can access the image files in the first container mirror warehouse, the reasoning platform can determine that Harbor’s replication strategy is activated. At this time, the reasoning platform will copy all the image files under Harbor’s source project to Harbor’s target project In the corresponding container image warehouse, the synchronization of container image files across warehouses is realized.

It should be noted that since Harbor supports the remote replication function of cross-data warehouse images, there is no need to save the image files locally or copy the local images to the inference platform during the process of synchronizing image files across image warehouses. When it is relatively large, it can also quickly realize the synchronization between different container image warehouses, thereby improving the synchronization efficiency of image files. Even if there are any errors in the synchronization process, resynchronization will not significantly reduce the synchronization efficiency.

Further, after synchronizing the image file to be synchronized from the first container image repository to the second container image repository, when the image file in the first container image repository is updated, the reasoning platform will update the image file from the first container image repository The container mirror repository is synchronized to the second container mirror repository.

It should be noted that since Harbor has the function of remote replication of images across data warehouses, when image files under Harbor’s source project are added or deleted, as long as the replication strategy is still active, the images in the first container image warehouse The changes of the files will be automatically synchronized to the second container mirror warehouse without manual operation by the user, which realizes the update of mirror files across mirror warehouses and improves the synchronization efficiency of mirror files.

In an optional embodiment, during the database synchronization phase, the reasoning platform executes steps S304 to S308 to realize mirror synchronization between the container mirror warehouse of the reasoning platform and the mirror database. At this stage, the inference platform can determine the synchronization method of the image file according to the file type corresponding to the target image file.

When the file type is the first file type (that is, the image file that the second container image warehouse has, but the target database does not have), the reasoning platform traverses the image files in the second container image warehouse and the image files in the target database, The traversal result is obtained, and the target file identifier of the first image file in the second container mirror repository and the target file size of the first image file are obtained according to the traversal result. When there is no mirror file whose file ID is the same as the target file ID of the first mirror file and whose file size is the same as the target file size of the first mirror file, the first mirror file is determined to be the first target mirror file in the target database.

After determining that the target image file is the first target image file, the inference platform obtains the service information of the services provided by the inference platform, the file identifier of the first target image file, and the image synchronization information of the first target image file, and based on the service information, the first target image file The file identification of a target mirror image file and the mirror synchronization information generate target data, and then, the target data and the first target mirror file are synchronized to the target database.

Optionally, FIG. 4 shows that when the file type is the first file type (that is, the mirror file that the second container mirror warehouse has, but the target database does not have), the image synchronization process between the container mirror warehouse and the mirror database . As shown in Figure 4, the reasoning platform first traverses the image files in the second container image warehouse, and traverses the image files in the target database, and detects whether there is a file ID in the target database that is the same as the target file ID of the first image file, and An image file having the same file size as the target file size of the first image file. If it exists, the reasoning platform continues to compare the next image file in the second container image repository with the image file in the target database until the image file in the second container image repository is exactly the same as the image file in the target database, or, Until there is no mirror file whose file ID is the same as the target file ID of the first mirror file, or whose file size is the same as the target file size of the first mirror file, does not exist in the target database.

In addition, as shown in Figure 4, when there is an image file with the same file ID as the target file ID of the first image file and the same file size as the target file size of the first image file in the target database, the inference platform can The identified data format combines the file identification (for example, the name of the image file) of the target image file, the service information of the service provided by the inference platform, and the image synchronization information of the target image file to obtain the target data, and insert the target data into In the target database of the inference platform, the first image file is simultaneously synchronized to the target database of the inference platform, so that not only the first image file but also related information of the first image file are stored in the target database of the inference platform.

It should be noted that the service information of the service provided by the above reasoning platform may include but not limited to the object information of the object receiving the above service, for example, the name of the object, the user group to which the object belongs, etc., wherein the object information indicates that the object has The permissions represent the services provided by the inference platform that the object can enjoy. For example, the inference platform provides services A and B to object 1, that is, object 1 can access the image files corresponding to services A and B; the inference platform provides Object 2 provides services A and C, that is, object 2 can access the image files corresponding to services A and C.

In addition, the above-mentioned image synchronization information may include but not limited to information such as the time for synchronizing the image file between the second container mirror repository and the target database, the size of the target image file, and the like.

When the file type is the second file type (that is, the image file that the target database does not have in the second container image repository), the reasoning platform traverses the image files in the second container image repository and the image files in the target database to obtain Traversing the results, and obtaining the target file identifier of the second mirror image file and the target file size of the second mirror image file in the target database according to the traverse results. When there is no image file with the same file ID as the target file ID of the second image file and the same file size as the target file size of the second image file in the second container image warehouse, the reasoning platform determines that the second image file is the first image file Two target image files.

After the target image file is determined, the reasoning platform deletes the second target image file from the target database.

Optionally, FIG. 5 shows the image synchronization process between the container mirror repository and the mirror database when the file type is the second file type (that is, the image file that the second container mirror repository does not have, but the target database has). As shown in Figure 5, the inference platform first traverses the image files in the second container image warehouse, and traverses the image files in the target database, and detects whether there is a file identifier and the target file identifier of the second image file in the second container image warehouse The image files are the same, and the file size is the same as the target file size of the second image file. If it exists, the reasoning platform continues to compare the next image file in the target database with the image file in the second container image repository until the image file in the target database is exactly the same as the image file in the second container image repository, or, There is no image file whose file ID is the same as the target file ID of the second image file and whose file size is the same as the target file size of the second image file in the second container mirror repository.

In addition, as shown in Figure 5, when there is no image file whose file ID is the same as the target file ID of the second image file in the second container image warehouse, and the file size is the same as the target file size of the second image file, reasoning The platform deletes the second target image file and related data corresponding to the second target image file from the target database.

It should be noted that the above method can not only ensure that the user's original data in the inference platform will not be deleted, but also ensure that the mirroring of other projects can be synchronized, thus realizing the mirroring between the container mirror warehouse of the inference platform and the mirror database Synchronize.

In addition, it should be noted that in practical applications, the reasoning platform needs to perform the above-mentioned two processing methods of the first file type and the second file type during the process of passing the mirror file between the container mirror warehouse and the mirror database. The above-mentioned two processing methods of the first file type and the second file type can be performed simultaneously, and the execution order can also be set according to the requirements. In this embodiment, the order of the two processing methods is not specifically limited. For example, FIG. 6 schematically It shows the synchronization flow of the image file in which the processing mode of the first file type is executed first, and then the processing mode of the second file type is executed.

In an optional embodiment, the user can slightly modify the solution provided in this embodiment to achieve mirror synchronization between the inference platform and other training platforms.

Specifically, when it is detected that the image file in the second container image repository is updated, the updated image file is obtained, and the updated image file is synchronized to the first container image repository. For example, in a practical application, the reasoning platform updates (for example, modifies, deletes or adds) the image file in the second container image warehouse according to actual needs, and at this time, the reasoning platform can synchronize the updated image file to the training platform In the container mirror warehouse (that is, the first container mirror warehouse), the image synchronization between the first container mirror warehouse and the second container mirror warehouse is realized.

It should be noted that the image synchronization process between the inference platform and other training platforms is similar to the process of synchronizing images from the training platform to the inference platform, and will not be repeated here.

In another optional embodiment, the solution provided in this embodiment may also be applied to image transfer between inference platforms. Specifically, after synchronizing the image file in the second container image warehouse to the target database based on the target image file, the inference platform responds to the platform synchronization command and synchronizes the image file corresponding to the first inference platform to the image file corresponding to the second inference platform. In the container mirror warehouse of the first reasoning platform, the container mirror warehouse of the first reasoning platform is the same as the container mirror warehouse of the second reasoning platform.

Optionally, after receiving the synchronization instruction, the source reasoning platform (that is, the first reasoning platform) can analyze the synchronization instruction to determine the relevant information of the target reasoning platform (that is, the second reasoning platform), for example, the target reasoning platform address, name, mirror access rights, etc., and then the source inference platform determines the mirror files that need to be synchronized according to the mirror access rights of the target reasoning platform, and synchronizes the mirror files that need to be synchronized to the target reasoning platform based on the address and name of the target reasoning platform. in the container registry.

It can be seen from the above that the solution provided in this embodiment can realize the synchronization of image files from the training environment or the user test environment to the inference platform, and can also realize the synchronization of image files between inference platforms. The operation is simple and the synchronization of image files is greatly improved. Speed, and this scheme can be applied to the mirror synchronization of the inference platform to other training platforms with a slight modification, and the portability is strong.

Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product in essence or the part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM, disk, CD) contains several instructions to enable a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

In this embodiment, there is also provided a device for synchronizing mirror files, which is used to implement the above embodiments and preferred implementation modes, and what has been explained will not be repeated here. As used below, the term "module" may be a combination of software and/or hardware that realizes a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementations in hardware, or a combination of software and hardware are also possible and contemplated.

FIG. 7 is a structural block diagram of an apparatus for synchronizing mirrored files according to an embodiment of the present application. As shown in FIG.

Wherein, the first synchronization module 701 is used to respond to the synchronization command and synchronize the image file to be synchronized from the first container mirror warehouse to the second container mirror warehouse, wherein the first container mirror warehouse is used to store the training platform as preset training For the image file generated by the scene, the second container image warehouse is used to store the image file required by the inference platform to provide services for the target object. The first container image warehouse and the second container image warehouse belong to the same type of image warehouse; the traversal module 703, For traversing the image files in the second container mirror warehouse, and traversing the image files in the target database to obtain the traversal result, wherein the target database is used to store the image files that can be recognized by the inference platform; the difference determination module 705 is used for according to the traversal As a result, it is determined that there are differences in the image files between the second container image warehouse and the target database, and the target image file is obtained; the second synchronization module 707 is used to synchronize the image files in the second container image warehouse to the target database based on the target image file , so that the inference platform uses the image file in the target database to provide services to the target object.

Optionally, the first synchronization module includes: a first determination module, a second determination module, a third determination module, and a third synchronization module. Wherein, the first determination module is used to analyze the synchronization instruction, determine the project identification of the source project, the project identification of the target project and the user information of the target user, wherein the target user is the user who issued the synchronization instruction; the second determination module, It is used to determine the first container mirror warehouse according to the project identification of the source project, and determine the second container mirror warehouse according to the project identification of the target project; the third determination module is used to establish the first container mirror warehouse and the second container mirror warehouse according to the user information of the target user. The connection of the container mirror warehouse, and determine the synchronization authority of the target user, wherein the synchronization authority represents whether the target user can access the image file in the first container mirror warehouse; the third synchronization module is used to achieve the target authority after the synchronization authority of the target user , the mirror file to be synchronized is synchronized from the first container mirror warehouse to the second container mirror warehouse.

Optionally, the synchronization device of the image file also includes: a fourth synchronization module, which is used to synchronize the image file in the first container image repository after the image file to be synchronized is synchronized from the first container image repository to the second container image repository. When the file is updated, the updated image file is synchronized from the first container mirror warehouse to the second container mirror warehouse.

Optionally, the difference determining module includes: a first obtaining module, a fourth determining module, a third obtaining module and a fifth determining module. Wherein, the first obtaining module is used to obtain the target file identifier and the target file size of the first image file in the second container mirror warehouse according to the traversal result; the fourth determining module is used for not having the file identifier and the target file size in the target database. The file identification is the same, and when the image file with the same file size and the target file size, determine that the first image file is the first target image file; the third acquisition module and the fifth determination module; the third acquisition module, for according to the traversal result Obtain the target file identifier and the target file size of the second mirror image file in the target database; the fifth determination module is used for not having the same file identifier and the target file identifier in the second container image warehouse, and the file size is the same as the target file size When the image files are the same, it is determined that the second image file is the second target image file.

Optionally, the second synchronization module includes: a second acquisition module, a generation module and a fifth synchronization module. Wherein, the second obtaining module is used to obtain the service information of the service provided by the reasoning platform, the file identification of the target mirror file, and the mirror synchronization information of the target mirror file when the target mirror file is the first target mirror file; the generation module uses The target data is generated based on the service information, the file identification of the target mirror file and the mirror synchronization information; the fifth synchronization module is used to synchronize the target data and the target mirror file to the target database.

Optionally, the second synchronization module includes: a file deletion module, configured to delete the target image file from the target database when the target image file is the second target image file.

Optionally, the device for synchronizing the image file further includes: a fourth acquiring module and a sixth synchronizing module. Wherein, the fourth obtaining module is used to obtain the updated image file when it is detected that the image file in the second container image warehouse is updated; the sixth synchronization module is used to synchronize the updated image file to the first container in the mirror repository.

Optionally, the device for synchronizing the image file also includes: a seventh synchronization module, configured to respond to the platform synchronization instruction after synchronizing the image file in the second container image warehouse to the target database based on the target image file, and convert the first inference The image file corresponding to the platform is synchronized to the container mirror warehouse corresponding to the second reasoning platform, wherein the container mirror warehouse of the first reasoning platform is the same as the container mirror warehouse of the second reasoning platform.

It should be noted that the above-mentioned modules can be realized by software or hardware. For the latter, it can be realized by the following methods, but not limited to this: the above-mentioned modules are all located in the same processor; or, the above-mentioned modules can be combined in any combination The forms of are located in different processors.

Embodiments of the present application also provide a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to perform the steps in any one of the above method embodiments when running.

In an exemplary embodiment, the above-mentioned computer-readable storage medium may include but not limited to: U disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM) , mobile hard disk, magnetic disk or optical disk and other media that can store computer programs.

The embodiment of the present application also provides an electronic device, wherein, FIG. 8 is a schematic diagram of an optional electronic device according to the embodiment of the present application. As shown in FIG. 8 , it includes a memory and a processor, and the memory stores There is a computer program, and the processor is configured to run the computer program to perform the steps in any one of the above method embodiments.

In an exemplary embodiment, the electronic device may further include a transmission device and an input and output device (not shown in FIG. 8 ), wherein the transmission device is connected to the processor, and the input and output device is connected to the processor.

For specific examples in this embodiment, reference may be made to the examples described in the foregoing embodiments and exemplary implementation manners, and details will not be repeated here in this embodiment.

Obviously, those skilled in the art should understand that each module or each step of the above-mentioned application can be realized by a general-purpose computing device, and they can be concentrated on a single computing device, or distributed in a network composed of multiple computing devices In fact, they can be implemented in program code executable by a computing device, and thus, they can be stored in a storage device to be executed by a computing device, and in some cases, can be executed in an order different from that shown here. Or described steps, or they are fabricated into individual integrated circuit modules, or multiple modules or steps among them are fabricated into a single integrated circuit module for implementation. As such, the present application is not limited to any specific combination of hardware and software.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the present application. For those skilled in the art, there may be various modifications and changes in the present application. Any modifications, equivalent replacements, improvements, etc. made within the principles of this application shall be included within the scope of protection of this application.

Claims (11)

1. A method for synchronizing an image file, comprising:

responding to a synchronization instruction, and synchronizing a mirror image file to be synchronized from a first container mirror image warehouse to a second container mirror image warehouse, wherein the first container mirror image warehouse is used for storing a mirror image file generated by a training platform for a preset training scene, the second container mirror image warehouse is used for storing a mirror image file required by an inference platform for providing service for a target object, and the first container mirror image warehouse and the second container mirror image warehouse belong to the same type of mirror image warehouse;

traversing the image file in the second container image warehouse, and traversing the image file in a target database to obtain a traversal result, wherein the target database is used for storing the image file which can be identified by the inference platform;

determining a mirror image file with a difference between the second container mirror image warehouse and the target database according to the traversal result to obtain a target mirror image file;

and synchronizing the mirror image file in the second container mirror repository to the target database based on the target mirror image file, so that the inference platform provides services to the target object by using the mirror image file in the target database.

2. The method of claim 1, wherein synchronizing the image file to be synchronized from the first container mirror repository to the second container mirror repository in response to the synchronization instruction comprises:

analyzing the synchronous instruction, and determining the item identification of a source item, the item identification of a target item and the user information of a target user, wherein the target user is the user who sends the synchronous instruction;

determining the first container mirror image warehouse according to the project identifier of the source project, and determining the second container mirror image warehouse according to the project identifier of the target project;

establishing connection between the first container mirror image warehouse and the second container mirror image warehouse according to the user information of the target user, and determining the synchronization authority of the target user, wherein the synchronization authority represents whether the target user can access the mirror image file in the first container mirror image warehouse;

and when the synchronization authority of the target user reaches the target authority, synchronizing the mirror image file to be synchronized from the first container mirror image warehouse to the second container mirror image warehouse.

3. The method of claim 2, wherein after synchronizing the image file to be synchronized from the first container mirror repository to the second container mirror repository, the method further comprises:

and when the image file in the first container image warehouse is updated, synchronizing the updated image file from the first container image warehouse to the second container image warehouse.

4. The method of claim 1, wherein determining an image file that is different between the second container image repository and the target database according to the traversal result, and obtaining a target image file comprises:

acquiring a target file identifier of a first image file in the second container image warehouse and a target file size of the first image file according to the traversal result; when the target database does not have a mirror image file with the file identifier which is the same as the target file identifier of the first mirror image file and the file size which is the same as the target file size of the first mirror image file, determining that the first mirror image file is a first target mirror image file;

acquiring a target file identifier of a second image file in the target database and the size of the target file of the second image file according to the traversal result; and when the mirror image file with the file identifier which is the same as the target file identifier of the second mirror image file does not exist in the second container mirror image warehouse and the file size which is the same as the target file size of the second mirror image file, determining that the second mirror image file is the second target mirror image file.

5. The method of claim 4, wherein synchronizing the image file in the second container image repository into the target database based on the target image file comprises:

when the target image file is the first target image file, acquiring service information of the service provided by the reasoning platform, a file identifier of the first target image file and image synchronization information of the first target image file;

generating target data based on the service information, the file identification of the first target image file and the image synchronization information;

and synchronizing the target data and the first target image file to the target database.

6. The method of claim 4, wherein synchronizing the image file in the second container image repository into the target database based on the target image file comprises:

and deleting the second target image file from the target database when the target image file is the second target image file.

7. The method of claim 1, further comprising:

when detecting that the image file in the second container image warehouse is updated, acquiring the updated image file;

and synchronizing the updated image file to the first container image warehouse.

8. The method of claim 1, wherein after synchronizing the image file in the second container image repository into the target database based on the target image file, the method further comprises:

and responding to a platform synchronization instruction, and synchronizing the mirror image file corresponding to the first reasoning platform to the container mirror image warehouse corresponding to the second reasoning platform, wherein the container mirror image warehouse of the first reasoning platform is the same as the container mirror image warehouse of the second reasoning platform.

9. An apparatus for synchronizing an image file, comprising:

the system comprises a first synchronization module, a second synchronization module and a third synchronization module, wherein the first synchronization module is used for responding to a synchronization instruction and synchronizing a mirror image file to be synchronized from a first container mirror image warehouse to a second container mirror image warehouse, the first container mirror image warehouse is used for storing a mirror image file generated by a training platform for a preset training scene, the second container mirror image warehouse is used for storing a mirror image file required by an inference platform for providing service for a target object, and the first container mirror image warehouse and the second container mirror image warehouse belong to the same type of mirror image warehouse;

the traversal module is used for traversing the image file in the second container image warehouse and traversing the image file in a target database to obtain a traversal result, wherein the target database is used for storing the image file which can be identified by the inference platform;

the difference determining module is used for determining the image file with the difference between the second container image warehouse and the target database according to the traversal result to obtain a target image file;

and the second synchronization module is used for synchronizing the mirror image file in the second container mirror image warehouse to the target database based on the target mirror image file so that the inference platform provides service for the target object by using the mirror image file in the target database.

10. A computer-readable storage medium, in which a computer program is stored, wherein the computer program, when executed by a processor, implements the method for synchronizing the image file according to any one of claims 1 to 8.

11. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method for synchronizing an image file according to any one of claims 1 to 8 when executing the computer program.

Images