WO2023103190A1

WO2023103190A1 - Multi-level linkage transparent sample model sharing apparatus for artificial intelligence platform

Info

Publication number: WO2023103190A1
Application number: PCT/CN2022/079255
Authority: WO
Inventors: 宋立华; 邱镇; 苏江文; 黄晓光; 吴佩颖
Original assignee: 福建亿榕信息技术有限公司; 国网信息通信产业集团有限公司; 国网四川省电力公司; 国家电网有限公司
Priority date: 2021-12-06
Filing date: 2022-03-04
Publication date: 2023-06-15
Also published as: CN114374701A; CN114374701B

Abstract

A multi-level linkage transparent sample model sharing apparatus for an artificial intelligence platform. The apparatus comprises: a global directory service subsystem, at least one transparent sample model sharing subsystem and at least one artificial intelligence platform, wherein the artificial intelligence platform and the transparent sample model sharing subsystem are deployed in a manner of pairing with each other on a one-to-one basis; each transparent sample model sharing subsystem is connected to the global directory service subsystem; all sample model directories are maintained by means of the global directory service subsystem, such that the consistency of the sample model directories is ensured; and a request from a local artificial intelligence platform is taken over by the transparent sample model sharing subsystem, data distribution over the whole network is queried in a manner of cooperating with the global directory service subsystem, and the storage and synchronous transmission of sample model data are then performed by means of the transparent sample model sharing subsystem, thereby satisfying related requirements of transparent sharing, secure sharing and efficient transmission of mass sample model data of a cross-region multi-level artificial intelligence platform.

Description

A multi-level linkage artificial intelligence platform sample model transparent sharing device

technical field

The invention relates to the technical field of artificial intelligence, in particular to a device for transparently sharing sample models of a multi-level linkage artificial intelligence platform.

Background technique

Artificial intelligence technology has gradually become a key element to promote the development of productivity, change the production operation mode, and improve production efficiency. In order to support the large-scale development and operation of artificial intelligence applications, large enterprises have also developed and launched their own "artificial intelligence platforms" to realize the aggregation and integration of artificial intelligence-related capabilities, including face authentication, process robots, knowledge Provide support for various enterprise artificial intelligence application scenarios such as retrieval and risk prevention and control.

The so-called artificial intelligence platform usually consists of "two libraries and one platform", as well as a sample library, a model library and an operating platform. Among them, the sample library is a component for storing and managing various professional and various sample resources, relying on functions such as sample storage, sample preprocessing, sample labeling, sample label management, and sample service catalog to provide sample resources for artificial intelligence model training; model As a component for storing and managing general-purpose and special-purpose models of various disciplines, the library provides various general-purpose and power-specific algorithm models, relying on functions such as model testing, image packaging, version management, model uploading, model downloading, and model service catalogs to provide AI applications Provides intelligent model resources; the operating platform provides functions such as model import, model verification, model deployment, service release, and cloud-edge collaboration, supporting model reasoning and application integration.

All kinds of data samples and models contained in the artificial intelligence platform require a lot of intellectual resources and human labor to produce. Whether it is outsourcing or self-developed, it is hoped that it can be used intensively throughout the enterprise to avoid repeated procurement or research and development. On the other hand, for large enterprises, including central enterprises, whose branches are distributed throughout the country and even the world, the network environment for applying artificial intelligence technology includes physically isolated intranets, extranets, and the Internet. On the job site, considering the performance of real-time access and the difficulty of application promotion, it is impossible to rely on a set of artificial intelligence platforms to provide services for all users, but it is necessary to deploy artificial intelligence platforms in different branch structures and networks, so that large enterprises have strong The willingness to open up the artificial intelligence platforms at each deployment point to realize the transparent sharing of data samples and model files between multi-level artificial intelligence platforms.

As shown in Figure 1, the requirements for transparent and secure sharing of sample models between multi-level artificial intelligence platforms of large enterprises; for the deployment of multi-level artificial intelligence platforms for large enterprises and the difficulties of transparent and secure sharing of sample models between multi-level platforms, the existing technologies Difficulties mainly include:

(1) It is difficult to share transparently: it is necessary to share all artificial intelligence models and samples between multi-level artificial intelligence platforms (including headquarters, regional centers, edge-side operation sites, etc.), and between different networks (intranet, extranet, Internet) Data, providing a unified directory and access means for all users. How to allow users in different regions to access the models and samples of the entire network in the case of a large amount of repeated data storage is a problem that needs to be considered;

(2) It is difficult to achieve unified compliance with security regulations: different levels of platforms and networks have different data security specifications and special devices (firewalls, information security isolation devices), which brings challenges to the realization of model sample sharing across network levels: different networks The "secrecy level" requirements are different, and the multi-level linkage artificial intelligence platform needs to meet the security level specifications of different network areas and provide consistent and complete support. The network of a large enterprise generally involves three types: a physically isolated proprietary information network (intranet), a logically isolated proprietary information network (external network), and the Internet. Among the three types of networks, the Internet area cannot store and use confidential data in any form, the external network can use and cache low-level files, and the internal network can use and store all confidential files for a long time;

(3) Transmission performance transmission and integrity verification of data transmission: The data that needs to be transmitted by the multi-level artificial intelligence platform is divided into two categories, one is a single GB-level model large file, and the other is a large number but a single file is small Data sample files (such as pictures, audio, etc.). Between different levels of platforms and different network environments, how to make full use of network bandwidth and information security isolation device bandwidth to realize efficient transmission and sharing of GB-level large models and KB-level small sample files is also a problem that needs to be considered as a whole.

Therefore, the technical problem that needs to be solved for the transparent and secure sharing of sample models between multi-level artificial intelligence platforms in large enterprises is mainly: how to realize the high-performance transmission of ultra-large-capacity artificial intelligence model data and sample data between multi-level organizations and multi-type networks , transparent sharing, and supports common security equipment, in line with enterprise security regulations.

At present, there is no public literature that provides an overall solution to the problem of transparent and secure sharing of sample models among multi-level artificial intelligence platforms in large enterprises. However, there are technical solutions for the technical problems involved, including high-performance transmission of large files and data transmission between networks. analyse as below:

The existing technical solution is mainly aimed at large files, and solves the problem of high-performance transmission of large files through file data fragmentation and multi-threaded parallel transmission. A typical comparative document is the name of the invention: a large file transmission method, device and system, and the application number is: 202011337777.9, which decomposes large file transmission into three links: file fragmentation, multi-thread transmission, and file identification-based merging. ; This solution better improves the performance of large file transfers and reduces the failure rate. However, it does not solve the time-consuming problem of file integrity guarantee or digital digest calculation involved in integrity guarantee.

To sum up, there is currently no public literature that provides an overall solution to the problem of transparent and secure sharing of sample models between multi-level artificial intelligence platforms in large enterprises; the high-performance transmission technology of massive files involved, the two-way data exchange technology between networks, etc. It cannot fully meet the current problems in efficient data transmission, security compliance, and transparent sharing pointed out in the background of the present invention, and is not fully applicable to the related issues of multi-level deployment of artificial intelligence platforms in large enterprises.

Contents of the invention

The technical problem to be solved by the present invention is to provide a multi-level linkage artificial intelligence platform sample model transparent sharing device to solve the related requirements of transparent sharing, safe sharing, and efficient transmission of massive sample model data on cross-regional multi-level artificial intelligence platforms.

The present invention provides a multi-level linkage artificial intelligence platform sample model transparent sharing device, comprising: a global directory service subsystem, at least one sample model transparent sharing subsystem and at least one artificial intelligence platform; the artificial intelligence platform is transparent to the sample model One-to-one pairing deployment of shared subsystems; each sample model transparent shared subsystem is connected to said global directory service subsystem;

Maintain all sample model directories through the global directory service subsystem to ensure consistency; take over requests from the local artificial intelligence platform through the sample model transparent sharing subsystem, and coordinate with the global directory service subsystem to query the data distribution of the entire network, and then pass The sample model transparent sharing subsystem stores and transmits the sample model data synchronously.

Further, the sample model transparent sharing subsystem includes local directory service, global synchronization service and data storage service; specifically includes sample model update and sample model cross-platform sharing;

The sample model update includes: the artificial intelligence platform calls the local directory service of the sample model transparent sharing system deployed in the same network area, and submits the file data; the local directory service calls the local data storage service to store the file data, and at the same time, the newly added The directory of the file data is submitted to the global directory service subsystem as a message text; the global directory service subsystem updates the directory;

The cross-platform sharing of the sample model includes: the local catalog service initiates a query to the global catalog service subsystem at intervals of a set time, and the global catalog service subsystem returns catalog data changes that occurred within the past set time to the global synchronization service ; After obtaining the changed global catalog data, the global synchronization service calls the local catalog service to merge and update the local catalog.

Further, the data storage service is provided with a network isolation device adaptation plug-in, which extracts the network isolation device adaptation function separately and designs a unified interface for adapting to different Firewall and information security isolation device in network environment.

Further, the data storage service is provided with a storage resource read-write module; the storage resource read-write module is a cloud storage protocol aimed at the mainstream in the Java language, which unifies the block data read-write interface and supports modification of all data through configuration files. The specific implementation adopted to realize plug-in management.

Further, the storage resource read-write module will need temporarily cached files according to the confidentiality level of the file and the enterprise's configuration requirement information on whether data with different confidentiality levels can be stored for a long time in different network areas, whether it can be temporarily cached, and how long the temporary cache time is. Write into the distributed cache, and set the expiration time at the same time; the distributed cache is IT middleware, which supports automatic deletion of configuration expiration; and the artificial intelligence platform accesses the sample model file according to the returned file path; for confidential data, the artificial intelligence The platform does not provide file secondary distribution function.

Further, the synchronous transmission is further specifically:

Before transmission, split the file into blocks with a set threshold MB until all blocks are less than or equal to the set threshold MB. If the file is smaller than the set threshold MB, do not split, calculate the digital summary of all blocks, and merge into a digital summary, and then block multi-threaded parallel transmission;

During the transmission process, the file receiver receives the file, calculates the digital summary of the fixed block in parallel, and saves them one by one;

After the transmission is completed, all blocks are merged into the original large file in order, and the digital digests of all blocks are also merged into one digital digest to obtain the sample model file and its corresponding digital digest obtained by the final synchronous transmission, and the obtained digital digest Compare with the merged digital digest before transmission, if the same, the file transfer is complete; if not, roll back and retransmit.

One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

The embodiment of the present application provides a multi-level linkage artificial intelligence platform sample model transparent sharing device, which consists of a "global directory service subsystem" and a "sample model transparent sharing subsystem" to form a basic service facility that supports the transparent sharing of artificial intelligence platform model samples The architecture system, and through the "transparent sharing mechanism of model files based on hierarchical directory", "high-performance sample model data synchronization and heterogeneous storage integration based on segmented transmission verification in cross-network environment", "cross- Network regional data security compliance utilization” and other solutions to solve the related needs of transparent sharing, safe sharing, and efficient transmission of massive sample model data on cross-regional multi-level artificial intelligence platforms.

The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable , the specific embodiments of the present invention are enumerated below.

Description of drawings

The present invention will be further described below in conjunction with the embodiments with reference to the accompanying drawings.

Figure 1 is a schematic diagram of the structure of the transparent and secure sharing of sample models between multi-level artificial intelligence platforms of large-scale enterprises in the prior art;

Fig. 2 is the overall architecture diagram of the device of the present invention;

Fig. 3 is a schematic diagram of the transparent sharing mechanism of model files based on hierarchical directories in the present invention;

Fig. 4 is a schematic diagram of the high-performance sample model data synchronization scheme based on segmented transmission and verification in the present invention;

FIG. 5 is a sequence diagram of the security compliance utilization of data across network regions based on the unified caching scheme in the present invention.

Detailed ways

The general idea of the technical solution in the embodiment of the application is as follows:

Aiming at the high-speed transmission of sample model data of different sizes on the multi-level linkage artificial intelligence platform, the transparent sharing and acquisition of sample model data distributed in different regional network levels, and the safe and compliant utilization of sample model data in different network areas, it provides a systematic, The overall approach provides a technical basis for the multi-level and cross-network deployment of large-scale enterprise artificial intelligence platforms. The content of the invention mainly includes the following parts:

(1) A system architecture that supports transparent sharing of multi-level linkage artificial intelligence platforms. It is proposed that the "global directory service subsystem" and the "sample model transparent sharing subsystem" constitute the architecture system of the basic service facilities supporting the transparent sharing of artificial intelligence platform model samples.

(2) Design of a transparent sharing mechanism for model files based on hierarchical directories. A transparent sharing mechanism based on a hierarchical directory is proposed, that is, the unified sample model directory of the entire network is maintained through the "global directory" to ensure global information consistency, and all requests from the local artificial intelligence platform are taken over through the "local directory service" and communicated with it. The collaboration of the global directory service can quickly query the data distribution of the entire network without the local artificial intelligence platform being aware of the global directory.

(3) A high-performance sample model data synchronization and heterogeneous storage integration design based on segmentation transmission verification in a cross-network environment. A high-performance sample model data synchronization design based on segmented transmission and verification is proposed. In view of the characteristics of multiple partial changes in the entire life cycle of the sample model data of the artificial intelligence platform, the design splits the file into segments, transfers and verifies by segment. The method can significantly improve the synchronization performance of sample model data; it is proposed to extract the network isolation device adaptation function separately and design it into a plug-in form with a unified interface, so as to realize integration with different devices and improve the adaptability of the system to different network environments ;Proposed to set up a separate "storage resource read-write module" module, and adapt to different storage resources through plug-in, and support the technical solution of the evolution of the storage technology route.

(4) A design for cross-network regional data security and compliance utilization based on a unified caching scheme. A data security and compliance utilization scheme based on unified cache is proposed, which converts the "cross-network security utilization problem" of confidential files into the cache time problem of files with different confidentiality levels in different network areas, completely avoiding additional encryption overhead, and satisfying enterprises The data security specification, to a certain extent, solves the problem of cross-network regional data security compliance and utilization at low cost.

Through the design of the overall system architecture, the transparent sharing mechanism design of model files based on hierarchical directories, the high-performance sample model data synchronization and heterogeneous storage integration design based on segmented transmission verification in a cross-network environment, and the cross-network regional data based on a unified caching scheme The specific implementation of the present invention is described in four aspects of security and compliance utilization design.

(1) Overall architecture design

As shown in Figure 2, the overall architecture is composed of "global directory service subsystem" and "sample model transparent sharing subsystem". Among them, the "global directory service subsystem" only needs to deploy one service example in the whole network; the "sample model transparent sharing subsystem" is deployed in a one-to-one pairing with the artificial intelligence platform, which can be used as part of the artificial intelligence platform service group or as a separate Service, providing complete sample model data storage and synchronous transmission services for artificial intelligence.

The main module composition and operating mechanism of the "global directory service subsystem" and "sample model transparent sharing subsystem" will be described in the following specific schemes.

(2) Design of transparent sharing mechanism for model files based on hierarchical directory

The hierarchically deployed artificial intelligence platform can upload samples and model data through any deployment point. In order to make these sample model data transparently shared by other deployment points, the present invention proposes a transparent sharing mechanism based on hierarchical directories, that is, maintains a unified sample model directory for the entire network through a "global directory" to ensure global information consistency; through " "Local directory service" takes over all requests from the local artificial intelligence platform, and through its collaboration with the global directory service, it can quickly query the data distribution of the entire network without the local artificial intelligence platform being aware of the global directory. The global directory and the local directory together constitute the AI sample model directory service that supports AI platforms at all levels.

It should be pointed out that the global synchronization is limited to directory data, and the files of the sample models are still maintained locally, and they are only transferred on demand when they need to be called from different places later. The directory data is much smaller than the sample model file itself, thus effectively avoiding repeated storage and transmission of a large amount of data while supporting network-wide sharing.

As shown in Figure 3, the global transparent sharing mechanism scheme includes a two-stage process of "sample model update" and "sample model cross-platform sharing":

1) Sample model update stage

Step 1: Upload data locally. In the artificial intelligence platform, after the user uploads the interface, modifies the sample with the labeling tool, or trains and generates a new model, the artificial intelligence platform calls the local "local directory service" of the "sample model transparent sharing system" deployed in the same network area, Submit file data.

Step 2: Submit to the global directory. The "local directory service" calls the local "data storage service" to store file data, and at the same time submits the new data directory (including name, metadata, etc.) as message text to the distributed message middleware of the "global directory".

Step 3: Update to the global catalog. "Global directory service" monitors the messages of the local distributed message middleware, and updates the content of the messages to the global directory. Relying on the high-availability and high-consistency features of the distributed message middleware, it can ensure that the content in the global directory is complete and non-repetitive.

2) Sample model cross-platform sharing

In order to ensure that the local artificial intelligence platform can query and browse the sample model catalog data of the entire network, it is necessary to periodically synchronize the catalog data from the "global catalog service". The specific steps are:

Step 1: Scheduled synchronization requests for the global directory. The local "global synchronization service" initiates a query to the global directory service periodically (eg, every hour), and the "global directory service" returns the directory data changes that occurred in the past hour to the "global synchronization service".

Step 2: Local directory merge updates. After obtaining the changed global catalog data, the "global synchronization service" calls the update interface of the local catalog service, and submits the changed catalog data to the local catalog for merge update.

(3) High-performance sample model data synchronization and heterogeneous storage integration design based on segmentation transmission verification in cross-network environment

The models and sample data in the artificial intelligence platform will be modified many times throughout the life cycle, such as adding sample annotations, or model superposition and fusion, etc. In the above-mentioned various data modifications, the file itself may only undergo partial changes. If only the changed content is transmitted as much as possible, the transmission efficiency of the sample model data between multiple platforms can be greatly improved.

On the other hand, after data transmission, it is necessary to ensure the integrity of the data before and after synchronization. The mainstream solution usually adopts digital digest technology (such as MD5), and calculates the digital digest of a single file before and after file synchronous transmission. If the two are completely equal, it can prove that the synchronized data is complete. However, since the execution of the data summary algorithm is usually very time-consuming and proportional to the file size, the method of calculating the digital summary for a single file takes a lot of time. If the running time of the digital summary algorithm can be reduced, it will help to improve the model data. synchronization transmission efficiency.

The present invention proposes a synchronous design of high-performance sample model data based on segmented transmission and verification. Aiming at the characteristics of multiple partial changes in the entire life cycle of the sample model data of the artificial intelligence platform, the design splits the file into segments, transmits them by segment, and The verification method can significantly improve the synchronization performance of sample model data. The overall scheme is shown in the figure.

As shown in Figure 4, the specific mechanism design is introduced through the following three aspects:

1) Segmented transmission and verification. It is executed by the "segmented transfer verification module" during the file transfer process. The specific process is:

Before transmission, split the large file of the artificial intelligence sample model into 1MB blocks (if some sample files are smaller than 1MB, they will not be split), calculate the digital summaries of all blocks, and merge them into one digital digest. Then block multi-threaded parallel transmission;

After the transmission is completed, all the blocks are merged into the original large file in order, and the digital digests of all the blocks are also merged into one digital digest to obtain the sample model file and its corresponding digital digest obtained by the final synchronous transmission. Compare the obtained digital digest with the merged digital digest before transmission, if they are the same, the file transfer is complete. If different, rollback the retransmission.

Aspect the data flow transmission, what the present invention specifically adopts is the streaming file transmission technology based on the Java Mina framework. This technology is a mature technology commonly used in the industry and will not be introduced here.

The segmented transmission and verification design provided by the present invention can effectively utilize the idle computing resources of the current multi-core computer system to carry out file transmission and digital summary calculation in parallel, thereby improving the performance of file transmission and integrity verification.

2) Network isolation device adapter plug-in. In an enterprise area network interconnection environment, different network partitions may be connected through "firewalls" or "information security isolation devices". These devices, especially "information security isolation devices", usually do not support transparent transmission, but provide unique interfaces Called by the data transfer process.

The present invention extracts the adaptation function of the network isolation device separately, and designs it into a plug-in form of a unified interface, so as to realize integration with different devices and improve the adaptability of the system to different network environments.

3) Read and write of heterogeneous storage resources. A single file of an artificial intelligence sample model may reach the size of GB. In a mature artificial intelligence platform, it may consume hundreds of terabytes or even petabytes of storage resources to store each sample model file, which puts forward high requirements on storage resources. Since the informatization infrastructure structures in different regions are inconsistent, and there may be many different storage resources at the same time (such as enterprise private cloud storage, distributed storage, centralized storage array equipment, etc.), the present invention sets up a separate "storage resource read "Write Module" module, similar to "Network Isolation Device Adaptation", adapts to different storage resources through plug-in, and supports the evolution of storage technology routes.

Specifically, in the "storage resource read-write module" module of the present invention, based on the Java language, the mainstream cloud storage protocol (such as the S3 protocol) and the block data read-write interface have been uniformly implemented, and it supports modification of all resources through configuration files. The specific implementation adopted to realize plug-in management. Among them, reading and writing for different storage resources is a common technology.

At the same time, the "storage resource read-write module" is also the main carrier to realize the "safe and compliant utilization of data across network regions based on the unified cache solution" solution, please refer to the next step for related introduction.

(4) Design of cross-network regional data security and compliance utilization based on unified caching scheme

In view of the problem that different network areas have different file security levels (secret levels), the mainstream method for data security across network areas is to encrypt files. However, due to the large number of artificial intelligence sample model files and single files up to GB level, encrypting and decrypting files requires a lot of computing resource overhead and time overhead, which is almost unacceptable in the actual application process.

The present invention proposes a data security and compliance utilization scheme based on unified caching, which converts the "cross-network security utilization problem" of confidential files into the caching time problem of files with different confidentiality levels in different network areas, completely avoids additional encryption overhead, and satisfies The enterprise data security specification that "the Internet area cannot store and use confidential data in any form, the external network can use and cache low-level files, and the internal network can use and store all confidential files for a long time" solves the problem at a low cost to a certain extent. Cross-network area data security and compliance use issues.

The specific scheme is shown in Figure 5. The "storage resource read-write module" is based on the confidentiality level of the file and the enterprise's configuration requirements for whether data with different confidentiality levels can be stored for a long time in different network areas, whether it can be temporarily cached, and how long the temporary cache time is. Write the files that need to be cached temporarily (such as when the files of ordinary secret level are used in the Internet area) into the "distributed cache", and set the expiration time at the same time. "Distributed cache" is the current mainstream IT middleware, which supports automatic deletion of configuration expiration, which can meet the requirements of this solution; the artificial intelligence platform system accesses the sample model files according to the returned file path. For confidential data, the platform does not provide secondary distribution functions such as file downloads on the interface, so as to achieve compliance with enterprise data security regulations.

Embodiment one

This embodiment provides a multi-level linkage artificial intelligence platform sample model transparent sharing device, including: a global directory service subsystem, at least one sample model transparent sharing subsystem and at least one artificial intelligence platform; the artificial intelligence platform is transparent to the sample model One-to-one pairing deployment of shared subsystems; each sample model transparent shared subsystem is connected to said global directory service subsystem;

Maintain all sample model directories through the global directory service subsystem to ensure consistency; take over requests from the local artificial intelligence platform through the sample model transparent sharing subsystem, and coordinate with the global directory service subsystem to query the data distribution of the entire network, and then pass The sample model transparent sharing subsystem stores and synchronously transmits sample model data;

The synchronous transmission is further specifically: before the transmission, the file is split into blocks of the set threshold MB until all blocks are less than or equal to the set threshold MB, if the file is smaller than the set threshold MB, then no splitting is performed, Calculate the digital summaries of all blocks, merge them into one digital summaries, and then transmit them in parallel in multi-threaded blocks;

The sample model transparent sharing subsystem includes local directory service, global synchronization service and data storage service; specifically includes sample model update and sample model cross-platform sharing;

The data storage service is provided with a network isolation device adapter plug-in, which extracts the network isolation device adaptation function separately and designs a unified interface for adapting to different network environments Firewall and information security isolation device.

The data storage service is provided with a storage resource read-write module; the storage resource read-write module is Java language for the mainstream cloud storage protocol, which unifies the block data read-write interface and supports modification of the specific data used by configuration files. Realize and implement plug-in management.

The storage resource read-write module writes the files that need to be temporarily cached into distribution according to the confidentiality level of the file and the enterprise's configuration requirement information on whether data with different confidentiality levels can be stored for a long time in different network areas, whether it can be temporarily cached, and how long the temporary cache time is. cache, and set the expiration time at the same time; the distributed cache is an IT middleware, which supports automatic deletion after configuration expiration; and the artificial intelligence platform accesses the sample model file according to the returned file path; for confidential data, the artificial intelligence platform does not provide File secondary distribution function.

Although the specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments we have described are only illustrative, rather than used to limit the scope of the present invention. Equivalent modifications and changes made by skilled personnel in accordance with the spirit of the present invention shall fall within the protection scope of the claims of the present invention.

Claims

A multi-level linkage artificial intelligence platform sample model transparent sharing device is characterized in that it includes: a global directory service subsystem, at least one sample model transparent sharing subsystem and at least one artificial intelligence platform; the artificial intelligence platform is transparent to the sample model One-to-one pairing deployment of shared subsystems; each sample model transparent shared subsystem is connected to said global directory service subsystem;

Maintain all sample model directories through the global directory service subsystem to ensure consistency; take over requests from the local artificial intelligence platform through the sample model transparent sharing subsystem, and coordinate with the global directory service subsystem to query the data distribution of the entire network, and then pass The sample model transparent sharing subsystem stores and transmits the sample model data synchronously.
A multi-level linkage artificial intelligence platform sample model transparent sharing device according to claim 1, characterized in that:

The sample model transparent sharing subsystem includes local directory service, global synchronization service and data storage service; specifically includes sample model update and sample model cross-platform sharing;

The sample model update includes: the artificial intelligence platform calls the local directory service of the sample model transparent sharing system deployed in the same network area, and submits the file data; the local directory service calls the local data storage service to store the file data, and at the same time, the newly added The directory of the file data is submitted to the global directory service subsystem as a message text; the global directory service subsystem updates the directory;

The cross-platform sharing of the sample model includes: the local catalog service initiates a query to the global catalog service subsystem at intervals of a set time, and the global catalog service subsystem returns catalog data changes that occurred within the past set time to the global synchronization service ; After obtaining the changed global catalog data, the global synchronization service calls the local catalog service to merge and update the local catalog.
A multi-level linkage artificial intelligence platform sample model transparent sharing device according to claim 2, characterized in that: the data storage service is provided with a network isolation device adapter plug-in, and the network isolation device adapter plug-in is The network isolation device adaptation function is extracted separately, and the design forms a unified interface form, which is used to adapt to firewalls and information security isolation devices in different network environments.
A kind of multi-level linkage artificial intelligence platform sample model transparent sharing device according to claim 2, it is characterized in that: described data storage service is provided with a storage resource read-write module; Described storage resource read-write module is Java language for The mainstream cloud storage protocol unifies the block data read and write interfaces, and supports the specific implementation adopted by modifying the configuration file to realize plug-in management.
A multi-level linkage artificial intelligence platform sample model transparent sharing device according to claim 4, characterized in that: the storage resource read-write module is based on the confidentiality level of the file and whether the enterprise can store data with different confidentiality levels for a long time in different network areas , whether it can be temporarily cached, and how long the temporary cache time is required to configure the information, write the files that need to be temporarily cached into the distributed cache, and set the expiration time at the same time; the distributed cache is an IT middleware that supports automatic deletion of configuration expiration; And the artificial intelligence platform accesses the sample model files according to the returned file path; for confidential data, the artificial intelligence platform does not provide the file secondary distribution function.
A multi-level linkage artificial intelligence platform sample model transparent sharing device according to claim 1, characterized in that: the synchronous transmission is further specifically:

Before transmission, split the file into blocks with a set threshold MB until all blocks are less than or equal to the set threshold MB. If the file is smaller than the set threshold MB, do not split, calculate the digital summary of all blocks, and merge into a digital summary, and then block multi-threaded parallel transmission;

During the transmission process, the file receiver receives the file, calculates the digital summary of the fixed block in parallel, and saves them one by one;

After the transmission is completed, all blocks are merged into the original large file in order, and the digital digests of all blocks are also merged into one digital digest to obtain the sample model file and its corresponding digital digest obtained by the final synchronous transmission, and the obtained digital digest Compare with the merged digital digest before transmission, if the same, the file transfer is complete; if not, roll back and retransmit.