WO2024000387A1

WO2024000387A1 - Ai model establishment evaluation system, video stream simulation module and method, and controller

Info

Publication number: WO2024000387A1
Application number: PCT/CN2022/102783
Authority: WO
Inventors: 樊林
Original assignee: 京东方科技集团股份有限公司
Priority date: 2022-06-30
Filing date: 2022-06-30
Publication date: 2024-01-04
Also published as: CN117642728A

Abstract

An AI model establishment evaluation system, comprising: a controller (11) and a simulation evaluation module (12). The controller (11) is configured to determine grouping information of an AI model; executing the following operations on an AI model of each group according to the grouping sequence: establishing an AI model of a current group to a testing environment, and sending a first notification to the simulation evaluation module (12). The simulation evaluation module (12) is configured to receive the first notification from the controller (11) and provide a video stream service to the AI model, the content of the video stream service being video stream content of a video file required for evaluating the AI model of the current group; performing AI capability evaluation on the AI model of the current group; and releasing a video stream once capability evaluation is completed.

Description

AI model construction and evaluation system, video stream simulation module and method, controller

Technical field

Embodiments of the present disclosure relate to, but are not limited to, the technical field of intelligent systems, and in particular, to an AI model construction and evaluation system, a video stream simulation module and method, and a controller.

Background technique

Computer vision artificial intelligence (Artificial Intelligence, AI) technology based on video processing requires a large amount of graphics processor (Graphics Processing Unit, GPU) resources for processing. Based on the dynamic model calling framework of Triton (Triton is an open source software released by NVIDIA), the AI model is dynamically arranged and processed as needed to achieve the purpose of saving physical resources. However, during the development process, each model update may involve changes in the accuracy of multiple AI functions. In order to resolve the dependencies between such AI models more quickly and at a lower cost, it is necessary to introduce DevOps (Development & Operations, Integration of development and operation) The process of packaging, deploying, testing and launching a full set of models, and conducting a comprehensive and automated assessment of the changed models to determine whether the effect of each model update is positive or negative.

Contents of the invention

The following is an overview of the topics described in detail in this article. This summary is not intended to limit the scope of the claims.

The embodiment of the present disclosure provides an AI model construction and evaluation system, including: a controller and a simulation evaluation module, wherein:

The controller is configured to determine the grouping information of the AI model; according to the grouping order, perform the following operations for the AI model of each group: construct the AI model of the current group to the test environment, and send the first notification to the simulation evaluation module;

The simulation evaluation module is configured to receive a first notification from the controller and provide a video streaming service to the AI model. The content of the video streaming service is a video stream of video files required to evaluate the AI model of the current group. Content; perform AI capability assessment on the currently grouped AI model; release the video stream after the capability assessment is completed.

The embodiment of the present disclosure also provides an AI model construction and evaluation method, including:

The controller determines model grouping information;

According to the grouping order, perform the following operations on the AI model of each group:

The controller constructs the AI model of the current group to the test environment, and sends the first notification to the simulation evaluation module;

The simulation evaluation module receives the first notification from the controller and provides a video streaming service to the AI model. The content of the video streaming service is the video streaming content of the video file required to evaluate the AI model of the current group; Perform an AI capability assessment on the currently grouped AI model; after the capability assessment is completed, release the video stream.

The controller determines model grouping information;

The controller performs the following operations on the AI model of each group in accordance with the grouping order: builds the AI model of the current group to the test environment, and notifies the video stream simulation module to push the video stream content of the video file required to evaluate the AI model of the current group. To the streaming media platform, notify the AI capability evaluation module to use the video stream of the streaming media platform to evaluate the AI capabilities of the current grouped AI model. After the capability assessment is completed, notify the video stream simulation module and the streaming media platform to release the video stream. .

An embodiment of the present disclosure also provides a controller, including a memory; and a processor coupled to the memory, the processor being configured to execute any implementation of the present disclosure based on instructions stored in the memory. The steps of the AI model construction and evaluation method described in the example.

Embodiments of the present disclosure also provide a storage medium on which a computer program is stored. When the program is executed by a processor, the AI model construction and evaluation method described in any embodiment of the present disclosure is implemented.

Embodiments of the present disclosure also provide a video stream simulation method, including:

The video stream simulation module pulls the corresponding video file from the video file address list, pushes the video stream content of the video file to the streaming media platform, and notifies the controller that the streaming media environment is completed;

The video stream simulation module receives the release instruction of the controller and stops pushing the video stream content;

The video stream simulation module sends a video stream release instruction to the streaming media platform.

Embodiments of the present disclosure also provide a video stream simulation module, including a memory; and a processor coupled to the memory, the processor being configured to execute any of the instructions of the present disclosure based on instructions stored in the memory. The steps of the video stream simulation method according to an embodiment.

An embodiment of the present disclosure also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the video stream simulation method as described in any embodiment of the present disclosure is implemented.

After reading and understanding the drawings and detailed description, other aspects can be understood.

Description of drawings

The drawings are used to provide a further understanding of the technical solution of the present disclosure, and constitute a part of the specification. They are used to explain the technical solution of the present disclosure together with the embodiments of the present disclosure, and do not constitute a limitation of the technical solution of the present disclosure. The shapes and sizes of components in the drawings do not reflect true proportions and are intended only to illustrate the present disclosure.

Figures 1A, 1B and 1C are schematic structural diagrams of three AI model construction and evaluation systems according to exemplary embodiments of the present disclosure;

Figure 2A is a schematic structural diagram of another AI model construction and evaluation system according to an exemplary embodiment of the present disclosure;

Figure 2B is a schematic diagram of a test case and required video stream of an AI model according to an exemplary embodiment of the present disclosure;

Figures 2C and 2D are schematic diagrams of two exemplary methods for group testing the test cases shown in Figure 2B;

Figure 2E is a schematic diagram of a test case and required video stream of another AI model according to an exemplary embodiment of the present disclosure;

Figure 2F is a schematic diagram of an exemplary method for group testing the test cases shown in Figure 2E;

Figure 3 is a schematic diagram of a method for grouping video streams required for simulation according to an exemplary embodiment of the present disclosure;

Figure 4 is a schematic diagram of a DevOps task initiation process according to an exemplary embodiment of the present disclosure;

Figure 5 is a schematic diagram of a video environment construction process according to an exemplary embodiment of the present disclosure;

Figure 6 is a schematic flowchart of an AI model construction and evaluation method according to an exemplary embodiment of the present disclosure;

Figure 7 is a schematic flowchart of another AI model construction and evaluation method according to an exemplary embodiment of the present disclosure;

Figure 8 is a schematic flowchart of a video stream simulation method according to an exemplary embodiment of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be noted that, as long as there is no conflict, the embodiments and features in the embodiments can be arbitrarily combined with each other.

Unless otherwise defined, the technical terms or scientific terms used in the disclosure of the embodiments of the present disclosure shall have the usual meanings understood by those with ordinary skill in the art to which the disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "include" or "include" mean that the elements or things preceding the word include the elements or things listed after the word and their equivalents, without excluding other elements or things.

If the cycle frequency of code changes, model reconstruction, model evaluation, and model launch can be accelerated, it will be equivalent to accelerating the maturity process of the entire AI model and strategy and improving research and development efficiency. When a large number of models and AI strategies are maintained at the same time, the number of video streams that the entire set of AI models and strategies rely on reaches a scale of several hundred. If an environment with so many video streams is always maintained, it will consume a lot of money. The central processing unit (Central Processing Unit, CPU) resources also bring difficulties to the evaluation of AI capabilities based on video services.

As shown in Figure 1A, the embodiment of the present disclosure provides an AI model construction and evaluation system, including: a controller 11 and a simulation evaluation module 12, wherein:

The controller 11 is configured to determine the grouping information of the AI model; according to the grouping order, perform the following operations for the AI model of each group: construct the AI model of the current group to the test environment, and send the first notification to the simulation evaluation module 12 ( The function of the first notification is to inform the simulation evaluation module 12 that the AI model of the current group has been built into the test environment);

The simulation evaluation module 12 is configured to receive the first notification from the controller 11 and provide a video streaming service to the AI model. The content of the video streaming service is a video stream of video files required to evaluate the AI model of the current group. Content; perform AI capability assessment on the currently grouped AI model; release the video stream after the capability assessment is completed.

The AI model construction and evaluation system of the embodiment of the present disclosure groups the AI models, sequentially builds the AI models of each group to the test environment, and provides the streaming media services required for evaluation on demand for the currently grouped AI models. After the evaluation is completed Finally, the video stream is released, and there is no need to maintain the video stream of all AI models at all times. This not only solves the problem of wasting a lot of resources in the video streaming environment, but also greatly reduces the computing resources required in the DevOps environment of the AI model, and also makes the model evaluation process more feasible. Control, improve the efficiency of AI technology research and development, and reduce costs.

The AI model described in the embodiments of the present disclosure refers to a program that can perceive, reason, act and adapt, while the machine learning model generally refers to a program that can continuously improve performance as the amount of data increases. The deep learning model is a machine A subset of learning models, deep learning models are programs that use multi-layer neural networks to learn from large amounts of data. Therefore, in terms of coverage, AI models are larger than machine learning models and are larger than deep learning models.

In this disclosed embodiment, when evaluating the AI capabilities of an AI model, one test case or multiple test cases can be designed for one AI model. Generally speaking, a test case is to design a situation in which the AI model must be able to run normally and achieve the expected execution results of the model design. For example, assuming that for an AI model used for VIP user detection, the test cases designed for it can include the following three:

Test case 1) Notify the AI model to pull video stream 1 (video stream 1 does not contain VIP users), and detect the AI model output results;

Test case 2) Notify the AI model to pull video stream 2 (video stream 2 contains 1 VIP user), and detect the AI model output result;

Test case 3) Notifies the AI model to pull video stream 3 (video stream 3 contains 2 VIP users), and detects the AI model output result.

When testing each test case, when the output result of the AI model matches the actual pulled video stream content, the test case passes; when the AI model output result does not match the actual pulled video stream content, the test Case failed.

When building a test environment, the controller 11 pulls the model code of the AI model from the code warehouse, compiles the model code and deploys it to the test environment; when testing, the AI model pulls the streaming media platform or streaming media server The video stream is output accordingly according to the pulled video stream. The simulation evaluation module detects the output result of the AI model, and determines whether the test case passes based on the output result of the AI model.

In the embodiment of the present disclosure, the simulation evaluation module 12 provides video streaming services to the AI model, which means that the simulation evaluation module provides one or more video stream control interfaces to the AI model (exemplarily, the video stream control interface can be a video stream address ), the AI model can pull a certain video stream by controlling different video stream control interfaces. After the capability evaluation is completed, the simulation evaluation module 12 releases the release stream through the release video stream control interface.

In some exemplary embodiments, the test environment may include one or more physical machines, which may receive the AI model deployed by the controller 11 and run the AI model. When building a test environment, the controller 11 constructs the AI model of the current group to the test environment; when testing, the simulation evaluation module 12 provides video streaming services to the AI model, and evaluates the AI capabilities of the AI model of the current group; After the capability assessment is completed, release the video stream. The simulation evaluation module 12 may be partially or entirely installed on the physical machine where the test environment is located, or may be installed on a physical machine outside the test environment. This is not limited in the embodiment of the present disclosure.

In some exemplary embodiments, as shown in Figure 1B, the simulation evaluation module 12 includes a video stream simulation server 120 and an AI capability evaluation module 121, where:

The video stream simulation server 120 is configured to receive the first notification from the controller 11 and provide a video stream service to the AI model. The content of the video stream service is the video of the video file required to evaluate the AI model of the current group. Stream content, send a second notification to the controller 11, so that the controller 11 sends a third notification to the AI capability evaluation module 121; receive the fifth notification from the controller 11, release the video stream;

The AI capability evaluation module 121 is configured to receive the third notification from the controller 11 and perform an AI capability evaluation on the AI model of the current group; after the capability evaluation is completed, send the fourth notification to the controller 11 so that the controller 11 Send the fifth notification to the video stream simulation server 120;

The controller 11 is further configured to receive a second notification from the video stream simulation server 120 and send a third notification to the AI capability evaluation module 121; upon receiving a fourth notification from the AI capability evaluation module 121, send a fifth notification. Notification to video stream simulation server 120.

In the embodiment of the present disclosure, when the video stream simulation server 120 is ready for the video stream service, it notifies the controller 11, and the controller 11 notifies the AI capability evaluation module 121 to start AI capability evaluation of the AI model of the current group. In an exemplary embodiment, the video stream simulation server 120 may also directly notify the AI capability evaluation module 121 to start performing AI capability evaluation on the AI model of the current group, and the embodiment of the present disclosure does not limit this. After the AI capability assessment module 121 completes the AI capability assessment, it notifies the controller 11, and the controller 11 notifies the video stream simulation server 120 to release the video stream. In other exemplary embodiments, the AI capability assessment module can also directly 121 notifies the video stream simulation server 120 to release the video stream, and the embodiment of the present disclosure does not limit this. The controller 11 notifies the AI capability evaluation module 121 to start AI capability evaluation of the current grouped AI model and the controller 11 notifies the video stream simulation server 120 to release the video stream, which enables the controller 11 to accurately control each The process of AI model evaluation makes the model evaluation process more controllable, improves the efficiency of AI technology research and development, and reduces costs.

In some exemplary implementations, as shown in Figure 1C, the video stream simulation server 120 includes a video stream simulation module 1201 and a streaming media platform 1202, wherein:

The video stream simulation module 1201 is configured to receive the first notification from the controller 11, push the video stream content of the video file required to evaluate the AI model of the current group to the streaming media platform 1202, and send the second notification to the controller 11 ; Receive the fifth notification from the controller 11, stop pushing the video stream content, and send the sixth notification to the streaming media platform 1202;

The streaming media platform 1202 is configured to receive the video stream content pushed by the video stream simulation module 1201 and provide video streaming services to the AI model; upon receiving the sixth notification from the video stream simulation module 1201, release the video stream.

In this embodiment, when building the test environment, the controller 11 builds the AI model of the current group into the test environment; when performing the test, the video stream simulation module 1201 adds the video of the video file required to evaluate the AI model of the current group. The streaming content is pushed to the streaming media platform 1202, and the streaming media platform 1202 provides video streaming services to the AI model in the test environment. The AI capability evaluation module 121 notifies the AI model to pull the video stream of the streaming media platform 1202, and evaluates the AI model's The output results are tested, and whether the test case passes is judged based on the output results of the AI model.

In some exemplary implementations, as shown in Figure 1C, embodiments of the present disclosure provide an AI model construction evaluation system, including: a controller 11, a video stream simulation module 1201, a streaming media platform 1202, and an AI capability evaluation module 121 ,in:

The controller 11 is configured to determine the model grouping information; according to the grouping order, perform the following operations for the AI model of each group: build the AI model of the current group to the test environment, send the first notification to the video stream simulation module 1201; receive The second notification is sent to the video stream simulation module 1201, and the third notification is sent to the AI capability evaluation module 121; the fourth notification is received from the AI capability evaluation module 121, and the fifth notification is sent to the video stream simulation module 1201;

The AI capability assessment module 121 is configured to receive the third notification from the controller 11, use the video stream to perform an AI capability assessment on the AI model of the current group, and after the capability assessment is completed, send the fourth notification to the controller 11;

Usually, the DevOps pipeline of AI models includes the following process: First, R&D personnel develop the code of the AI model → compile + deploy the AI model code to the test environment (this step is collectively referred to as the build process and can be realized through automation) → test + result analysis (This step is collectively called the evaluation process and can be realized through automation) → After repeated debugging and defect modification, after the test is OK, it will be launched on the production platform. Due to the debugging and defect modification of the AI model, it is necessary to cycle through the construction, evaluation and defect modification processes multiple times. Only after all defects have been cleared can the launch process be entered. The construction script of each AI model is usually completed by R&D personnel, the evaluation process is mainly led by testers, and the online process is mainly led by operation and maintenance personnel.

In the embodiment of the present disclosure, the actual physical locations of the video stream simulation module 1201 and the AI capability evaluation module 121 can be located at any location, as long as they are connected to the controller 11 and the streaming media platform 1202 through the network. For example, the video stream simulation module 1201 can be located on the physical machine where the controller 11 is located, or on the physical machine where the streaming media platform 1202 is located, or it can be located on a physical machine other than the controller 11 or the streaming media platform 1202 superior. For example, the AI capability assessment module 121 can be located on the physical machine where the test environment is located. When building the test environment, the code corresponding to the AI capability assessment module 121 and the model code corresponding to the AI model can be pulled from the code warehouse, and then compiled. Deployed together to the test environment, the code corresponding to the AI capability assessment module 121 may include script code required to run automated tests, and run preset test cases for different AI models to conduct specific capability assessments. For example, each test case can specify the pulled video stream address. When running each test case, the AI capability evaluation module 121 notifies the AI model to pull the video stream of the specified video stream address, and the AI model will pull the video stream of the specified video stream address. As input and corresponding output, the AI capability evaluation module determines whether the output results of the AI model match the actual pulled video stream content, and determines whether each test case passes the test. When both the AI capability evaluation module 121 and the AI model are deployed on the physical machine where the test environment is located, the physical resources required to build the entire AI model construction and evaluation system can be reduced, system resource utilization can be improved, system complexity can be reduced, and AI capabilities can be reduced Evaluate network communication data between module 121 and other modules. In other examples, the AI capability evaluation module 121 may also be located on a physical machine outside the test environment.

In this disclosed embodiment, the streaming media platform 1202 can be a streaming media server. When the streaming media platform 1202 provides video streaming services, the video stream simulation module 1201 can specify the push address corresponding to each video stream. After the video stream is released , the push address stops providing the corresponding video streaming service.

In some exemplary embodiments, as shown in FIG. 2A , the AI model construction evaluation system also includes a code warehouse 105 , where the code warehouse 105 is configured to store the code information of the AI model, the grouping information of the model, and the construction script of the model. , the strategy code that organizes the operation of the AI model, etc.

For example, the code repository 105 may be a GIT code repository. GIT is an open source distributed version control system for agile and efficient handling of any project, small or large. The code repository 105 can also be other types of code repositories, such as SVN (Subversion), etc., and the embodiment of the present disclosure does not limit this. Each AI model can correspond to a project in the code warehouse.

In Figure 2A, the AI capability evaluation module is located on the physical machine where the test environment is located. The physical machine where the test environment is located is deployed with an AI model. The AI model needs to pull the video stream of the streaming media platform, which is equivalent to the physical machine where the AI capability assessment module is located. The video stream of the streaming media platform needs to be pulled. Therefore, there is a connection relationship between the AI capability evaluation module in Figure 2A and the streaming media platform. In some other exemplary implementations, as shown in Figure 1C, the AI capability evaluation module can also be located on a physical machine other than the physical machine where the test environment is located. In this case, between the AI capability evaluation module in Figure 1C and the streaming media platform There is no connection between them.

In some exemplary implementations, building the AI model of the current group to the test environment includes:

Pull the model code of the currently grouped AI model;

Compile the model code of the currently grouped AI model;

Deploy the AI model of the current group to the test environment.

In some exemplary embodiments, whenever a developer submits model code to the code repository 105, the code repository 105 can automatically trigger the controller 11 to perform a build evaluation process. When the controller 11 builds the AI model of the current group to the test environment, the controller 11 pulls the model code of the AI model of the current group (which can be pulled to the physical machine where the controller 11 is located). The controller 11 (in the controller 11 (on the physical machine where it is located) compiles the model code of the AI model of the current group, and finally the controller 11 deploys the AI model of the current group to the test environment.

In some other exemplary implementations, the model code after compiling the AI model of the current group can be stored in the code warehouse. When the AI model of the current group is not updated, building the AI model of the current group to the test environment includes: pulling The model code after compiling the AI model of the current group; deploy the AI model of the current group to the test environment.

In other exemplary embodiments, the controller 11 starts a unified build evaluation process at regular intervals every day.

Assume that a R&D team of hundreds of people develops dozens or even hundreds of AI models, and each model corresponds to a development group. Then, the controller 11 regularly starts the unified construction and evaluation process for dozens or even hundreds of models every day, so that The model evaluation process becomes more controllable, improving the efficiency of AI technology research and development.

In some exemplary embodiments, the controller 11 is further configured to organize and merge the construction scripts of multiple AI models and then push them to the task queue.

In this embodiment, the controller 11 needs to maintain a task queue, which can include: grouping information of the AI model and the deployment order of the AI model grouping, so that the following two dimensions of control can be achieved through the task queue: 1. How to Group AI models; 2. Deployment order after grouping (that is, which batch of AI models to deploy first, and which batch of AI models to deploy next).

In some examples, the task queue also includes: sequential DevOps process for each AI model. The sequential DevOps process of each AI model includes the process from compilation→deployment→testing→result analysis.

In some exemplary implementations, the build scripts pushed to the task queue can be grouped by specialized grouping personnel. At this time, the grouping personnel can store the grouping code in the code warehouse 105, and the controller 11 pulls the grouping code and obtains the grouping. information.

In other exemplary embodiments, build scripts pushed to the task queue may be grouped by controller 11 .

In some exemplary embodiments, the controller 11 determines model grouping information, including:

Obtain the construction scripts of multiple AI models to be built;

The multiple AI models to be built are grouped according to the resources consumed by each AI model, and the resources consumed by each group of AI models are less than the resources of the test environment.

The AI model construction and evaluation system of the disclosed embodiment, with the help of the Triton inference framework, can run multiple AI models simultaneously on a single GPU to maximize utilization, and can be integrated with Kubernetes for orchestration, indicators and automation. Extension. Optionally, a single GPU can be a GPU on the physical machine where the test environment is located.

In the embodiment of the present disclosure, the multiple AI models to be built are grouped according to the resources consumed by each AI model. The resources here can be hardware resources (such as inference card resource size, etc.) or software resources ( For example, number of threads, etc.). For example, the resource can be a hardware resource. The multiple AI models to be built are grouped according to the hardware resources consumed by each AI model. Since the hardware resources are easier and more convenient to read, the feasibility of model grouping is enhanced. implement.

For example, assuming that 10 AI models are deployed on a system (optional, the system can be a test environment), and the inference card resource size of the system is 16G, then the total inference card occupied by these 10 AI models Resource size should be less than 16G. In some exemplary embodiments, the inference card may be a graphics card, that is, the graphics card may be used for inference. In other exemplary embodiments, the inference card may be a non-graphics card, that is, the inference card may not have a display function. By grouping AI models, the embodiments of this disclosure reduce the requirements on hardware resources of the test environment. Assume that there are 100 AI models that need to be tested. If the AI models are not grouped, then the test environment needs to deploy 100 AI models at the same time. The hardware resource requirements of the environment are relatively high; if the 100 AI models are divided into 10 groups, each group has 10 AI models, then the test environment only needs to deploy 10 AI models in each group at the same time, and deploy 10 groups in turn, which reduces the load of the test environment. Hardware resource requirements.

In some exemplary embodiments, the controller 11 determines model grouping information, further including:

The multiple AI models to be built are grouped according to the relationship between different AI models, and the AI models with dependencies are grouped into one group.

Still taking each group of 10 AI models as an example, among these 10 AI models, there may be business dependencies between some or all models, or there may be no business dependencies between all models, but there are business dependencies. Sexual AI models should be grouped into one group.

In some exemplary embodiments, the controller 11 determines dependencies between multiple AI models (ie, whether there are business dependencies) according to the policy code that organizes the operation of the AI models.

The business dependency described in the embodiments of this disclosure refers to the specific logical relationship between multiple different AI models set in a certain AI application or AI business in order to achieve a certain goal, for example: multiple The execution sequence of AI models, the output of AI model A as the input of AI model B, etc. The controller 11 determines whether there are business dependencies between multiple AI models by organizing the strategy code for running the AI models. The strategy code for organizing the running of the AI models is usually stored in a code warehouse.

For example, suppose there is an AI application that needs to detect women aged 20 to 30 wearing red clothes. Then, the strategy code for organizing the operation of the AI model may be as follows: detect all women in the video stream through model 1, if not detected, return an empty result; use model 2 to detect women between 20 and 30 years old in the output of model 1 If the person between them is not detected, an empty result will be returned; if the person wearing red clothes among the 20- to 30-year-old women output by Model 2 is detected by Model 3, an empty result will be returned. Then, according to the policy code for running the organization's AI model, model 1, model 2 and model 3 have business dependencies and should be divided into one group (in addition to model 1, model 2 and model 3, this group of AI models can also Including other AI models, or not including other AI models, depending on the hardware resources of the test environment), that is, Model 1, Model 2 and Model 3 are constructed and evaluated together.

The multiple AI models to be constructed are grouped according to the video streams required for the AI capability assessment of the multiple AI models, and the AI models that require the same video stream for the AI capability assessment are grouped into one group.

At this time, each AI model can design only one test case, or multiple test cases can be designed. The video streams required by the multiple test cases can be the same or different. For example, model 1, model 2, and model 3 only require video stream a when evaluating their capabilities. Then, model 1, model 2, and model 3 can be grouped into one group. For another example, the capability assessment of Model 1 requires video streams a and b, the capability assessment of Model 2 requires video streams a and c, the capability assessment of Model 3 requires video streams a and d, and the capability assessment of Model 4 requires video streams e and f. , then, model 1, model 2 and model 3 can still be grouped into one group.

In other exemplary embodiments, the controller 11 is configured to:

Determine the model grouping information; according to the grouping order, perform the following operations for the AI models of each group:

Construct the AI model of the current group to the test environment; group the test cases of the AI model of the current group; perform the following operations for the test cases of each group according to the grouping order: send the first notification to the video stream simulation module 1201; receive After receiving the second notification from the video stream simulation module 1201, the third notification is sent to the AI capability evaluation module 121; upon receiving the fourth notification from the AI capability evaluation module 121, the fifth notification is sent to the video stream simulation module 1201.

In the embodiment of the present disclosure, when evaluating the AI capabilities of each grouped AI model, the test cases of each grouped AI model can be grouped, and the test cases of each grouped AI model can be grouped to make the test Maximum efficiency and enables testability with limited resources.

In some exemplary embodiments, the test cases of the currently grouped AI model are grouped according to the CPU resources consumed by the video streams required to run the test cases, and at the same time, the resources consumed by the video streams required to run each group of test cases are smaller than those of streaming media Platform 1202 resources. For example, the CPU resources consumed by the video streams required to run each set of test cases simultaneously are less than the CPU resources of the streaming media platform 1202 . In some other exemplary embodiments, the test cases of the currently grouped AI model can also be grouped based on the fact that the GPU resources consumed by the video stream required to run each group of test cases simultaneously is less than the GPU resources of the streaming media platform 1202. This disclosure The embodiment does not limit this.

In this disclosed embodiment, since the corresponding relationship between test cases and video streams has been set in advance, the controller 11 groups the test cases according to the resource size of the streaming media platform 1202 occupied by the video streams required to run the test case. Resource requirements for the streaming media platform 1202 can be reduced.

For example, taking the current batch of grouped AI models as including 10, assuming that each AI model has 10 test cases (i.e., 10 AI capabilities to be tested), assuming that the resources of the streaming media platform 1202 (exemplary , resource is CPU resource) can provide 10 video streaming services at the same time. Then, each AI model tests 1 test case at a time, then the system can test 10 test cases at the same time in one round (assuming that the video streams of these 10 test cases are different ), a total of 10 rounds of testing are required, that is, the test cases are divided into 10 groups. In actual use, the video streams required for each group of multiple test cases may be the same or may be different, and the embodiments of the present disclosure do not limit this. The controller 11 notifies the video stream simulation module 1201 to prepare the video stream content of the video files required for the test case of the current group, and pushes the video stream content of these video files to the streaming media platform 1202 for video streaming through the video stream simulation module 1201 Provided, after the video stream required for the current grouped test case is ready on the streaming media platform 1202, the controller 11 notifies the AI capability evaluation module 121 to start testing (the AI capability evaluation module 121 notifies the AI model to pull the streaming media platform 1202 video stream, and detect the output results of the AI model. In the embodiment of the present disclosure, after the AI model pulls the video stream of the streaming media platform 1202, it can be processed frame by frame, or it can be processed in multiple frames. , the embodiment of the present disclosure does not limit this). After the testing of this group of test cases is completed, the AI capability evaluation module 121 notifies the controller 11 that the testing is completed. The controller 11 notifies the video stream simulation module 1201 to stop pushing the video stream content, and the video stream simulation module 1201 notifies the streaming media platform 1202 to release the video stream. Then, according to the above method, the controller 11 notifies the video stream simulation module 1201 to prepare the video stream content of the video file required for the test case of the next group until the test case testing of all groups is completed. Then, re-construct the next batch of grouped AI models to the test environment, and test the test cases of the next batch of grouped AI models according to the above method.

In actual use, the test time required for each test case may be the same or different. Therefore, the test cases of each group can be divided into multiple groups, and the test cases using the same video stream are divided into one group. When a test After the test case or the test case of a certain group is completed, the AI capability evaluation module 121 can notify the controller 11 that the test case or the test case of this group is completed, and the controller 11 notifies the video stream simulation module 1201 to stop pushing the test case or the test case. The video stream content corresponding to the group's test case, the video stream simulation module 1201 notifies the streaming media platform 1202 to release the test case or the video stream corresponding to the group's test case, so that the resources of the streaming media platform 1202 can be released as early as possible and improve parallel efficiency. . For example, as shown in Figure 2B, assume that the test cases of the currently grouped AI model include 20 test cases, and the 20 test cases require three video streams. Among them, the 10 test cases A1 to A10 require video streams. a, the five test cases B1 to B5 require video stream b, and the five test cases C1 to C5 require video stream c. Assume that the streaming media platform 1202 can provide video streams a and b at the same time or provide video streams a and c at the same time. Then, when grouping the test cases of the currently grouped AI model, as shown in Figure 2C, these 20 test cases can be divided into 10 groups. Among them, the first group of test cases includes A1 and B1, and the second group of test cases Including A2, B2,..., the fifth group of test cases includes A5, B5, the sixth group of test cases includes A6, C1, the seventh group of test cases includes A7, C2,..., the tenth group of test cases includes A10, C5, in After evaluating the fifth set of test cases, video stream b can be released and video stream c provided, and then the sixth set of test cases can be evaluated. In this disclosed embodiment, the 10 test cases from A1 to A10 can be divided into one group, the 5 test cases from B1 to B5 can be divided into one group, and the 5 test cases from C1 to C5 can be divided into one group. When each group's test case testing is completed, the streaming media platform 1202 releases the video stream corresponding to the group's test case, so that the resources of the streaming media platform 1202 can be released as early as possible and improve parallel efficiency. Assuming that the streaming media platform 1202 can provide video streams a, b, and c at the same time, then, when grouping the test cases of the currently grouped AI model, as shown in Figure 2D, these 20 test cases can be divided into 10 groups, where , the first group of test cases includes A1, B1, C1, the second group of test cases includes A2, B2, C2,..., the fifth group of test cases includes A5, B5, C5, and the sixth group of test cases includes A6 , the seventh group of test cases includes A7,..., and the tenth group of test cases includes A10. During the above test process, each AI model only runs one test case at the same time. In actual use, each AI model can run multiple test cases at the same time. The video streams corresponding to the multiple test cases can be the same or different. This disclosure The embodiment does not limit this.

As an example, the current batch of AI models includes 10 (that is, 10 AI capabilities to be tested), each AI model has 10 test cases, and the 100 test cases correspond to different video streams. Assume The CPU resource performance of the streaming media platform 1202 is very strong, and it can provide video streaming services corresponding to the 100 test cases at the same time, and the hardware resources of the test environment are very strong. Each AI model can test 10 test cases at a time, and the system is in one round. You can test 100 test cases at the same time, that is, all test cases are divided into one group. At this time, the controller 11 notifies the video stream simulation module 1201 to prepare the video stream content of the video files required for these 100 test cases, and pushes the video stream content of these video files to the streaming media platform 1202 through the video stream simulation module 1201 Provide video streams. After the video streams required for these 100 test cases are ready on the streaming media platform 1202, the controller 11 notifies the AI capability evaluation module 121 to start testing (the AI capability evaluation module 121 notifies the AI model to pull the streaming media The video stream of the platform 1202 and detect the output results of the AI model). After the testing of these 100 test cases is completed, the AI capability evaluation module 121 notifies the controller 11 that the test is completed, the controller 11 notifies the video stream simulation module 1201 to stop pushing video stream content, and the video stream simulation module 1201 notifies the streaming media platform 1202 to release the video stream. (Here, when a test case or a test case of a certain group using the same video stream is completed, the controller 11 notifies the video stream simulation module 1201 to release the video file corresponding to the test case or the group's test case. , the video stream simulation module 1201 notifies the streaming media platform 1202 to release the video stream corresponding to the test case or the test case of the group). Then, according to the above method, the controller 11 reconstructs the next batch of grouped AI models to the test environment, and according to the above method, tests the test cases of the next batch of grouped AI models.

In some exemplary implementations, grouping the test cases of the currently grouped AI model further includes: grouping according to video streams required to run the test cases.

For example, assume that AI model 1 needs to detect human faces, and AI model 2 needs to detect human heads, as shown in Figure 2E. If the five test cases from D1 to D5 of AI model 1 require video stream a, E1 to The five test cases of E5 require video stream b, the five test cases of AI model 2 from F1 to F5 require video stream a, and the five test cases of G1 to G5 require video stream c. The streaming media platform 1202 can provide two at the same time. Video streams cannot provide three video streams at the same time. Then, as shown in Figure 2F, the first set of test cases can include D1, E1, and F1 (requiring video streams a and b), and the second set of test cases can include D2 and E2. , F2 (requires video streams a, b), ..., the fifth group of test cases can include D5, E5, F5 (requires video streams a, b), and the sixth group of test cases can include G1, G2 and G3 (requires video streams c), the seventh group of test cases can include G4 and G5 (requires video stream c). When actually using the AI model of the present disclosure to build an evaluation system, the specific method of grouping test cases can be set as needed, and the embodiments of the present disclosure do not limit this.

In some exemplary implementations, when a test case fails to pass the test, relevant developers can be reminded by sending corresponding alarm information, emails, etc., so that the developers can modify the model code as early as possible to improve research and development efficiency.

As shown in Figure 3, this embodiment of the present disclosure combines the video streaming environment requirements required for the AI capability goals targeted by DevOps, combined with the case requirements required for automated verification, uses video files to simulate video streaming technology, and converts the media in the media file library The file is simulated into a video stream, the construction process is completed, and then released, which solves the problem of excessive CPU and memory resource overhead caused by the huge video stream required by some methods to simultaneously carry out the DevOps construction process of multiple AI services.

In some exemplary embodiments, before the test case test of the current group is completed, the video stream simulation module 1201 pushes the video stream content required for the test case of the current group to the streaming media platform 1202 for cyclic provision.

In this embodiment, by pushing the video stream content to the streaming media platform 1202 for cyclic provision, the AI capability evaluation module 121 can determine whether the AI model can provide inference results within a relative playback duration, that is, for each test case Having a relative video start time reduces the difficulty of evaluation. For example, assume that the video stream playback duration of a certain test case is 15s. If the AI model gives the inference result within 15s, the inference is considered successful. Then, since the video stream content is pushed in a loop, if within 15s from a certain moment, If the AI model does not provide an inference result, the AI capability evaluation module 121 may determine that the inference fails. In some other exemplary embodiments, the AI capability evaluation module 121 can accurately determine the start time of the AI model pulling the video stream and the inference time when obtaining the inference result through the timestamp of the video frame, thereby obtaining an accurate inference duration. Although This method of obtaining accurate inference duration does not require looping the video stream, but the execution steps are relatively complex and the evaluation difficulty is relatively high.

In this disclosed embodiment, the video stream simulation module 1201 pushes the video stream content required for the current grouped test case to the streaming media platform 1202 for cyclic provision, including any of the following two situations: 1) Video stream simulation module 1201 cyclically pushes the video stream content required for the test case of the current group to the streaming media platform 1202, so that the streaming media platform 1202 cyclically provides the video stream required for the test case of the current group; 2) the video stream simulation module 1201 will The video stream content required for the test case of the current group is pushed to the streaming media platform 1202 (pushed only once), but the streaming media platform 1202 provides the video stream required for the test case of the current group in a loop, that is, the streaming media platform 1202 has a loop mode, at this time, the video stream simulation module 1201 only needs to push the video stream content once, and the streaming media platform 1202 provides the video stream in a loop.

In some exemplary embodiments, as shown in FIG. 2A , the AI model building evaluation system further includes a media file library 106 , wherein the media file library 106 is configured to store video files required for evaluating the AI model of each group. .

In this disclosed embodiment, the video files corresponding to the test cases are stored in the media file library 106 . From a playback perspective, the video stream simulation module 1201 plays the video file from the media file library 106, and then pushes the video stream content of the played video file to the streaming media platform 1202. The video files in the embodiment of the present disclosure are videos that are pre-stored in the media file library 106 and are used to evaluate the function or performance of the AI model to achieve automated testing. By setting up the media file library 106 (located on a server outside the streaming media platform 1202), the streaming media platform 1202 can be more focused on implementing the push-pull streaming function. The storage of video files required by the streaming media platform 1202 is completed by the media file library 106. , which can provide richer video file content, which can make the evaluation of the AI model more accurate.

In some exemplary implementations, when the format of the video file is a non-streaming media file format, the video stream simulation module 1201 notifies the streaming media platform 1202 to convert the format of the video file into a streaming media file format.

In this embodiment of the present disclosure, the format of the video file may be MPEG-1 or any other video file format, and the streaming media platform 1202 may convert it into MPEG-4 or any other streaming media file format. This embodiment of the present disclosure is suitable for This is not a limitation.

In some exemplary implementations, as shown in Figures 4 and 5, the AI model construction evaluation system simulates on demand the construction evaluation process of the video streaming environment required in the AI intelligent service DevOps process, including the following steps:

1) R&D personnel submit code to the GIT code warehouse, and the GIT code warehouse automatically triggers the DevOps build evaluation process;

2) The DevOps controller organizes the build process, merges it and pushes it to the task queue;

3) The DevOps controller obtains the build task, pulls the group code from the GIT code warehouse, obtains the model group information, then pulls the model code of the current batch group, compiles and deploys it to the test environment;

4) The DevOps controller reads the configuration involved in all AI capability catalogs (here, the AI capability catalog refers to the list of AI functions that need to be tested, that is, test cases) in the current batch group (this configuration includes the AI model and all Required video stream address list), sort out the video stream address list required for testing;

5) The DevOps controller sends the required video stream address list to the video stream simulation module for processing;

6) The video stream simulation module obtains the video stream information in the case library (determines whether the video file format is a streaming media format, and if not, notifies the streaming media platform to convert the video file format to a streaming media file format), and pulls the corresponding video file;

7) The video stream simulation module pushes the video stream content of the video file to the streaming media platform for cyclic provision based on the one-to-one correspondence between the video file and the video stream address (the video stream simulation module cyclically pushes the video stream content of the video file to The streaming media platform, or the streaming media platform provides the video streaming content of the video files in a loop) and verifies them one by one;

8) After receiving the signal that the video streaming environment construction is complete, the DevOps controller starts the AI capability evaluation module to evaluate the model. After the model evaluation is completed, it sends a release instruction to the video streaming simulation module;

9) The video stream simulation module stops pushing video stream content that is no longer in use based on the usage of the video stream, and sends a release instruction to the streaming media platform to complete the release of the video stream.

In Figure 5, when an error occurs in the DevOps controller and video stream simulation module during the execution of the above steps (for example, the DevOps controller does not pull the group code from the GIT code repository, the DevOps controller does not read the current Configurations involved in all AI capability directories in the batch group, video stream simulation modules have not obtained video stream information, etc.), relevant developers can be reminded by sending corresponding alarm information, emails, etc., so that developers can modify the build as soon as possible Scripts or configurations to improve R&D efficiency.

The AI model construction and evaluation system provided by the embodiments of the present disclosure temporarily builds the required video stream according to the needs of the evaluation object, and releases it immediately after completing the model evaluation, which greatly reduces the resources required in the DevOps environment and makes the model evaluation process more feasible. control; the model evaluation process has a video start time for the case, which reduces the difficulty of evaluation; the construction, evaluation, and online processes can be carried out in groups, which improves the efficiency of AI technology research and development and reduces costs.

As shown in Figure 6, embodiments of the present disclosure also provide an AI model construction and evaluation method, including:

Step 601: The controller determines the grouping information of the AI model;

Step 602: According to the grouping order, perform the following operations on the AI models of each group:

The controller builds the AI model of the current group to the test environment and sends the first notification to the simulation evaluation module; the simulation evaluation module receives the first notification from the controller and provides video streaming services to the AI model. The content of the video streaming service is to evaluate the current grouping. The video stream content of the video file required by the AI model; perform an AI capability assessment on the currently grouped AI model; release the video stream after the capability assessment is completed.

The AI model construction and evaluation method of the embodiment of the present disclosure groups the AI models, sequentially constructs the AI models of each group to the test environment, and provides the streaming media services required for evaluation on demand for the currently grouped AI models. During the model evaluation After completion, the video stream is released, and there is no need to maintain the video stream of all AI models at all times. This not only solves the problem of wasting a lot of resources in the video streaming environment, but also greatly reduces the computing resources required in the DevOps environment of the AI model, and also makes the model evaluation process more efficient. Controllable, it improves the efficiency of AI technology research and development and reduces costs.

In some exemplary implementations, the controller determines grouping information of the AI model, including:

The controller obtains the construction scripts of multiple AI models to be built;

The controller groups the multiple AI models to be built according to the resources consumed by each AI model. The hardware resources consumed by each group of AI models are less than the hardware resources of the test environment.

For example, assuming that 10 AI models are deployed on a system, and the inference card resource size of the system is 16G, then the total inference card resource size occupied by these 10 AI models should be less than 16G.

The controller groups the multiple AI models to be built according to the resources consumed by each AI model and the relationship between different AI models. The hardware resources consumed by each group of AI models are less than the hardware of the test environment. Resources, AI models with dependencies are grouped into a group.

Still taking the above 10 AI models as an example, among these 10 AI models, there may be business dependencies between some or all models, and the AI models with dependencies are grouped into one group.

In some exemplary embodiments, the controller determines dependencies between multiple AI models according to a policy code that organizes the operation of the AI models.

In other exemplary implementations, the AI model construction and evaluation method further includes:

The controller obtains the grouping information of the test cases of the currently grouped AI model;

The AI capability evaluation of the currently grouped AI models includes: performing an AI capability evaluation on the test cases of each group according to the grouping order of the test cases.

In some exemplary implementations, the controller groups the test cases of the currently grouped AI model according to the resources consumed by the video stream required to simulate the test case, and at the same time, the resources consumed by the video stream required to simulate each group of test cases are less than the resource consumed by the video stream. Media platform resources.

In some exemplary embodiments, before the test case test of the current group is completed, the video stream simulation module pushes the video stream content of the video file required by the test case of the current group to the streaming media platform to cyclically provide the test case of the current group. The video stream required for the test case (the video stream simulation module cyclically pushes the video stream content of the video file required to evaluate the current grouped AI model to the streaming media platform, or the streaming media platform cyclically provides the video stream content of the video file ).

In some exemplary implementations, after the test case of a certain test case or a test case of a certain group is tested, the video stream simulation module notifies the streaming media platform to release the video stream corresponding to the test case or the test case of the group so as to release it as soon as possible. Resources for streaming platforms.

In some exemplary implementations, the controller builds the current grouped AI model to the test environment, including:

Pull the model code of the currently grouped AI model;

Compile the model code of the currently grouped AI model;

Deploy the AI model of the current group to the test environment.

In some exemplary implementations, the grouping information of the AI model, the code information of the AI model, the construction script of the AI model, and the policy code for organizing the operation of the AI model are all stored in the code warehouse.

For example, the code repository can be a GIT code repository. GIT is an open source distributed version control system for agile and efficient handling of any project, small or large. Whenever developers submit code to the GIT code repository, the GIT code repository can automatically trigger the DevOps build evaluation process.

In some example embodiments, the video files required to evaluate the AI model for each group are stored in a media file library.

In some exemplary implementations, when the format of the video file is a non-streaming media file format, the video stream simulation module notifies the streaming media platform to convert the format of the video file into a streaming media file format.

As shown in Figure 7, the embodiment of the present disclosure also provides an AI model construction and evaluation method, which is applied to the controller. The AI model construction and evaluation method includes:

Step 701: Determine model grouping information;

Step 702: Perform the following operations on the AI model of each group according to the grouping order: build the AI model of the current group to the test environment, and notify the video stream simulation module to evaluate the video stream content of the video file required by the AI model of the current group. Push it to the streaming media platform, and notify the AI capability evaluation module to use the video stream of the streaming media platform to evaluate the AI capability of the current grouped AI model. After the capability assessment is completed, notify the video stream simulation module and streaming media platform to release the video stream.

In some exemplary implementations, determining model grouping information includes:

Obtain the construction scripts of multiple AI models to be built;

The multiple AI models to be built are grouped according to the resources consumed by each AI model, and the hardware resources consumed by each group of AI models are less than the hardware resources of the test environment.

Obtain the construction scripts of multiple AI models to be built;

The multiple AI models to be built are grouped according to the resources consumed by each AI model and the relationship between different AI models. The hardware resources consumed by each group of AI models are less than the hardware resources of the test environment, AI models with dependencies are grouped into a group.

In some exemplary implementations, dependencies between multiple AI models are determined according to a policy code that organizes the operation of the AI model.

In some exemplary embodiments, the video stream simulation module in step 702 pushes the video stream content of the video file required to evaluate the AI model of the current group to the streaming media platform, and notifies the AI capability evaluation module to use the video of the streaming media platform. The stream performs an AI capability assessment on the current grouped AI model. After the capability assessment is completed, it notifies the video stream simulation module and the streaming media platform to release the video stream, including:

Group the test cases of the currently grouped AI model;

According to the grouping order, perform the following operations for each grouped test case: notify the video stream simulation module to push the video stream content of the video file required for the current grouped test case to the streaming media platform, so that it can be displayed on the streaming media platform. Form a video stream that can be pulled by the AI model; notify the AI capability evaluation module to use the video stream of the streaming media platform to test the test cases of each group. After the test of a certain test case is completed or the test of the current group After the case test is completed, the video stream simulation module is notified to stop pushing the video stream content, and the video stream simulation module notifies the streaming media platform to release the video stream.

In some exemplary implementations, the test cases of the currently grouped AI model are grouped according to the resources consumed by the video streams required to simulate the test cases, and at the same time, the resources consumed by the video streams required to simulate each group of test cases are smaller than those of the streaming media platform H.

Pull the model code of the currently grouped AI model;

Compile the model code of the currently grouped AI model;

Deploy the AI model of the current group to the test environment.

In some exemplary implementations, notifying the video stream simulation module and the streaming media platform to release the video stream includes:

The video stream simulation module is notified to stop pushing the video stream content, and the video stream simulation module sends a video stream release instruction to the streaming media platform.

In this disclosed embodiment, the video source is a video stream simulation module, which pushes streams to the streaming media platform; the AI model pulls streams from the streaming media platform, and the streaming media platform is in the middle, equivalent to a forwarding function, equivalent to AI The model is connected to the video streaming simulation module through the streaming platform.

In one example, the controller may include: a first processor, a first memory and a first bus system, wherein the first processor and the first memory are connected through the first bus system, the first memory is used to store instructions, and the first bus system is used to store instructions. A processor is used to execute instructions stored in the first memory. Specifically, the first processor determines the model grouping information; according to the grouping order, performs the following operations for the AI model of each group: builds the AI model of the current group to the test environment, and notifies the video stream simulation module to evaluate the AI model of the current group The video stream content of the required video file is pushed to the streaming media platform, and the AI capability assessment module is notified to use the video stream of the streaming media platform to evaluate the AI capabilities of the current grouped AI model. After the capability assessment is completed, the video stream simulation module is notified and Streaming platforms unleash video streaming.

It should be understood that the first processor can be a central processing unit (Central Processing Unit, CPU), and the first processor can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), off-the-shelf programmable processors, etc. Gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

The first memory may include read-only memory and random access memory and provide instructions and data to the first processor. A portion of the first memory may also include non-volatile random access memory. For example, the first memory may also store device type information.

In addition to the data bus, the first bus system may also include a power bus, a control bus, a status signal bus, etc.

During implementation, the processing performed by the processing device may be completed by instructions in the form of hardware integrated logic circuits or software in the first processor. That is to say, the method steps of the embodiments of the present disclosure may be implemented by a hardware processor, or may be executed by a combination of hardware and software modules in the processor. Software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media. The storage medium is located in the first memory, and the first processor reads the information in the first memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

Embodiments of the present disclosure also provide a storage medium on which a computer program is stored. When the program is executed by a processor, the AI model construction and evaluation method described in any embodiment of the present disclosure is implemented. This AI model construction and evaluation method can determine model grouping information; according to the grouping order, perform the following operations for the AI model of each group: build the AI model of the current group to the test environment, and notify the video stream simulation module to evaluate the AI model of the current group The video stream content of the required video file is pushed to the streaming media platform, and the AI capability assessment module is notified to use the video stream of the streaming media platform to evaluate the AI capabilities of the current grouped AI model. After the capability assessment is completed, the video stream simulation module is notified and The streaming media platform releases video streams. Since there is no need to maintain the video streams of all AI models at all times, it not only solves the problem of wasting a lot of resources in the video streaming environment, but also significantly reduces the computing resources required in the DevOps environment of the AI models, and also makes the model evaluation process easier. It is more controllable, improves the efficiency of AI technology research and development, and reduces costs. The method of driving AI model construction and evaluation by executing executable instructions is basically the same as the AI model construction and evaluation method provided by the above embodiments of the present disclosure, and will not be described again here.

As shown in Figure 8, an embodiment of the present disclosure also provides a video stream simulation method, including:

Step 801: The video stream simulation module receives a list of video file addresses required to evaluate the AI model of the current group;

Step 802: The video stream simulation module pulls the corresponding video file from the video file address list, and pushes the video stream content of the video file to the streaming media platform, notifying the controller that the streaming media environment is completed;

Step 803: The video stream simulation module receives the release instruction from the controller and stops pushing the video stream content;

Step 804: The video stream simulation module sends a video stream release instruction to the streaming media platform.

During the Devops model evaluation process, if a large amount of resources are invested in simulating multiple video streams at the same time and starting and stopping according to the needs of the request, although this video stream simulation method is relatively easy to develop, it requires a lot of resources and the Devops environment requirements are high, so The large amount of CPU and memory computing resources used is greatly wasted. The video stream simulation method of the embodiment of the present disclosure provides streaming services required for evaluation on demand by grouping AI models. After the model evaluation is completed, the video stream is released, and there is no need to keep all AI models at all times. The video stream of the model not only solves the problem of wasting a lot of resources in the video streaming environment, but also greatly reduces the computing resources required in the DevOps environment of the AI model. It also makes the model evaluation process more controllable, improves the efficiency of AI technology research and development, and reduces costs. .

In actual application, the video source can be a camera, and the camera pushes the stream to the streaming media platform; the AI inference platform pulls the stream from the streaming media platform, and the streaming media platform is in the middle, equivalent to a forwarding function, which is equivalent to the AI inference platform passing the stream The media platform establishes a connection with the camera. During the system construction and evaluation process, the video source is the video stream simulation module, which pushes streams to the streaming media platform; the AI inference platform pulls streams from the streaming media platform, the streaming media platform is in the middle, and the AI inference platform passes the streaming media platform Establish a connection with the video streaming simulation module.

In some exemplary embodiments, the video files required to evaluate the AI model of each group are stored in a media file library, and the video stream simulation module pulls the corresponding video files from the media file library.

In some exemplary implementations, the video stream simulation module pushes the video stream content of the video file to the streaming media platform for cyclic provision.

In this disclosed embodiment, the video stream simulation module pushes the video stream content of the video file to the streaming media platform for cyclic provision, including any of the following two situations: 1) The video stream simulation module pushes the video stream content of the video file to Cyclically pushed to the streaming media platform, so that the streaming media platform cyclically provides the video stream of the video file; 2) The video stream simulation module pushes the video stream content of the video file to the streaming media platform (only pushed once), but the streaming media platform Provides a video stream of video files in a loop.

In one example, the video stream simulation module may include: a second processor, a second memory, and a second bus system, wherein the second processor and the second memory are connected through the second bus system, and the second memory is used to store instructions. , the second processor is used to execute instructions stored in the second memory. Specifically, the second processor receives the video file address list required to evaluate the AI model of the current group, pulls the corresponding video file from the video file address list, and pushes the video stream content of the video file to the streaming media platform, notifying The construction of the controller streaming media environment is completed; it receives the release instruction of the controller, stops pushing the video stream content, and sends the video stream release instruction to the streaming media platform.

It should be understood that the second processor can be a central processing unit (Central Processing Unit, CPU). The second processor can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), off-the-shelf programmable processors, etc. Gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

The second memory may include read-only memory and random access memory and provide instructions and data to the second processor. A portion of the second memory may also include non-volatile random access memory. For example, the second memory may also store device type information.

In addition to the data bus, the second bus system may also include a power bus, a control bus, a status signal bus, etc.

During implementation, the processing performed by the processing device may be completed by instructions in the form of hardware integrated logic circuits or software in the second processor. That is to say, the method steps of the embodiments of the present disclosure may be implemented by a hardware processor, or may be executed by a combination of hardware and software modules in the processor. Software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media. The storage medium is located in the second memory, and the second processor reads the information in the second memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

An embodiment of the present disclosure also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the video stream simulation method as described in any embodiment of the present disclosure is implemented. The video stream simulation method can receive a video file address list required to evaluate the current grouped AI model, pull the corresponding video file from the video file address list, and push the video stream content of the video file to the streaming media platform, Notify the controller that the construction of the streaming media environment is completed; receive the release instruction from the controller, stop pushing the video stream content, and send the video stream release instruction to the streaming media platform. There is no need to maintain the video streams of all AI models at all times. It solves the problem of a large amount of resource waste in the video streaming environment, greatly reduces the computing resources required in the DevOps environment of the AI model, and makes the model evaluation process more controllable, improving the efficiency of AI technology research and development and reducing costs. The method of driving video stream simulation by executing executable instructions is basically the same as the video stream simulation method provided by the above embodiments of the present disclosure, and will not be described again here.

Those of ordinary skill in the art can understand that all or some steps, systems, and functional modules/units in the devices disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may consist of several physical components. Components execute cooperatively. Some or all of the components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. removable, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disk (DVD) or other optical disk storage, magnetic cassettes, tapes, disk storage or other magnetic storage devices, or may Any other medium used to store the desired information and that can be accessed by a computer. Additionally, it is known to those of ordinary skill in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .

Although the embodiments disclosed in the present disclosure are as above, the described contents are only used to facilitate the understanding of the present disclosure and are not intended to limit the present invention. Any person skilled in the art can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope of the disclosure. However, the patent protection scope of the present invention must still be based on the above. The scope defined by the appended claims shall prevail.

Claims

An AI model construction and evaluation system includes: a controller and a simulation evaluation module, wherein:

The controller is configured to determine the grouping information of the AI model; according to the grouping order, perform the following operations for the AI model of each group: construct the AI model of the current group to the test environment, and send the first notification to the simulation evaluation module;

The simulation evaluation module is configured to receive a first notification from the controller and provide a video streaming service to the AI model. The content of the video streaming service is a video stream of video files required to evaluate the AI model of the current group. Content; perform AI capability assessment on the currently grouped AI model; release the video stream after the capability assessment is completed.
The AI model construction evaluation system according to claim 1, wherein the simulation evaluation module includes a video stream simulation server and an AI capability evaluation module;

The video stream simulation server is configured to receive a first notification from the controller and provide a video stream service to the AI model. The content of the video stream service is the video of the video file required to evaluate the AI model of the current group. Stream content, send a second notification to the controller, so that the controller sends a third notification to the AI capability evaluation module; receive a fifth notification from the controller, release the video stream;

The AI capability evaluation module is configured to receive the third notification from the controller, perform an AI capability evaluation on the AI model of the current group; after the capability evaluation is completed, send a fourth notification to the controller, so that the controller sends the third notification. Five notifications to the video streaming simulation server;

The controller is further configured to receive a second notification from the video stream simulation server and send a third notification to the AI capability assessment module; upon receiving a fourth notification from the AI capability assessment module, send a fifth notification. Notify the video stream simulation server.
The AI model construction evaluation system according to claim 2, wherein the video stream simulation server includes a video stream simulation module and a streaming media platform;

The video stream simulation module is configured to receive a first notification from the controller, push the video stream content of the video file required to evaluate the AI model of the current group to the streaming media platform, and send a second notification to The controller; receiving the fifth notification from the controller, stops pushing the video stream content, and sends the sixth notification to the streaming media platform;

The streaming media platform is configured to receive the video stream content pushed by the video stream simulation module and provide video streaming services to the AI model; and release the video stream upon receiving the sixth notification from the video stream simulation module.
The AI model construction evaluation system according to claim 3, wherein the video stream simulation module cyclically pushes the video stream content of the video files required to evaluate the currently grouped AI model to the streaming media platform, or, the The streaming media platform cyclically provides video streaming content of the video file.
The AI model construction evaluation system according to claim 3, wherein the video files required to evaluate the AI model of each group are stored in the media file library;

Pushing the video stream content of the video files required to evaluate the AI model of the current group to the streaming media platform includes: pulling the video files required to evaluate the AI model of the current group from the media file library, and push the video stream content of the video file to the streaming media platform.
The AI model construction evaluation system according to claim 1, wherein the determining the grouping information of the AI model includes:

Obtain the construction scripts of multiple AI models to be built;

The multiple AI models to be built are grouped according to the resources consumed by each AI model, and the resources consumed by each group of AI models are less than the resources of the test environment.
The AI model construction evaluation system according to claim 1, wherein the determining the grouping information of the AI model includes:

Obtain the construction scripts of multiple AI models to be built;

The multiple AI models to be built are grouped according to the resources consumed by each AI model and the relationship between different AI models. The resources consumed by each group of AI models are less than the resources of the test environment, and there are Dependent AI models are grouped into a group.
The AI model construction evaluation system according to claim 7, wherein dependencies between a plurality of the AI models are determined according to a policy code that organizes the operation of the AI models.
The AI model construction evaluation system according to claim 1, wherein before sending the first notification to the simulation evaluation module, the controller is further configured to group the test cases of the currently grouped AI model; according to In the grouping order of the test cases, the operation of sending the first notification to the simulation evaluation module is performed for each grouped test case.
The AI model construction evaluation system according to claim 9, wherein the simulation evaluation module includes a streaming media platform, and the streaming media platform is configured to provide video streaming services to the AI model;

Group the test cases of the currently grouped AI model according to the resources consumed by the video stream required to run the test case, and at the same time, the resources consumed by the video stream required to run each group of test cases are less than the resources of the streaming media platform .
The AI model construction evaluation system according to claim 9, wherein the video stream is released after the capability evaluation is completed, specifically: after the test case test corresponding to one of the video streams is completed, the video is released. flow.
The AI model construction evaluation system according to claim 1, wherein said constructing the current grouped AI model to the test environment includes:

Pull the model code of the AI model of the current group;

Compile the model code of the currently grouped AI model;

Deploy the AI model of the current group to a test environment.
The AI model construction evaluation system according to claim 1, wherein the grouping information of the AI model, the code information of the AI model, the construction script of the AI model, and the strategy code for organizing the operation of the AI model are all stored in the code repository.
The AI model construction evaluation system according to claim 1, wherein the controller is further configured to: organize and merge the construction scripts of multiple AI models and then push them to a task queue, the task queue includes: the AI The grouping information of the model and the deployment order of the AI model grouping.
The AI model construction evaluation system according to claim 1, wherein the controller is further configured to: when determining that the grouping information of the AI model fails, send alarm information or an alarm email;

The simulation evaluation module is also configured to: when providing the video streaming service to the AI model fails, send an alarm message or an alarm email.
An AI model construction and evaluation method, including:

The controller determines model grouping information;

According to the grouping order, perform the following operations on the AI model of each group:

The controller constructs the AI model of the current group to the test environment, and sends the first notification to the simulation evaluation module;

The simulation evaluation module receives the first notification from the controller and provides a video streaming service to the AI model. The content of the video streaming service is the video streaming content of the video file required to evaluate the AI model of the current group; Perform an AI capability assessment on the currently grouped AI model; after the capability assessment is completed, release the video stream.
An AI model construction and evaluation method, including:

The controller determines model grouping information;

The controller performs the following operations on the AI model of each group in accordance with the grouping order: constructs the AI model of the current group to the test environment, and notifies the video stream simulation module to evaluate the video stream of the video file required for the AI model of the current group. The content is pushed to the streaming media platform, and the AI capability evaluation module is notified to use the video stream of the streaming media platform to evaluate the AI capability of the current grouped AI model. After the capability assessment is completed, the video stream simulation module and the streaming media platform are notified to release the video stream. .
The AI model construction and evaluation method according to claim 17, wherein the notification video stream simulation module pushes the video stream content of the video file required to evaluate the currently grouped AI model to the streaming media platform, and notifies the AI capability evaluation module to use The video stream of the streaming media platform performs an AI capability assessment on the currently grouped AI model. After the capability assessment is completed, the video stream simulation module and the streaming media platform are notified to release the video stream, including:

Group the test cases of the currently grouped AI model;

According to the grouping order, perform the following operations for each grouped test case: notify the video stream simulation module to push the video stream content of the video file required for the current grouped test case to the streaming media platform, so that it can be displayed on the streaming media platform. Form a video stream that can be pulled by the AI model; notify the AI capability evaluation module to use the video stream of the streaming media platform to test the test cases of each group. After the test case test is completed, notify the video stream simulation module to stop The video stream content is pushed, and the video stream simulation module notifies the streaming media platform to release the video stream.
A controller comprising a memory; and a processor coupled to the memory, the processor configured to execute the method of any one of claims 17 to 18 based on instructions stored in the memory Steps of AI model construction and evaluation methods.
A storage medium with a computer program stored thereon, which when executed by a processor implements the AI model construction and evaluation method according to any one of claims 17 to 18.
A video stream simulation method, including:

The video stream simulation module receives a list of video file addresses required to evaluate the AI model of the current group;

The video stream simulation module pulls the corresponding video file from the video file address list, pushes the video stream content of the video file to the streaming media platform, and notifies the controller that the streaming media environment is completed;

The video stream simulation module receives the release instruction of the controller and stops pushing the video stream content;

The video stream simulation module sends a video stream release instruction to the streaming media platform.
The video stream simulation method according to claim 21, wherein the video stream simulation module cyclically pushes the video stream content of the video file to the streaming media platform, or the streaming media platform cyclically provides the video The video stream content of the file.
A video stream simulation module comprising a memory; and a processor coupled to the memory, the processor being configured to execute as claimed in any one of claims 21 to 22 based on instructions stored in the memory The steps of the video stream simulation method described above.
A storage medium on which a computer program is stored, which implements the video stream simulation method as claimed in any one of claims 21 to 22 when executed by a processor.