CN117642728A - AI model construction evaluation system, video stream simulation module and method, and controller - Google Patents
AI model construction evaluation system, video stream simulation module and method, and controller Download PDFInfo
- Publication number
- CN117642728A CN117642728A CN202280002046.XA CN202280002046A CN117642728A CN 117642728 A CN117642728 A CN 117642728A CN 202280002046 A CN202280002046 A CN 202280002046A CN 117642728 A CN117642728 A CN 117642728A
- Authority
- CN
- China
- Prior art keywords
- model
- video stream
- video
- controller
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004088 simulation Methods 0.000 title claims abstract description 164
- 238000011156 evaluation Methods 0.000 title claims abstract description 133
- 238000000034 method Methods 0.000 title claims description 72
- 238000010276 construction Methods 0.000 title claims description 45
- 238000012360 testing method Methods 0.000 claims abstract description 267
- 238000013515 script Methods 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 description 33
- 238000012854 evaluation process Methods 0.000 description 17
- 238000005516 engineering process Methods 0.000 description 13
- 238000012545 processing Methods 0.000 description 13
- 238000012827 research and development Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 9
- 238000011161 development Methods 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 5
- 239000002699 waste material Substances 0.000 description 5
- 230000007547 defect Effects 0.000 description 4
- 238000013461 design Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 125000004122 cyclic group Chemical group 0.000 description 3
- 238000013136 deep learning model Methods 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- GPRLSGONYQIRFK-MNYXATJNSA-N triton Chemical compound [3H+] GPRLSGONYQIRFK-MNYXATJNSA-N 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000035800 maturation Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/10—Requirements analysis; Specification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- Quality & Reliability (AREA)
- Entrepreneurship & Innovation (AREA)
- Computer Hardware Design (AREA)
- Strategic Management (AREA)
- Economics (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Game Theory and Decision Science (AREA)
- Educational Administration (AREA)
- Development Economics (AREA)
- Software Systems (AREA)
- Operations Research (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Debugging And Monitoring (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Mobile Radio Communication Systems (AREA)
- Monitoring And Testing Of Transmission In General (AREA)
Abstract
An AI model building evaluation system includes: a controller (11) and a simulation evaluation module (12), the controller (11) being configured to determine grouping information of the AI model; the following operations are performed for the AI model of each packet in the order of the packets: constructing an AI model of the current group to a test environment, and sending a first notification to a simulation evaluation module (12); the simulation evaluation module (12) is configured to receive a first notification from the controller (11), provide a video streaming service to the AI model, the content of the video streaming service being the video streaming content of a video file required to evaluate the AI model of the current packet; carrying out AI capacity evaluation on the AI model of the current group; and after the capability assessment is completed, releasing the video stream.
Description
The embodiment of the disclosure relates to the technical field of intelligent systems, and particularly relates to an AI model construction evaluation system, a video stream simulation module and method and a controller.
Computer vision artificial intelligence (Artificial Intelligence, AI) technology based on video processing requires a significant amount of graphics processor (Graphics Processing Unit, GPU) resources to process. A dynamic model based on Triton calls a framework (Triton is an open source software issued by Inlet and Vicat (NVIDIA)) to dynamically arrange and process the AI model according to the needs so as to achieve the aim of saving physical resources. However, in the Development process, the update of each model may involve precision changes of a plurality of AI functions, so that in order to solve the dependency relationship between AI models more quickly and at lower cost, a process of packaging, deploying, testing and online based on a complete set of Development Operations and maintenance (Development and maintenance integration) needs to be introduced, and the changed models are subjected to omnibearing automatic evaluation, so as to determine whether the effect achieved by the update of each model is a positive effect or a negative effect.
Disclosure of Invention
The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.
The embodiment of the disclosure provides an AI model building evaluation system, comprising: the controller and simulation evaluation module, wherein:
the controller is configured to determine grouping information of an AI model; the following operations are performed for the AI model of each packet in the order of the packets: constructing an AI model of a current group to a test environment, and sending a first notification to a simulation evaluation module;
the simulation evaluation module is configured to receive a first notification of the controller, provide video streaming service for the AI model, wherein the content of the video streaming service is the video streaming content of a video file required by the AI model of the current packet; carrying out AI capacity evaluation on the AI model of the current group; and after the capability assessment is completed, releasing the video stream.
The embodiment of the disclosure also provides an AI model construction evaluation method, which comprises the following steps:
the controller determines model grouping information;
the following operations are performed for the AI model of each packet in the order of the packets:
the controller builds an AI model of the current group to a test environment and sends a first notification to the simulation evaluation module;
The simulation evaluation module receives a first notification of the controller and provides video streaming service for the AI model, wherein the content of the video streaming service is the video streaming content of a video file required by the AI model for evaluating the current packet; carrying out AI capacity evaluation on the AI model of the current group; and after the capability assessment is completed, releasing the video stream.
The embodiment of the disclosure also provides an AI model construction evaluation method, which comprises the following steps:
the controller determines model grouping information;
the controller performs the following operations for the AI model of each packet in the order of the packets: the AI model of the current packet is constructed to a test environment, a video stream simulation module is notified to push video stream contents of video files required by the AI model of the current packet to a streaming media platform, an AI capability assessment module is notified to carry out AI capability assessment on the AI model of the current packet by using the video stream of the streaming media platform, and after the capability assessment is completed, the video stream simulation module and the streaming media platform are notified to release the video stream.
The embodiment of the disclosure also provides a controller, which comprises a memory; and a processor coupled to the memory, the processor configured to perform the steps of the AI model build evaluation method as described in any one of the embodiments of the disclosure based on instructions stored in the memory.
The embodiments of the present disclosure also provide a storage medium having stored thereon a computer program which, when executed by a processor, implements the AI model building evaluation method according to any of the embodiments of the present disclosure.
The embodiment of the disclosure also provides a video stream simulation method, which comprises the following steps:
the video stream simulation module pulls the corresponding video file from the video file address list, pushes the video stream content of the video file to the streaming media platform, and informs the controller that the streaming media environment is constructed;
the video stream simulation module receives a release instruction of the controller and stops pushing the video stream content;
and the video stream simulation module sends a video stream release instruction to the streaming media platform.
The embodiment of the disclosure also provides a video stream simulation module, which comprises a memory; and a processor coupled to the memory, the processor configured to perform the steps of the video stream simulation method according to any of the embodiments of the present disclosure based on instructions stored in the memory.
The embodiments of the present disclosure also provide a storage medium having stored thereon a computer program which, when executed by a processor, implements a video stream simulation method according to any of the embodiments of the present disclosure.
Other aspects will become apparent upon reading and understanding the accompanying drawings and detailed description.
The accompanying drawings are included to provide a further understanding of the disclosed embodiments and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain, without limitation, the embodiments of the disclosure. The shapes and sizes of various components in the drawings are not to scale true, and are intended to be illustrative of the present disclosure.
FIGS. 1A, 1B and 1C are schematic structural diagrams of three AI model building assessment systems in accordance with exemplary embodiments of the present disclosure;
FIG. 2A is a schematic diagram of an alternative AI model building assessment system in accordance with an exemplary embodiment of the disclosure;
FIG. 2B is a diagram of a test case and desired video flow of an AI model in accordance with an exemplary embodiment of the disclosure;
FIGS. 2C and 2D are schematic diagrams illustrating two methods of group testing the test cases shown in FIG. 2B;
FIG. 2E is a diagram of test cases and desired video streams for another AI model in accordance with an exemplary embodiment of the disclosure;
FIG. 2F is a schematic diagram of an exemplary method of group testing the test cases shown in FIG. 2E;
FIG. 3 is a schematic diagram of a method of grouping video streams required for simulation in accordance with an exemplary embodiment of the present disclosure;
FIG. 4 is a schematic diagram of a DevOps task initiation process in accordance with an exemplary embodiment of the present disclosure;
FIG. 5 is a schematic diagram of a video environment construction process according to an exemplary embodiment of the present disclosure;
FIG. 6 is a flow chart of an AI model building evaluation method in accordance with an exemplary embodiment of the disclosure;
FIG. 7 is a flow chart of another AI model building assessment method in accordance with an exemplary embodiment of the disclosure;
fig. 8 is a flow chart of a video stream simulation method according to an exemplary embodiment of the present disclosure.
For the purposes of making the objects, technical solutions and advantages of the present disclosure more apparent, embodiments of the present disclosure will be described in detail hereinafter with reference to the accompanying drawings. It should be noted that, without conflict, the embodiments of the present disclosure and features of the embodiments may be arbitrarily combined with each other.
Unless otherwise defined, technical or scientific terms used in the disclosure of the embodiments of the present disclosure should be given the ordinary meaning as understood by one of ordinary skill in the art to which the present disclosure pertains. The terms "first," "second," and the like, as used in embodiments of the present disclosure, do not denote any order, quantity, or importance, but rather are used to distinguish one element from another. The word "comprising" or "comprises", and the like, is intended to mean that elements or items preceding the word encompass the elements or items listed thereafter and equivalents thereof without precluding other elements or items.
If the cycle frequency of the process of code change, model reconstruction, model evaluation and model online can be accelerated, the whole AI model and strategy maturation process is accelerated, and the research and development efficiency is improved. When the number of the models and the AI strategies maintained at the same time is relatively large, the video streams on which the whole set of AI models and strategies depend also reach hundreds of scales, and if the environment for always keeping so many video streams exists, a large amount of resources of a central processing unit (Central Processing Unit, CPU) are required to be consumed, and meanwhile, the evaluation of the AI capacity based on the video service is difficult.
As shown in fig. 1A, an embodiment of the present disclosure provides an AI model building evaluation system, including: a controller 11 and an analog evaluation module 12, wherein:
a controller 11 configured to determine grouping information of the AI model; the following operations are performed for the AI model of each packet in the order of the packets: constructing the AI model of the current group to the test environment, and sending a first notification to the simulation evaluation module 12 (the first notification plays a role of notifying the simulation evaluation module 12 that the AI model of the current group has been constructed to the test environment);
a simulation evaluation module 12 configured to receive a first notification from the controller 11, and provide a video streaming service to the AI model, the content of the video streaming service being the video streaming content of a video file required for evaluating the AI model of the current packet; carrying out AI capacity evaluation on the AI model of the current group; and after the capability assessment is completed, releasing the video stream.
According to the AI model construction evaluation system, the AI models of each group are sequentially constructed to the test environment by grouping the AI models, streaming media services required by evaluation are provided for the AI models of the current group according to the needs, and after the evaluation is completed, video streams are released, and the video streams of all the AI models are not required to be kept all the time, so that the problem of massive resource waste of the video stream environment is solved, the calculation resources required by the AI model DevOps environment are greatly reduced, the model evaluation process is more controllable, the AI technology research and development efficiency is improved, and the cost is reduced.
The AI model according to the embodiments of the present disclosure refers to a program capable of sensing, reasoning, acting and adapting, while the machine learning model generally refers to a program capable of continuously improving performance with increasing data amount, the deep learning model is a subset of the machine learning model, and the deep learning model is a program that learns from a large amount of data using a multi-layer neural network. Therefore, AI models are larger than machine learning models than deep learning models in terms of coverage.
In the embodiment of the disclosure, when the AI capacity evaluation is performed on the AI model, one test case may be designed for one AI model, or a plurality of test cases may be designed. Generally, a test case is a case of design, in which the AI model must be able to function properly and achieve the expected execution results of the model design. By way of example, assuming an AI model for VIP user detection, test cases designed for it may include 3 of:
Test case 1) notify AI model pull video stream 1 (video stream 1 does not contain VIP user), detect AI model output result;
test case 2) notify AI model pull video stream 2 (video stream 2 contains 1 VIP user), detect AI model output result;
test case 3) informs the AI model to pull video stream 3 (video stream 3 contains 2 VIP users), and detects AI model output results.
When each test case is tested, the output result of the AI model is matched with the content of the video stream which is actually pulled, and the test case passes; the test case fails when the output of the AI model does not match the actual pulled video stream content.
When the test environment is constructed, the controller 11 pulls the model code of the AI model from the code warehouse, compiles the model code and deploys the model code to the test environment; when testing is carried out, the AI model pulls the video stream of the streaming media platform or the streaming media server, corresponding output is carried out according to the pulled video stream, the output result of the AI model is detected by the simulation evaluation module, and whether a test case passes or not is judged according to the output result of the AI model.
In the embodiment of the disclosure, the analog evaluation module 12 provides a video stream service to the AI model, which means that the analog evaluation module provides one or more video stream control interfaces (for example, the video stream control interfaces may be video stream addresses) to the AI model, and the AI model may pull a certain video stream by controlling different video stream control interfaces. When the capability assessment is completed, the analog assessment module 12 releases the release stream through the release video stream control interface.
In some example embodiments, the test environment may include one or more physical machines that may receive the AI model deployed by the controller 11 and run the AI model. When constructing the test environment, the controller 11 constructs an AI model of the current packet to the test environment; during testing, the analog evaluation module 12 provides video streaming service for the AI model, and the AI capacity of the current grouping AI model is evaluated; and after the capability assessment is completed, releasing the video stream. The simulation evaluation module 12 may be partially or completely disposed on a physical machine where the test environment is located, or may be disposed on a physical machine outside the test environment, which is not limited by the embodiment of the present disclosure.
In some exemplary embodiments, as shown in fig. 1B, the simulation evaluation module 12 includes a video stream simulation server 120 and an AI capability assessment module 121, wherein:
a video stream simulation server 120 configured to receive a first notification from the controller 11, provide a video stream service to the AI model, the content of the video stream service being the video stream content of a video file required to evaluate the AI model of the current packet, and send a second notification to the controller 11 to cause the controller 11 to send a third notification to the AI capability evaluation module 121; receiving a fifth notification from the controller 11, releasing the video stream;
An AI capability evaluation module 121 configured to receive a third notification of the controller 11, perform AI capability evaluation on the AI model of the current packet; after the capability assessment is completed, a fourth notification is sent to the controller 11, so that the controller 11 sends a fifth notification to the video stream simulation server 120;
the controller 11 is further configured to receive the second notification from the video stream simulation server 120 and send a third notification to the AI capability assessment module 121; the fourth notification of the AI capability assessment module 121 is received and a fifth notification is sent to the video stream simulation server 120.
In the embodiment of the present disclosure, after the video stream simulation server 120 is ready for the video stream service, the controller 11 is notified, and the AI-capability evaluation module 121 is notified by the controller 11 that the AI-capability evaluation of the AI model of the current packet may begin, and in other exemplary embodiments, the AI-capability evaluation module 121 may also be directly notified by the video stream simulation server 120 to begin the AI-capability evaluation of the AI model of the current packet, which is not limited in the embodiment of the present disclosure. After the AI capability assessment module 121 completes the AI capability assessment, the controller 11 is notified, and the controller 11 notifies the video stream simulation server 120 to release the video stream, or in other exemplary embodiments, the AI capability assessment module 121 may directly notify the video stream simulation server 120 to release the video stream, which is not limited in this embodiment of the disclosure. The controller 11 informs the AI capability assessment module 121 that AI capability assessment can be started on the AI model of the current packet and the controller 11 informs the video stream simulation server 120 to release the video stream, so that the controller 11 can accurately control the process of each AI model assessment, thereby enabling the model assessment process to become more controllable, improving AI technology development efficiency and reducing cost.
In some exemplary embodiments, as shown in fig. 1C, video stream simulation server 120 includes a video stream simulation module 1201 and a streaming media platform 1202, wherein:
a video stream simulation module 1201 configured to receive a first notification from the controller 11, push video stream content of a video file required to evaluate the AI model of the current packet to the streaming platform 1202, and send a second notification to the controller 11; receiving a fifth notification from the controller 11, stopping pushing the video streaming content, and sending a sixth notification to the streaming platform 1202;
a streaming media platform 1202 configured to receive video streaming content pushed by the video streaming simulation module 1201 and provide a video streaming service to the AI model; a sixth notification of the video stream simulation module 1201 is received, releasing the video stream.
In this embodiment, when the test environment is constructed, the controller 11 constructs the AI model of the current packet to the test environment; during testing, the video stream simulation module 1201 pushes the video stream content of the video file required for evaluating the current grouped AI model to the streaming media platform 1202, the streaming media platform 1202 provides video stream service for the AI model in the testing environment, the AI capability evaluation module 121 notifies the AI model to pull the video stream of the streaming media platform 1202, the output result of the AI model is detected, and whether the testing case passes or not is judged according to the output result of the AI model.
In some exemplary implementations, as shown in fig. 1C, an embodiment of the present disclosure provides an AI model building evaluation system, including: a controller 11, a video stream simulation module 1201, a streaming media platform 1202 and an AI capability assessment module 121, wherein:
a controller 11 configured to determine model grouping information; the following operations are performed for the AI model of each packet in the order of the packets: constructing an AI model of the current packet to a test environment, and sending a first notification to a video stream simulation module 1201; receiving the second notification of the video stream simulation module 1201, sending a third notification to the AI capability assessment module 121; receiving the fourth notification from the AI capability assessment module 121, sending a fifth notification to the video stream simulation module 1201;
a video stream simulation module 1201 configured to receive a first notification from the controller 11, push video stream content of a video file required to evaluate the AI model of the current packet to the streaming platform 1202, and send a second notification to the controller 11; receiving a fifth notification from the controller 11, stopping pushing the video streaming content, and sending a sixth notification to the streaming platform 1202;
the AI capability evaluation module 121 is configured to receive the third notification from the controller 11, perform AI capability evaluation on the AI model of the current packet using the video stream, and send a fourth notification to the controller 11 after the capability evaluation is completed;
A streaming media platform 1202 configured to receive video streaming content pushed by the video streaming simulation module 1201 and provide a video streaming service to the AI model; a sixth notification of the video stream simulation module 1201 is received, releasing the video stream.
In general, the DevOps pipeline of the AI model includes the following processes: firstly, a developer develops codes of an AI model, compiles and deploys the AI model codes to a test environment (the steps are collectively called a construction process and can be realized through automation), tests and result analysis (the steps are collectively called an evaluation process and can be realized through automation), repeated debugging and defect modification are carried out, and the test is carried out on the test platform. Because of the debugging and defect modification of the AI model, the construction, evaluation and defect modification processes are required to be circulated for a plurality of times, and the online process can be started after all defects are cleared. The construction script of each AI model is usually completed by a developer, the evaluation process is mainly dominated by testers, and the online process is mainly dominated by operation and maintenance personnel.
In the embodiment of the present disclosure, the actual physical locations of the video stream simulation module 1201 and the AI capability evaluation module 121 may be located at any location, as long as they are connected to the controller 11 and the streaming platform 1202 through a network. The video stream simulation module 1201 may be located on a physical machine where the controller 11 is located, on a physical machine where the streaming platform 1202 is located, or on a physical machine other than the controller 11 or the streaming platform 1202. For example, the AI capability assessment module 121 may be located on a physical machine where the test environment is located, and when the test environment is constructed, the code corresponding to the AI capability assessment module 121 and the model code corresponding to the AI model may be pulled from the code library, and then compiled and deployed to the test environment together, where the code corresponding to the AI capability assessment module 121 may include script code required for running an automated test, and preset test cases may be run for different AI models to perform a specific capability assessment. For example, each test case may specify a pulled video stream address, and when each test case is run, the AI-capability evaluation module 121 notifies the AI model to pull the video stream with the specified video stream address, and the AI model takes the video stream with the specified video stream address as input and performs corresponding output, and the AI-capability evaluation module determines whether the output result of the AI model matches with the content of the actually pulled video stream, and determines whether each test case passes the test. When the AI capacity evaluation module 121 and the AI model are both deployed on a physical machine where the test environment is located, physical resources required for constructing the whole AI model to construct an evaluation system can be reduced, the utilization rate of system resources is improved, the complexity of the system is reduced, and network communication data between the AI capacity evaluation module 121 and other modules is reduced. In other examples, AI capability assessment module 121 may also be located on a physical machine outside of the testing environment.
In the embodiment of the present disclosure, the streaming platform 1202 may be a streaming server, and when the streaming platform 1202 provides the video streaming service, the video streaming simulation module 1201 may specify a push address corresponding to each video stream, and after releasing the video stream, the push address stops providing the corresponding video streaming service.
In some exemplary embodiments, as shown in fig. 2A, the AI model build evaluation system further includes a code repository 105, wherein the code repository 105 is configured to store code information for AI models, grouping information for models, build scripts for models, policy codes for organizing AI model operations, and the like.
Illustratively, the code repository 105 may be a GIT code repository. GIT is an open-source distributed version control system for agilely and efficiently handling any item, small or large. The code repository 105 may also be other types of code repositories, such as SVN (Subversion), etc., to which embodiments of the present disclosure are not limited. Each AI model may correspond to an engineering at the code repository.
In fig. 2A, the AI capability assessment module is located on a physical machine where the test environment is located, where the physical machine where the test environment is located is deployed with an AI model, where the AI model needs to pull a video stream of the streaming media platform, that is, the AI model is equivalent to the video stream of the streaming media platform that the physical machine where the AI capability assessment module is located needs to pull, so there is a connection relationship between the AI capability assessment module and the streaming media platform in fig. 2A. In other exemplary embodiments, as shown in fig. 1C, the AI capability assessment module may also be located on a physical machine other than the physical machine in which the test environment is located, where there is no connection between the AI capability assessment module and the streaming platform in fig. 1C.
In some example embodiments, constructing the AI model of the current packet to the test environment includes:
pulling a model code of an AI model of the current group;
compiling a model code of an AI model of the current group;
the AI model of the current packet is deployed to the test environment.
In some exemplary embodiments, the code repository 105 may automatically trigger the controller 11 to perform a build evaluation process each time a developer submits model code to the code repository 105. When the controller 11 constructs the AI model of the current packet to the test environment, the controller 11 pulls the model code of the AI model of the current packet (may pull to the physical machine where the controller 11 is located), the controller 11 compiles the model code of the AI model of the current packet (on the physical machine where the controller 11 is located), and finally the controller 11 deploys the AI model of the current packet to the test environment.
In other exemplary embodiments, the model code after compiling the AI model of the current packet may be stored to a code repository, and when the AI model of the current packet is not updated, the AI model of the current packet is built into the test environment, including: pulling a model code compiled by an AI model of the current group; the AI model of the current packet is deployed to the test environment.
In other exemplary embodiments, the controller 11 initiates a unified build evaluation process at daily timings.
Assuming a hundreds of people develop a team to develop tens or even hundreds of AI models, each model corresponds to a development group, then the controller 11 starts a unified construction and evaluation process for tens or even hundreds of models at regular daily timings, so that the model evaluation process becomes more controllable, and AI technology development efficiency is improved.
In some exemplary embodiments, the controller 11 is further configured to sort and combine the construction scripts of the plurality of AI models and push the combined construction scripts to the task queue.
In this embodiment, the controller 11 needs to maintain a task queue, and the task queue may include: the grouping information of the AI model and the deployment sequence of the AI model grouping can realize the control of the following two dimensions through a task queue: 1. how to group AI models; 2. the deployment order after grouping (i.e., which AI model is deployed first and then which AI model is deployed).
In some examples, the task queue further comprises: sequential DevOps process for each AI model. The sequential DevOps process for each AI model includes the process from compile-deploy-test-result analysis.
In some exemplary embodiments, the build script pushed to the task queue may be grouped by a specialized grouping agent, at which point the grouping agent may store the group code in the code repository 105 and the controller 11 pulls the group code to retrieve the group information.
In other exemplary embodiments, the build scripts pushed to the task queue may be grouped by the controller 11.
In some exemplary embodiments, the controller 11 determines model grouping information, including:
acquiring construction scripts of a plurality of AI models to be constructed;
and grouping the AI models to be constructed according to the consumed resources of each AI model, wherein the consumed resources of each AI model are smaller than the resources of the test environment.
The AI model building assessment system of the disclosed embodiments, with the help of the Triton reasoning framework, can run multiple AI models simultaneously on a single GPU to more greatly improve utilization, and can be integrated with Kubernetes for orchestration, indexing, and automatic expansion. Alternatively, the single GPU may be a GPU on a physical machine where the test environment is located.
In the embodiment of the disclosure, the plurality of AI models to be constructed are grouped according to the resources consumed by each AI model, where the resources may be hardware resources (such as the size of an inference card resource, etc.), or may be software resources (such as the number of threads, etc.). The resource may be a hardware resource, and the plurality of AI models to be constructed are grouped according to the hardware resource consumed by each AI model, so that the reading of the hardware resource is easy and convenient, and the executable of model grouping is enhanced.
By way of example, assuming that 10 AI models are deployed on a system (which may alternatively be a test environment) with an inference card resource size of 16G, the overall occupation of the inference card resource size by the 10 AI models should be less than 16G. In some exemplary embodiments, the inference card may be a graphics card, i.e., a graphics card may be used for inference. In other exemplary embodiments, the inference card may be a non-graphics card, i.e., the inference card may or may not have display functionality. According to the embodiment of the disclosure, the AI models are grouped, so that the requirement on hardware resources of the test environment is reduced, if 100 AI models need to be tested, and if the AI models are not grouped, the test environment needs to be deployed with 100 AI models at the same time, and the requirement on the hardware resources of the test environment is higher; if the 100 AI models are divided into 10 groups of 10 AI models, only 10 AI models are deployed simultaneously for each group, and 10 groups are deployed in turn, so that the hardware resource requirement of the test environment is reduced.
In some exemplary embodiments, the controller 11 determines model grouping information, further comprising:
and grouping the AI models to be constructed according to the relation among different AI models, wherein the AI models with the dependency relation are grouped into a group.
Still taking 10 AI models per group as an example, among the 10 AI models, there may be a traffic dependency between some models or between all models, or there may be no traffic dependency between all models, but the AI models with traffic dependency should be divided into one group.
In some exemplary embodiments, the controller 11 determines the dependencies (i.e., whether there are business dependencies) between the plurality of AI models according to policy codes that organize the AI models to run.
The service dependency according to the embodiments of the present disclosure refers to a specific logic relationship between a plurality of different AI models set to achieve a certain target, where the specific logic relationship is on a certain AI application or AI service, for example: the execution sequencing of the multiple AI models, the output of AI model a as input to AI model B, etc. The controller 11 determines whether there is a traffic dependency between the AI models by organizing the policy codes operated by the AI models, which are typically stored in a code repository.
For example, assume that there is one AI application that needs to detect women 20 to 30 years old wearing red clothing. Then, the policy code that organizes the AI model execution may be such that: detecting all females in the video stream by the model 1, and if not, returning an empty result; detecting a person between 20 and 30 years old in women output by the model 1 through the model 2, and returning an empty result if the person is not detected; the person wearing red clothing in a woman aged 20 to 30 years who outputs from model 2 is detected by model 3, and if not, an empty result is returned. Then, according to the policy code operated by the organization AI model, the models 1, 2 and 3 have service dependency, and the models should be grouped (the group AI model may include other AI models besides the models 1, 2 and 3, or may not include other AI models, and is determined according to hardware resources possessed by the test environment), that is, the models 1, 2 and 3 together perform construction evaluation.
In some exemplary embodiments, the controller 11 determines model grouping information, further comprising:
and grouping the AI models to be constructed according to video streams required by AI capability assessment on the AI models, wherein the AI models requiring the same video streams are grouped into a group when the AI capability assessment is carried out.
At this time, each AI model may design only one test case, or may design a plurality of test cases, and video streams required for the plurality of test cases may be the same or different. For example, only video stream a is needed for each of model 1, model 2 and model 3 capability evaluations, then model 1, model 2 and model 3 may be grouped together. As another example, the video streams a and b are required for the capability assessment of the model 1, the video streams a and c are required for the capability assessment of the model 2, the video streams a and d are required for the capability assessment of the model 3, and the video streams e and f are required for the capability assessment of the model 4, so that the models 1, 2 and 3 can be still divided into a group.
In other exemplary embodiments, the controller 11 is configured to:
determining model grouping information; the following operations are performed for the AI model of each packet in the order of the packets:
constructing an AI model of the current group to a test environment; grouping the test cases of the AI model of the current grouping; the following operations are performed for each grouped test case according to the grouping sequence: sending a first notification to the video stream simulation module 1201; receiving the second notification of the video stream simulation module 1201, sending a third notification to the AI capability assessment module 121; the fourth notification of the AI capability assessment module 121 is received and a fifth notification is sent to the video stream simulation module 1201.
In the embodiment of the disclosure, when the AI capacity of each group of AI models is evaluated, the test cases of each group of AI models can be grouped, and the test efficiency is highest and can be measured under the condition of limited resources by grouping the test cases of each group of AI models.
In some exemplary embodiments, the test cases of the AI model of the current group are grouped according to the CPU resources consumed by the video stream required to run the test cases, while the resources consumed by the video stream required to run each group of test cases are less than the resources of the streaming platform 1202. Illustratively, the video stream required to run each set of test cases simultaneously consumes less CPU resources than the CPU resources of streaming media platform 1202. In other exemplary embodiments, the test cases of the current grouped AI model may also be grouped according to the GPU resources that are required to run each set of test cases simultaneously to consume less video streams than GPU resources of streaming media platform 1202, which is not limited by the disclosed embodiments.
In the embodiment of the disclosure, since the correspondence between the test cases and the video streams is preset, the controller 11 groups the test cases according to the size of the resources of the streaming media platform 1202 occupied by the video streams required for running the test cases, so as to reduce the resource requirements of the streaming media platform 1202.
Taking the current batch of AI models as an example, assuming that each AI model has 10 test cases (i.e., 10 AI capabilities to be tested), assuming that resources of the streaming platform 1202 (e.g., resources are CPU resources) can simultaneously provide 10 video streaming services, then 1 test case can be tested at a time per AI model, then a system can test 10 test cases simultaneously (assuming that the video streams of the 10 test cases are different) for a total of 10 times, i.e., the test cases are divided into 10 groups. In practice, the video streams required for each set of multiple test cases may or may not be identical, which is not a limitation of the embodiments of the present disclosure. The controller 11 notifies the video stream simulation module 1201 of the video stream contents of the video files required for the test cases of the current packet, pushes the video stream contents of the video files to the streaming media platform 1202 for providing the video stream through the video stream simulation module 1201, and after the streaming media platform 1202 is ready for the video stream required for the test cases of the current packet, the controller 11 notifies the AI capability assessment module 121 to start the test (the AI capability assessment module 121 notifies the AI model to pull the video stream of the streaming media platform 1202 and detects the output result of the AI model, in the embodiment of the present disclosure, the AI model may process the video stream of the streaming media platform 1202 frame by frame, or process the video stream of the multiple frames frame by frame, which is not limited in the embodiment of the present disclosure. After the test cases of the present set are tested, the AI ability assessment module 121 notifies the controller 11 that the test is complete. The controller 11 notifies the video stream simulation module 1201 to stop pushing video stream content, and the video stream simulation module 1201 notifies the streaming media platform 1202 to release the video stream. Then, the controller 11 notifies the video stream simulation module 1201 of the video stream contents of the video file required for the test case of the next packet until the test case test of all the packets is completed, in the above-described manner. Then, reconstructing the AI model of the next batch group to the testing environment, and testing the testing cases of the AI model of the next batch group according to the method.
In practical use, the test time required by each test case may be the same or different, so that each grouped test case may be divided into a plurality of subgroups, the test cases using the same video stream may be divided into a subgroup, when the test case of a certain test case or a certain subgroup is tested, the AI capability assessment module 121 may notify the controller 11 that the test case of the test case or the subgroup is tested, the controller 11 notifies the video stream simulation module 1201 to stop pushing the video stream content corresponding to the test case or the subgroup, and the video stream simulation module 1201 notifies the streaming platform 1202 to release the video stream corresponding to the test case or the subgroup, so that the resources of the streaming platform 1202 may be released as early as possible, and the parallel efficiency may be improved. By way of example, as shown in fig. 2B, assuming that the test cases of the AI model currently being grouped include 20 test cases and the 20 test cases require three video streams, wherein the 10 test cases A1 through a10 require video streams a, the 5 test cases B1 through B5 require video streams B, the 5 test cases C1 through C5 require video streams C, then, upon grouping the test cases of the AI model currently being grouped, as shown in fig. 2C, the 20 test cases may be grouped into 10 groups, wherein the first group of test cases includes A1, B1, the second group of test cases includes A2, B2, …, the fifth group of test cases includes A5, B5, the sixth group of test cases includes A6, C1, the seventh group of test cases includes A7, C2, …, the tenth group of test cases includes a10, C5, the fifth group of test cases may be released upon completion of the fifth group of test cases, and the sixth group of test cases may be provided for video evaluation. In the disclosed embodiment, the 10 test cases A1 to a10 may be grouped into a subset, the 5 test cases B1 to B5 into a subset, and the 5 test cases C1 to C5 into a subset. When the test case test of each group is completed, the streaming media platform 1202 releases the video stream corresponding to the test case of the group, so that the resources of the streaming media platform 1202 can be released as soon as possible, and the parallel efficiency is improved. Assuming that the streaming platform 1202 can provide video streams a, B, C simultaneously, when the test cases of the current-grouped AI model are grouped, as shown in fig. 2D, the 20 test cases can be divided into 10 groups, wherein the first group of test cases includes A1, B1, C1, the second group of test cases includes A2, B2, C2, …, the fifth group of test cases includes A5, B5, C5, the sixth group of test cases includes A6, the seventh group of test cases includes A7, …, and the tenth group of test cases includes a10. In the above test process, each AI model may run only one test case at the same time, and in the actual use process, each AI model may run multiple test cases at the same time, where the video streams corresponding to the multiple test cases may be the same or different, which is not limited by the embodiments of the present disclosure.
For example, the current batch of AI models still includes 10 (i.e. 10 AI capabilities to be tested), each AI model has 10 test cases, and video streams corresponding to 100 test cases are different, and assuming that the CPU resource performance of the streaming media platform 1202 is very strong, the video stream service corresponding to 100 test cases can be provided simultaneously, and the hardware resource of the test environment is very strong, each AI model can test 10 test cases at a time, the system can test 100 test cases simultaneously once, i.e. all test cases are divided into a group. At this time, the controller 11 notifies the video stream simulation module 1201 of the video stream contents of the video files required for the 100 test cases, and pushes the video stream contents of the video files to the streaming platform 1202 through the video stream simulation module 1201 for providing the video stream, and after the streaming platform 1202 is ready for the 100 test cases, the controller 11 notifies the AI capability assessment module 121 to start the test (the AI capability assessment module 121 notifies the AI model to pull the video stream of the streaming platform 1202 and detects the output result of the AI model). After the 100 test cases are tested, the AI capability assessment module 121 notifies the controller 11 that the test is completed, the controller 11 notifies the video stream simulation module 1201 to stop pushing the video stream content, and the video stream simulation module 1201 notifies the streaming media platform 1202 to release the video stream (here, when the test cases of a certain test case or a certain group using the same video stream are tested, the controller 11 may notify the video stream simulation module 1201 to release the video file corresponding to the test case or the test case of the group, and the video stream simulation module 1201 notifies the streaming media platform 1202 to release the video stream corresponding to the test case or the test case of the group). Then, the controller 11 reconstructs the AI model of the next batch group to the test environment according to the above method, and tests the test cases of the AI model of the next batch group according to the above method.
In some example embodiments, grouping the test cases of the current grouped AI models further comprises: grouping is performed according to the video stream required to run the test cases.
For example, assuming AI model 1 needs to detect a face, AI model 2 needs to detect a head, as shown in fig. 2E, if 5 test cases D1 to D5 of AI model 1 need video stream a, 5 test cases E1 to E5 need video stream b, 5 test cases F1 to F5 of AI model 2 need video stream a, 5 test cases G1 to G5 need video stream c, streaming media platform 1202 may provide two video streams simultaneously, and cannot provide three video streams simultaneously, then, as shown in fig. 2F, a first set of test cases may include D1, E1, F1 (need video streams a, b), a second set of test cases may include D2, E2, F2 (need video streams a, b), …, a fifth set of test cases may include D5, E5, F5 (need video streams a, b), a sixth set of test cases may include G1, G2, and G3 (need video stream c), and a seventh set of test cases may include G4 (need video stream c). In constructing the evaluation system using the AI model of the present disclosure, how to group the test cases can be specifically set according to needs, which is not limited by the embodiments of the present disclosure.
In some exemplary embodiments, when a certain test case fails the test, relevant developers can be reminded by sending corresponding alarm information, mails and the like, so that the developers can modify the model codes as early as possible, and the research and development efficiency is improved.
As shown in fig. 3, in the embodiment of the disclosure, in combination with the video stream environment requirement required by the AI capability target aimed by the DevOps and the case requirement required by the automatic verification, the video file simulation video stream technology is used to simulate the media files in the media file library into video streams, so as to complete the construction process and release the video streams, thereby solving the problem of overlarge CPU and memory resource expenditure required by the huge video streams caused by the simultaneous construction process of a plurality of AI services DevOps by some methods.
In some exemplary embodiments, before the test case test of the current packet is completed, video stream simulation module 1201 pushes the video stream content required for the test case of the current packet to streaming media platform 1202 for circular provision.
In this embodiment, by pushing the video streaming content to the streaming platform 1202 for cyclic provision, the AI capability assessment module 121 can determine whether the AI model can give the reasoning result within a relative playing duration, i.e. a relative video start time for each test case, so as to reduce the assessment difficulty. For example, assuming that the video stream playing duration of a certain test case is 15s, if the AI model gives an inference result within 15s, the inference is considered to be successful, then, since the video stream content is being circularly pushed, if the AI model does not give an inference result within 15s from a certain moment, the AI capability evaluation module 121 may determine that the inference fails. In other exemplary embodiments, the AI capability assessment module 121 may accurately determine the start time of the AI model pulling the video stream and the inference time when the inference result is obtained through the timestamp of the video frame, so as to obtain an accurate inference duration.
In the embodiment of the present disclosure, the video stream simulation module 1201 pushes the video stream content required by the current packet of test cases to the streaming media platform 1202 for cyclic provision, including any one of the following two cases: 1) The video stream simulation module 1201 pushes the video stream content loop required for the current grouped test cases to the streaming media platform 1202 such that the streaming media platform 1202 loops the video stream required for providing the current grouped test cases; 2) The video stream simulation module 1201 pushes the video stream content required for the current packet of test cases to the streaming media platform 1202 (push only once), but the streaming media platform 1202 loops to provide the video stream required for the current packet of test cases, i.e. the streaming media platform 1202 has a loop mode, at which time the video stream simulation module 1201 only needs to push the video stream content once and the streaming media platform 1202 loops to provide the video stream.
In some exemplary embodiments, as shown in fig. 2A, the AI model build evaluation system further includes a media file library 106, wherein the media file library 106 is configured to store video files required to evaluate the AI model for each group.
In the disclosed embodiment, the video files corresponding to the test cases are stored in the media file library 106. From a playback perspective, the video stream simulation module 1201 plays a video file from the media file library 106 and then pushes the video stream content of the played video file onto the streaming media platform 1202. The video files in the disclosed embodiments are videos pre-stored in the media file library 106 for evaluating the function or performance of the AI model for automated testing. By setting the media file library 106 (located on a server outside the streaming media platform 1202), the streaming media platform 1202 can be more focused on realizing the push-pull function, and the storage of video files required by the streaming media platform 1202 is completed by the media file library 106, so that richer video file contents can be provided, and further, the evaluation of the AI model can be more accurate.
In some exemplary embodiments, when the format of the video file is a non-streaming media file format, streaming media platform 1202 is notified by video stream simulation module 1201 to convert the format of the video file to a streaming media file format.
In embodiments of the present disclosure, the format of the video file may be MPEG-1 or any other video file format, and the streaming platform 1202 may convert the video file to MPEG-4 or any other streaming file format, which embodiments of the present disclosure are not limited.
In some exemplary embodiments, as shown in fig. 4 and 5, the AI model build evaluation system simulates a build evaluation process of a video stream environment required in an AI intelligent service DevOps process as needed, comprising the steps of:
1) The research personnel submits codes to a GIT code warehouse, and the GIT code warehouse automatically triggers the DevOps to construct an evaluation process;
2) The DevOps controller sorts the construction process, combines the construction process and pushes the construction process to a task queue;
3) The DevOps controller obtains a construction task, pulls group codes from the GIT code bin library, obtains model grouping information, pulls model codes of the current batch of groups, and compiles and deploys the model codes to a test environment;
4) The DevOps controller reads all the AI capability directories (here, AI capability directory refers to AI function list to be tested, i.e. test case) in the current batch packet, and sorts the video stream address list required by the test;
5) The DevOps controller sends the required video stream address list to a video stream simulation module for processing;
6) The video stream simulation module obtains video stream information in the case library (determining whether the video file format is a streaming media format or not, if not, notifying the streaming media platform to convert the video file format into the streaming media file format), and pulling the corresponding video file;
7) The video stream simulation module pushes the video stream content of the video file to the streaming media platform for circulation provision according to the one-to-one correspondence between the video file and the video stream address (the video stream simulation module pushes the video stream content of the video file to the streaming media platform in a circulation manner, or the streaming media platform circularly provides the video stream content of the video file), and performs one-to-one verification;
8) After the DevOps controller obtains a video stream environment construction completion signal, starting an AI ability evaluation module to perform model evaluation, and after the model evaluation is completed, sending a release instruction to a video stream simulation module;
9) And the video stream simulation module stops pushing the video stream content which is not used currently according to the use condition of the video stream, and sends a release instruction to the streaming media platform to finish the release of the video stream.
In fig. 5, when an error occurs in the execution process of the aforementioned steps by the DevOps controller and the video stream simulation module (for example, the DevOps controller does not pull the group codes from the GIT code bin library, the DevOps controller does not read the configuration related to all the AI capability directories in the current batch of packets, the video stream simulation module does not acquire the video stream information, etc.), relevant developers can be reminded by sending corresponding alarm information, mails, etc., so that the developers modify the construction script or configuration as early as possible, and the development efficiency is improved.
According to the AI model construction evaluation system provided by the embodiment of the disclosure, the required video stream is constructed temporarily according to the requirements of an evaluation object, and immediately released after model evaluation is completed, resources required by a DevOps environment are reduced greatly, and meanwhile, the model evaluation process is controlled more; the model evaluation process aims at video starting time of the case, so that the evaluation difficulty is reduced; the construction, evaluation and online processes can be performed in groups, so that the AI technology research and development efficiency is improved, and the cost is reduced.
As shown in fig. 6, the embodiment of the present disclosure further provides an AI model building evaluation method, including:
step 601, a controller determines grouping information of an AI model;
Step 602, according to the grouping sequence, for each AI model of the group, performing the following operations:
the controller builds an AI model of the current group to a test environment, and sends a first notification to the simulation evaluation module; the simulation evaluation module receives a first notification of the controller, and provides video streaming service for the AI model, wherein the content of the video streaming service is the video streaming content of a video file required by the AI model for evaluating the current packet; carrying out AI capacity evaluation on the AI model of the current group; and after the capability assessment is completed, releasing the video stream.
According to the AI model construction evaluation method, the AI models of each group are sequentially constructed to the test environment by grouping the AI models, streaming media services required by evaluation are provided for the AI models of the current group according to the needs, and after the model evaluation is completed, video streams are released, so that the video streams of all the AI models are not required to be kept all the time, the problem of massive resource waste of the video stream environment is solved, the calculation resources required by the AI model DevOps environment are greatly reduced, the model evaluation process is more controllable, the AI technology research and development efficiency is improved, and the cost is reduced.
In some example embodiments, the controller determines grouping information for the AI model, including:
The method comprises the steps that a controller obtains construction scripts of a plurality of AI models to be constructed;
the controller groups the AI models to be constructed according to the consumed resources of each AI model, and the consumed hardware resources of each group of AI models are smaller than the hardware resources of the test environment.
By way of example, assuming 10 AI models are deployed on a system with an inference card resource size of 16G, the overall occupation of the inference card resource size by the 10 AI models should be less than 16G.
In some example embodiments, the controller determines grouping information for the AI model, including:
the method comprises the steps that a controller obtains construction scripts of a plurality of AI models to be constructed;
the controller groups the AI models to be constructed according to the consumed resources of each AI model and the relation among different AI models, the hardware resources consumed by each group of AI models are smaller than the hardware resources possessed by the test environment, and the AI models with the dependency relation are divided into a group.
Taking the above 10 AI models as an example, among the 10 AI models, there may be a business dependency between some models or all models, and the AI models with the dependency relationship are grouped into a group.
In some example embodiments, the controller determines dependencies between the plurality of AI models based on policy codes that organize AI model runs.
In other exemplary embodiments, the AI model building evaluation method further includes:
the controller obtains grouping information of test cases of an AI model of a current grouping;
the AI capability assessment of the AI model of the current packet includes: the AI capability assessment is performed on each grouped test case in the order of the grouping of test cases.
In some exemplary embodiments, the controller groups the test cases of the AI model currently grouped according to the resources consumed by the video stream required to simulate the test cases, while the resources consumed by the video stream required to simulate each group of test cases are less than the resources of the streaming platform.
In some example embodiments, before the test case test of the current packet is completed, the video stream simulation module pushes video stream content of the video file required for the test case of the current packet to the streaming media platform to cyclically provide the video stream required for the test case of the current packet (the video stream simulation module cyclically pushes video stream content of the video file required for evaluating the AI model of the current packet to the streaming media platform, or the streaming media platform cyclically provides the video stream content of the video file).
In some exemplary embodiments, after the test case or the test case of a certain group is tested, the video stream simulation module notifies the streaming media platform to release the video stream corresponding to the test case or the test case of the group, so as to release the resources of the streaming media platform as soon as possible.
In some example embodiments, the controller builds an AI model of the current packet to the test environment, comprising:
pulling a model code of an AI model of the current group;
compiling a model code of an AI model of the current group;
the AI model of the current packet is deployed to the test environment.
In some exemplary embodiments, the grouping information of the AI model, the code information of the AI model, the build script of the AI model, and the policy code that organizes the operation of the AI model are all stored in a code repository.
The code repository may be, for example, a GIT code repository. GIT is an open-source distributed version control system for agilely and efficiently handling any item, small or large. The GIT code repository may automatically trigger the DevOps build evaluation process each time a developer submits a code to the GIT code repository.
In some exemplary embodiments, the video files required to evaluate the AI model for each packet are stored in a media file library.
In some exemplary embodiments, when the format of the video file is a non-streaming media file format, the streaming media platform is notified by the video stream simulation module to convert the format of the video file to a streaming media file format.
As shown in fig. 7, the embodiment of the present disclosure further provides an AI model construction evaluation method, applied to a controller, including:
step 701, determining model grouping information;
step 702, according to the grouping sequence, for each AI model of the group, performing the following operations: the AI model of the current packet is constructed to a test environment, a video stream simulation module is notified to push video stream contents of video files required by the AI model of the current packet to a streaming media platform, an AI capability assessment module is notified to carry out AI capability assessment on the AI model of the current packet by using the video stream of the streaming media platform, and after the capability assessment is completed, the video stream simulation module and the streaming media platform are notified to release the video stream.
According to the AI model construction evaluation method, the AI models of each group are sequentially constructed to the test environment by grouping the AI models, streaming media services required by evaluation are provided for the AI models of the current group according to the needs, and after the model evaluation is completed, video streams are released, so that the video streams of all the AI models are not required to be kept all the time, the problem of massive resource waste of the video stream environment is solved, the calculation resources required by the AI model DevOps environment are greatly reduced, the model evaluation process is more controllable, the AI technology research and development efficiency is improved, and the cost is reduced.
In some exemplary embodiments, determining model grouping information includes:
acquiring construction scripts of a plurality of AI models to be constructed;
and grouping the AI models to be constructed according to the consumed resources of each AI model, wherein the consumed hardware resources of each AI model are smaller than the hardware resources of the test environment.
In some exemplary embodiments, determining model grouping information includes:
acquiring construction scripts of a plurality of AI models to be constructed;
and grouping the AI models to be constructed according to the consumed resources of each AI model and the relation among different AI models, wherein the hardware resources consumed by each AI model are smaller than the hardware resources possessed by the test environment, and the AI models with the dependency relation are grouped into a group.
In some example embodiments, the dependencies between the plurality of AI models are determined according to policy codes that organize AI model runs.
In some exemplary embodiments, the notifying video stream simulation module in step 702 pushes video stream contents of a video file required for evaluating the AI model of the current packet to the streaming media platform, notifies the AI capability evaluation module to perform AI capability evaluation on the AI model of the current packet using the video stream of the streaming media platform, and after the capability evaluation is completed, notifies the video stream simulation module and the streaming media platform to release the video stream, including:
Grouping the test cases of the AI model of the current grouping;
the following operations are performed for each grouped test case according to the grouping sequence: notifying a video stream simulation module to push video stream contents of video files required by a current grouped test case to a streaming media platform so as to form a video stream which can be pulled by the AI model on the streaming media platform; and notifying an AI capability assessment module to test each group of test cases by using the video stream of the streaming media platform, notifying a video stream simulation module to stop pushing the video stream content after a certain test case test is completed or after the test case test of the current group is completed, and notifying the streaming media platform to release the video stream by the video stream simulation module.
In some exemplary embodiments, the test cases of the AI model of the current group are grouped according to the resources consumed by the video stream required to simulate the test cases, while the resources consumed by the video stream required to simulate each group of test cases are less than the resources of the streaming platform.
In some example embodiments, constructing the AI model of the current packet to the test environment includes:
pulling a model code of an AI model of the current group;
Compiling a model code of an AI model of the current group;
the AI model of the current packet is deployed to the test environment.
In some exemplary embodiments, notifying the video stream simulation module and the streaming media platform to release the video stream includes:
and notifying the video stream simulation module to stop pushing the video stream content, and sending a video stream release instruction to the streaming media platform by the video stream simulation module.
In the embodiment of the disclosure, the video source is a video stream simulation module, and the video stream simulation module pushes streams to the streaming media platform; the AI model is pulled from the streaming media platform, the streaming media platform is in the middle and is equivalent to a forwarding function, and the AI model is equivalent to the connection between the streaming media platform and the video stream simulation module.
In some exemplary embodiments, the video files required to evaluate the AI model for each packet are stored in a media file library.
The embodiment of the disclosure also provides a controller, which comprises a memory; and a processor coupled to the memory, the processor configured to perform the steps of the AI model build evaluation method as described in any one of the embodiments of the disclosure based on instructions stored in the memory.
In one example, the controller may include: the system comprises a first processor, a first memory and a first bus system, wherein the first processor and the first memory are connected through the first bus system, the first memory is used for storing instructions, and the first processor is used for executing the instructions stored in the first memory. Specifically, the first processor determines model grouping information; the following operations are performed for the AI model of each packet in the order of the packets: the AI model of the current packet is constructed to a test environment, a video stream simulation module is notified to push video stream contents of video files required by the AI model of the current packet to a streaming media platform, an AI capability assessment module is notified to carry out AI capability assessment on the AI model of the current packet by using the video stream of the streaming media platform, and after the capability assessment is completed, the video stream simulation module and the streaming media platform are notified to release the video stream.
It should be appreciated that the first processor may be a central processing unit (Central Processing Unit, CPU), but the first processor may also be other general purpose processors, digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The first memory may include read only memory and random access memory and provide instructions and data to the first processor. A portion of the first memory may also include a nonvolatile random access memory. For example, the first memory may also store information of the device type.
The first bus system may include a power bus, a control bus, a status signal bus, and the like in addition to the data bus.
In an implementation, the processing performed by the processing device may be performed by integrated logic circuits of hardware in the first processor or by instructions in the form of software. That is, the method steps of the embodiments of the present disclosure may be embodied as hardware processor execution or as a combination of hardware and software modules in a processor. The software modules may be located in random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, and other storage media. The storage medium is located in a first memory, and the first processor reads the information in the first memory, and the steps of the method are completed by combining the hardware of the first processor. To avoid repetition, a detailed description is not provided herein.
The embodiments of the present disclosure also provide a storage medium having stored thereon a computer program which, when executed by a processor, implements the AI model building evaluation method according to any of the embodiments of the present disclosure. The AI model construction evaluation method can determine model grouping information; the following operations are performed for the AI model of each packet in the order of the packets: the method comprises the steps of constructing an AI model of a current packet to a test environment, informing a video stream simulation module to push video stream contents of video files required by the AI model of the current packet to a streaming media platform, informing an AI capability assessment module to assess the AI capability of the AI model of the current packet by using the video stream of the streaming media platform, informing the video stream simulation module and the streaming media platform to release the video stream after the capability assessment is completed, and not only solving the problem of wasting a large amount of resources of the video stream environment because the video stream of all the AI models is not required to be always maintained, but also greatly reducing calculation resources required by the AI model DevOps environment, enabling the model assessment process to be more controllable, improving the AI technology research and development efficiency and reducing the cost. The method for driving the AI model construction evaluation by executing the executable instruction is substantially the same as the AI model construction evaluation method provided in the above embodiment of the present disclosure, and will not be described herein.
As shown in fig. 8, an embodiment of the present disclosure further provides a video stream simulation method, including:
step 801, a video stream simulation module receives a video file address list required for evaluating an AI model of a current packet;
step 802, a video stream simulation module pulls a corresponding video file from a video file address list, pushes video stream contents of the video file to a streaming media platform, and informs a controller that streaming media environment construction is completed;
803, the video stream simulation module receives a release instruction of the controller and stops pushing the video stream content;
step 804, the video stream simulation module sends a video stream release instruction to the streaming media platform.
In the evaluating process of the Devops model, if a large amount of resources are input to simulate a plurality of video streams at the same time, starting and stopping are carried out according to the requirement of a request, the video stream simulation method has small research and development difficulty, but the required resources are more, the Devops environment requirement is high, and a large amount of used CPU and memory are wasted in calculating resources. According to the video stream simulation method, the AI models are grouped, streaming media services required for evaluation are provided for the current grouped AI models according to needs, and after model evaluation is completed, video streams are released without always keeping video streams of all the AI models, so that the problem of waste of a large amount of resources in a video stream environment is solved, the calculation resources required by an AI model DevOps environment are greatly reduced, the model evaluation process is more controllable, the AI technology research and development efficiency is improved, and the cost is reduced.
In the actual application process, the video source can be a camera, and the camera pushes streams to the streaming media platform; the AI reasoning platform pulls from the streaming media platform, the streaming media platform is in the middle and is equivalent to a forwarding function, and the AI reasoning platform is equivalent to the connection between the streaming media platform and the camera. In the system construction and evaluation process, the video source is a video stream simulation module, and the video stream simulation module pushes streams to a streaming media platform; the AI reasoning platform pulls the stream from the streaming media platform, the streaming media platform is in the middle, and the AI reasoning platform establishes connection with the video stream simulation module through the streaming media platform.
In some exemplary embodiments, the video files required to evaluate the AI model for each packet are stored in a media file library from which the video stream simulation module pulls the corresponding video file.
In some exemplary embodiments, the video stream simulation module pushes video stream content of the video file to the streaming media platform for cyclical provision.
In the embodiment of the disclosure, the video stream simulation module pushes the video stream content of the video file to the streaming media platform for cyclic provision, including any one of the following two cases: 1) The video stream simulation module circularly pushes video stream contents of the video file to the streaming media platform so that the streaming media platform circularly provides video streams of the video file; 2) The video stream simulation module pushes the video stream content of the video file to the streaming media platform (push only once), but the streaming media platform cyclically provides the video stream of the video file.
The embodiment of the disclosure also provides a video stream simulation module, which comprises a memory; and a processor coupled to the memory, the processor configured to perform the steps of the video stream simulation method according to any of the embodiments of the present disclosure based on instructions stored in the memory.
In one example, the video stream simulation module may include: the system comprises a second processor, a second memory and a second bus system, wherein the second processor is connected with the second memory through the second bus system, the second memory is used for storing instructions, and the second processor is used for executing the instructions stored by the second memory. Specifically, the second processor receives a video file address list required by the AI model of the current packet, pulls a corresponding video file from the video file address list, pushes video stream content of the video file to a streaming media platform, and informs a controller that streaming media environment construction is completed; and receiving a release instruction of the controller, stopping pushing the video stream content, and sending the video stream release instruction to a streaming media platform.
It should be appreciated that the second processor may be a central processing unit (Central Processing Unit, CPU), but the second processor may also be other general purpose processors, digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The second memory may include read only memory and random access memory and provide instructions and data to the second processor. A portion of the second memory may also include a nonvolatile random access memory. For example, the second memory may also store information of the device type.
The second bus system may comprise, in addition to a data bus, a power bus, a control bus, a status signal bus, etc.
In an implementation, the processing performed by the processing device may be performed by integrated logic circuits of hardware in the second processor or by instructions in the form of software. That is, the method steps of the embodiments of the present disclosure may be embodied as hardware processor execution or as a combination of hardware and software modules in a processor. The software modules may be located in random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, and other storage media. The storage medium is located in a second memory, and the second processor reads information in the second memory, and the steps of the method are completed by combining hardware of the second processor. To avoid repetition, a detailed description is not provided herein.
The embodiments of the present disclosure also provide a storage medium having stored thereon a computer program which, when executed by a processor, implements a video stream simulation method according to any of the embodiments of the present disclosure. The video stream simulation method can receive a video file address list required by the AI model of the current packet, pull a corresponding video file from the video file address list, push video stream contents of the video file to a streaming media platform and inform a controller of the completion of streaming media environment construction; and receiving a release instruction of the controller, stopping pushing the video stream content, and sending the video stream release instruction to the streaming media platform, wherein video streams of all the AI models are not required to be kept all the time, so that the problem of massive resource waste of the video stream environment is solved, the computing resources required by the AI model DevOps environment are greatly reduced, the model evaluation process is more controllable, the AI technology research and development efficiency is improved, and the cost is reduced. The method for driving video stream simulation by executing the executable instruction is substantially the same as the video stream simulation method provided in the above embodiment of the present disclosure, and will not be described herein.
Those of ordinary skill in the art will appreciate that all or some of the steps, systems, functional modules/units in the apparatus, and methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between the functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed cooperatively by several physical components. Some or all of the components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as known to those skilled in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Furthermore, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
While the embodiments disclosed in this disclosure are described above, the embodiments are only used for facilitating understanding of the disclosure, and are not intended to limit the present invention. Any person skilled in the art will recognize that any modifications and variations can be made in the form and detail of the present disclosure without departing from the spirit and scope of the disclosure, which is defined by the appended claims.
Claims (24)
- An AI model building evaluation system, comprising: the controller and simulation evaluation module, wherein:the controller is configured to determine grouping information of an AI model; the following operations are performed for the AI model of each packet in the order of the packets: constructing an AI model of a current group to a test environment, and sending a first notification to a simulation evaluation module;the simulation evaluation module is configured to receive a first notification of the controller, provide video streaming service for the AI model, wherein the content of the video streaming service is the video streaming content of a video file required by the AI model of the current packet; carrying out AI capacity evaluation on the AI model of the current group; and after the capability assessment is completed, releasing the video stream.
- The AI model building evaluation system of claim 1, wherein the analog evaluation module comprises a video stream analog server and an AI capability evaluation module;the video stream simulation server is configured to receive a first notification of the controller, provide a video stream service for the AI model, wherein the content of the video stream service is the video stream content of a video file required by the AI model for evaluating the current packet, and send a second notification to the controller so that the controller sends a third notification to the AI capability evaluation module; receiving a fifth notification of the controller, and releasing the video stream;the AI capability assessment module is configured to receive a third notification from the controller and perform AI capability assessment on the AI model of the current packet; after the capability assessment is completed, sending a fourth notification to the controller so that the controller sends a fifth notification to the video stream simulation server;the controller is further configured to receive a second notification from the video stream simulation server, and send a third notification to the AI capability assessment module; and receiving a fourth notification of the AI capability assessment module, and sending a fifth notification to the video stream simulation server.
- The AI model building evaluation system of claim 2, wherein the video stream simulation server comprises a video stream simulation module and a streaming platform;the video stream simulation module is configured to receive a first notification from the controller, push video stream contents of a video file required for evaluating an AI model of a current packet to the streaming media platform, and send a second notification to the controller; receiving a fifth notification of the controller, stopping pushing video stream content, and sending a sixth notification to the streaming media platform;the streaming media platform is configured to receive the video stream content pushed by the video stream simulation module and provide video stream service for the AI model; and receiving a sixth notification of the video stream simulation module, and releasing the video stream.
- The AI model build evaluation system of claim 3, wherein the video stream simulation module cyclically pushes video stream content of a video file required to evaluate a current packet of AI models to the streaming media platform or the streaming media platform cyclically provides video stream content of the video file.
- The AI model building evaluation system of claim 3, wherein video files required to evaluate the AI model for each group are stored in a media file library;Pushing video streaming content of a video file required for evaluating an AI model of a current packet to the streaming platform comprises: and pulling the video files required by the AI model of the current packet from the media file library, and pushing the video stream content of the video files to the streaming media platform.
- The AI model building evaluation system of claim 1, wherein the determining grouping information of AI models comprises:acquiring construction scripts of a plurality of AI models to be constructed;and grouping the AI models to be constructed according to the consumed resources of each AI model, wherein the consumed resources of each AI model are smaller than the resources of the test environment.
- The AI model building evaluation system of claim 1, wherein the determining grouping information of AI models comprises:acquiring construction scripts of a plurality of AI models to be constructed;and grouping the AI models to be constructed according to the consumed resources of each AI model and the relation among different AI models, wherein the consumed resources of each group of AI models are smaller than the resources of the test environment, and the AI models with the dependency relation are grouped into a group.
- The AI model build evaluation system of claim 7, wherein the dependencies between the plurality of AI models are determined from policy codes that organize the AI model runs.
- The AI model build evaluation system of claim 1, wherein prior to sending the first notification to the analog evaluation module, the controller is further configured to group test cases of the currently grouped AI model; and executing the operation of sending the first notification to the simulation evaluation module for each grouped test case according to the grouping sequence of the test cases.
- The AI model build evaluation system of claim 9, wherein the simulated evaluation module comprises a streaming platform configured to provide video streaming services to an AI model;and grouping the test cases of the current grouped AI model according to the resources consumed by the video stream required for running the test cases, wherein the resources consumed by the video stream required for running each group of test cases are smaller than the resources of the streaming media platform.
- The AI model building evaluation system of claim 9, wherein after the capability evaluation is completed, releasing the video stream, in particular: and after the test of the test case corresponding to one video stream is completed, releasing the video stream.
- The AI model build evaluation system of claim 1, wherein the building of the AI model of the current packet to the test environment comprises:pulling a model code of the AI model of the current group;compiling a model code of the AI model of the current group;and deploying the AI model of the current packet to a test environment.
- The AI model build evaluation system of claim 1, wherein the grouping information of the AI models, the code information of the AI models, the build script of the AI models, and the policy code that organizes the AI model operations are all stored in a code repository.
- The AI model build evaluation system of claim 1, wherein the controller is further configured to: the method comprises the steps of sorting and merging construction scripts of a plurality of AI models and pushing the construction scripts to a task queue, wherein the task queue comprises: grouping information of the AI model and deployment sequence of the AI model group.
- The AI model build evaluation system of claim 1, wherein the controller is further configured to: when the packet information of the AI model is determined to fail, sending alarm information or alarm mail;the analog assessment module is further configured to: and when the video streaming service provided to the AI model fails, sending alarm information or alarm mail.
- An AI model construction evaluation method, comprising:the controller determines model grouping information;the following operations are performed for the AI model of each packet in the order of the packets:the controller builds an AI model of the current group to a test environment and sends a first notification to the simulation evaluation module;the simulation evaluation module receives a first notification of the controller and provides video streaming service for the AI model, wherein the content of the video streaming service is the video streaming content of a video file required by the AI model for evaluating the current packet; carrying out AI capacity evaluation on the AI model of the current group; and after the capability assessment is completed, releasing the video stream.
- An AI model construction evaluation method, comprising:the controller determines model grouping information;the controller performs the following operations on the AI model of each packet in the packet order: the AI model of the current packet is constructed to a test environment, a video stream simulation module is notified to push video stream contents of video files required by the AI model of the current packet to a streaming media platform, an AI capability assessment module is notified to carry out AI capability assessment on the AI model of the current packet by using the video stream of the streaming media platform, and after the capability assessment is completed, the video stream simulation module and the streaming media platform are notified to release the video stream.
- The AI model building and evaluating method of claim 17, wherein the notifying the video stream simulation module pushes video stream contents of a video file required for evaluating the AI model of the current packet to a streaming media platform, notifying the AI capability evaluation module to evaluate the AI capability of the AI model of the current packet using the video stream of the streaming media platform, and notifying the video stream simulation module and the streaming media platform to release the video stream after the capability evaluation is completed comprises:grouping the test cases of the AI model of the current grouping;the following operations are performed for each grouped test case according to the grouping sequence: notifying a video stream simulation module to push video stream contents of video files required by a current grouped test case to a streaming media platform so as to form a video stream which can be pulled by the AI model on the streaming media platform; and notifying an AI capability assessment module to test each grouped test case by using the video stream of the streaming media platform, notifying a video stream simulation module to stop pushing the video stream content after the test of the test cases to be tested is completed, and notifying the streaming media platform to release the video stream by the video stream simulation module.
- A controller comprising a memory; and a processor coupled to the memory, the processor configured to perform the steps of the AI model building evaluation method of any of claims 17-18 based on instructions stored in the memory.
- A storage medium having stored thereon a computer program which, when executed by a processor, implements the AI model building evaluation method of any one of claims 17 to 18.
- A video stream simulation method, comprising:the video stream simulation module receives a video file address list required for evaluating the AI model of the current packet;the video stream simulation module pulls the corresponding video file from the video file address list, pushes the video stream content of the video file to the streaming media platform, and informs the controller that the streaming media environment is constructed;the video stream simulation module receives a release instruction of the controller and stops pushing the video stream content;and the video stream simulation module sends a video stream release instruction to the streaming media platform.
- The video stream simulation method of claim 21, wherein the video stream simulation module cyclically pushes the video stream content of the video file to the streaming media platform or the streaming media platform cyclically provides the video stream content of the video file.
- A video stream simulation module comprising a memory; and a processor coupled to the memory, the processor configured to perform the steps of the video stream simulation method of any of claims 21 to 22 based on instructions stored in the memory.
- A storage medium having stored thereon a computer program which, when executed by a processor, implements the video stream simulation method of any of claims 21 to 22.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2022/102783 WO2024000387A1 (en) | 2022-06-30 | 2022-06-30 | Ai model establishment evaluation system, video stream simulation module and method, and controller |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117642728A true CN117642728A (en) | 2024-03-01 |
Family
ID=89383532
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202280002046.XA Pending CN117642728A (en) | 2022-06-30 | 2022-06-30 | AI model construction evaluation system, video stream simulation module and method, and controller |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN117642728A (en) |
| WO (1) | WO2024000387A1 (en) |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180357543A1 (en) * | 2016-01-27 | 2018-12-13 | Bonsai AI, Inc. | Artificial intelligence system configured to measure performance of artificial intelligence over time |
| US11710034B2 (en) * | 2019-02-27 | 2023-07-25 | Intel Corporation | Misuse index for explainable artificial intelligence in computing environments |
| CN112304952B (en) * | 2019-07-23 | 2024-04-02 | 纬创资通股份有限公司 | Image recognition device, image recognition method and computer program product thereof |
| CN113923441B (en) * | 2020-07-08 | 2023-10-20 | 钉钉控股（开曼）有限公司 | Video quality assessment method and device and electronic equipment |
| CN114004328A (en) * | 2020-07-27 | 2022-02-01 | 华为技术有限公司 | AI model updating method, device, computing equipment and storage medium |
| CN112416753A (en) * | 2020-11-02 | 2021-02-26 | 中关村科学城城市大脑股份有限公司 | Method, system and equipment for standardized management of urban brain application scene data |
| CN114610271A (en) * | 2020-11-25 | 2022-06-10 | 共达地创新技术(深圳)有限公司 | AI model customization method, electronic device and computer-readable storage medium |
| CN114492764A (en) * | 2022-02-21 | 2022-05-13 | 深圳市商汤科技有限公司 | Artificial intelligence model testing method and device, electronic equipment and storage medium |
-
2022
- 2022-06-30 CN CN202280002046.XA patent/CN117642728A/en active Pending
- 2022-06-30 WO PCT/CN2022/102783 patent/WO2024000387A1/en unknown
Also Published As
| Publication number | Publication date |
|---|---|
| WO2024000387A1 (en) | 2024-01-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110795219A (en) | Resource scheduling method and system suitable for multiple computing frameworks | |
| CN111813686B (en) | Game testing method and device, testing terminal and storage medium | |
| CN113535399B (en) | NFV resource scheduling method, device and system | |
| CN113282402B (en) | Test task scheduling method oriented to complex resource constraint | |
| CN114217966A (en) | Deep learning model dynamic batch processing scheduling method and system based on resource adjustment | |
| US20210294730A1 (en) | Managing resources used during a development pipeline | |
| CN117642728A (en) | AI model construction evaluation system, video stream simulation module and method, and controller | |
| CN116194934A (en) | Modular model interaction system and method | |
| CN110958138B (en) | Container expansion method and device | |
| CN110752964B (en) | Network equipment testing method and device | |
| US8930908B2 (en) | Aspect and system landscape capability-driven automatic testing of software applications | |
| CN109284222A (en) | Project testing method, device and equipment in software unit, data processing system | |
| US20220206773A1 (en) | Systems and methods for building and deploying machine learning applications | |
| US20230065994A1 (en) | Offload server, offload control method, and offload program | |
| CN113032543A (en) | Dialog diagnosis method, device, equipment and medium based on stack structure | |
| CN112765015A (en) | Test task distribution method based on terminal equipment, scheduling terminal and test system | |
| Schumacher et al. | Cause and effect of nondeterministic behavior in sequential and parallel SystemC simulators | |
| CN111448547A (en) | Method for executing and converting a computer program in a computer network, in particular for controlling a microscope | |
| CN112667491A (en) | Function test method and device of virtual machine | |
| CN112667597B (en) | Algorithm model full life cycle management tool system and implementation method thereof | |
| CN113268436B (en) | Multi-granularity computer simulation operation information acquisition method based on hook points | |
| CN115481021B (en) | Debugging method of automatic process, electronic equipment and storage medium | |
| Akhtar et al. | Formal Specification and Verification of Multi-Agent Robotics Software Systems-A Case Study. | |
| CN117369960A (en) | Task scheduling method and device, storage medium and electronic equipment | |
| Augusto Alonso et al. | RETORCH: Resource-Aware End-to-End Test Orchestration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |