WO2022222558A1 - 一种人工智能模型在线训练方法及系统 - Google Patents

一种人工智能模型在线训练方法及系统 Download PDF

Info

Publication number
WO2022222558A1
WO2022222558A1 PCT/CN2022/071937 CN2022071937W WO2022222558A1 WO 2022222558 A1 WO2022222558 A1 WO 2022222558A1 CN 2022071937 W CN2022071937 W CN 2022071937W WO 2022222558 A1 WO2022222558 A1 WO 2022222558A1
Authority
WO
WIPO (PCT)
Prior art keywords
model
data
training
inference
level
Prior art date
Application number
PCT/CN2022/071937
Other languages
English (en)
French (fr)
Inventor
周浩
Original Assignee
烽火通信科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 烽火通信科技股份有限公司 filed Critical 烽火通信科技股份有限公司
Publication of WO2022222558A1 publication Critical patent/WO2022222558A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting

Definitions

  • the invention belongs to the technical field of artificial intelligence, and more particularly, relates to an artificial intelligence model online training method and system.
  • the artificial intelligence model is mainly divided into two stages: training and inference.
  • training phase a large amount of historical data that has occurred in a specific scenario is used to tune the model and train to obtain a pre-trained model that meets the expectations.
  • inference phase of the production environment input the real-time data to the pre-trained model to obtain the inference results of the model.
  • the situation and data in specific scenarios are also changing all the time, and the pre-training model based on historical structure and data also has the problem of "fresh-keeping period". If the model is not updated for a long time, the inference accuracy will inevitably become worse and worse. .
  • Digital twin is a multi-disciplinary, multi-physical, multi-scale, multi-probability simulation process that makes full use of physical models, sensor updates, and operating history to complete the mapping in virtual space, thereby reflecting the full life cycle of the corresponding physical equipment. process.
  • the digital twin environment can generate data that is difficult to observe due to the constraints of ethics, cost, security and other factors. Therefore, inference and verification of the AI pre-training model in the digital twin environment can obtain a model that is completely consistent with the real scene, or even more excellent.
  • the present invention provides an online training method and system for an artificial intelligence model, which solves the problem of the continuous accuracy of reasoning in the training and reasoning process of artificial intelligence field models, the true accuracy of training and the security. And other issues.
  • an artificial intelligence model online training system including an inference system, a terminal and a training pre-release system, wherein:
  • the reasoning system belongs to the business system of AI model application, and is responsible for model orchestration and inference operation, including AI orchestrator, service-level AI engine and service-level AI data pool;
  • the terminal belongs to the object of inference system management and collaboration, and at the same time cooperates with the network management to carry out intelligent network operation and maintenance, including terminal-level AI engine and terminal-level AI data pool;
  • the training pre-release system is responsible for model development, training, and model validation and pre-release, including the pre-release environment, AI model development, and platform-level AI data pool.
  • the AI choreographer is used to manage the collection strategy and application strategy of the AI model, and arrange the processing flow of the AI application instance, and each type of AI application has a special strategy to define the data it should collect , the selected model, and the application strategy of the AI engine inference results.
  • the service-level AI engine and the terminal-level AI engine are used to manage AI models, provide inference services for AI applications, and complete data processing, reasoning, online training, and model evaluation for specific AI applications. And the process of model publishing; the service-level AI engine is deployed at the inference system level, the terminal-level AI engine is deployed at the terminal level, and the service-level AI engine in the inference system processes the data required for AI applications from the terminal and the inference system.
  • the terminal-level AI engine of the terminal processes data based on the management information base on the terminal.
  • the service-level AI data pool, the terminal-level AI data pool, and the platform-level AI data pool are used for AI data collection and data processing; wherein the data collection is responsible for data collection from the inference system, terminal or other
  • the data system collects the required training data to form the basic data set; data processing includes data cleaning and aggregation, which is used to process the data into training or inference data that can be directly used by the selected AI algorithm.
  • the pre-release environment is used for verifying, rehearsing and updating the model and training results, and after evaluating the effect of the model in the pre-release environment, the verified model is released to the inference system, and receive feedback from the inference system.
  • the AI model development is used for the development and management of the entire life cycle of the AI model, completes data processing based on the capabilities of the AI data pool to generate training sets and verification sets, and then schedules hardware resources to perform model stand-alone.
  • parallel distributed training store the trained model in the knowledge base after completing the evaluation, and interact with the pre-release environment to complete the evaluation and verification of the model, and publish the verified model to the inference system. .
  • an online training method for an artificial intelligence model comprising:
  • S1 Obtain the full amount of historical data corresponding to the AI model from the AI data pool, and use the full amount of historical data for model development and training;
  • the development and training process includes data preprocessing, model code editing, model debugging, model single-machine training, and model distributed parallelism training;
  • the AI data pool includes a service-level AI data pool, a terminal-level AI data pool, and a platform-level AI data pool;
  • S2 Validate and pre-release the pre-training model in the twin network, and publish it to the inference system after meeting the requirements; use the twin model to simulate various scene data, verify the application of the developed and trained models, and evaluate the model in various scenarios. Accuracy and security; the models that achieve the expected goals are stored in the knowledge base of the AI model development system, and published to the knowledge base in the inference system;
  • S3 Use real-time data for model inference and evaluation; in the inference system, the inference data is obtained from the AI data pool according to the collection and application strategy of the AI orchestrator, the pre-trained model is used to obtain the inference results, and the model is processed according to the online training strategy. Evaluate;
  • S4 Use real-time incremental data for online model training and evaluation and release; obtain real-time incremental training data from the AI data pool according to the collection and application strategy of the AI orchestrator, and perform incremental online training and evaluation of the model according to the online training strategy , and automatically publish the model that meets the predetermined target to the knowledge base in the AI engine, where the AI engine includes a service-level AI engine and a terminal-level AI engine.
  • the step S1 includes the following steps:
  • S12 Process the original data in the AI data pool to generate training sets and validation sets;
  • the AI data pool has big data processing capabilities, supports data cleaning, completion and normalization of raw data, and generates, Store training and validation sets;
  • S13 Develop AI model based on the training set; on the basis of the training set, according to the specific application scenario, use the appropriate development language to develop and debug the AI model code to obtain the appropriate model code;
  • S14 Perform AI model training based on the training set; use a suitable computing framework, input the training set to the model, and train the model;
  • the step S2 includes the following steps:
  • S21 Publish the pre-trained model to the pre-release environment; perform evaluation on the validation set, and release the pre-trained model whose evaluation results meet expectations to the pre-release environment;
  • S22 Use the twin network to generate real-time data of the target network; use the twins in the twin network to generate the real-time running data of the target network;
  • S23 use the twin network real-time data to perform pre-training model inference in the pre-release environment; input the target network real-time operation data generated by the twin body into the pre-training model to obtain an AI model inference result;
  • the step S3 includes the following steps:
  • the AI data pool obtains the real-time operation data of the terminal according to the collection strategy; in the inference system, the AI data pool collects the real-time operation data required for training from the terminal or other data systems according to the collection strategy of the AI orchestrator and performs safe storage;
  • the real-time operation data is cleaned and processed and stored as real-time raw data;
  • the collected real-time operation data generally has problems such as dirty data and missing data; the data needs to be cleaned and processed according to the collection strategy of the AI orchestrator and stored as real-time raw data data;
  • the real-time raw data is processed to generate inference data
  • the AI engine uses the big data processing capability of the AI data pool to process the stored or real-time raw data, and obtain Inference data that meets the requirements of the AI model
  • S34 Use the inference data to apply the applicable model for inference; the AI engine inputs the inference data that meets the requirements of the model into the model to obtain the required inference result;
  • the step S4 includes the following steps:
  • S41 Obtain real-time incremental data from the service-level AI data pool; for models that require incremental training, obtain stored or real-time incremental operation data according to the collection strategy of the AI orchestrator;
  • the real-time incremental data is processed to generate the real-time incremental training data set and the verification set; based on the big data processing capability of the service-level AI data pool, the service-level AI engine adjusts the incremental data according to the application strategy of the AI orchestrator. process the data to generate real-time incremental training data sets and validation sets;
  • S43 Use the real-time incremental training data set to perform online incremental training on the model; the service-level AI engine runs the model incremental training code, and uses the incremental training data set to train the model online;
  • S44 Use the real-time incremental verification data set to evaluate the model online to obtain a retrained model; after the training is completed, use the incremental verification data set to evaluate the retrained model; the retrained model that meets the predetermined target is marked as available, otherwise marked is unavailable;
  • S45 Publish the retrained model to the knowledge base for use in inference; store the model marked as available in the knowledge base, and update the current inference model.
  • the present invention enables the artificial intelligence model to accurately measure the real accuracy rate and security during training, and at the same time maintains the accuracy of the model through an online training mechanism during inference, reducing and upgrading model cost.
  • FIG. 1 is a structural block diagram of an artificial intelligence model online training system in an embodiment of the present invention
  • FIG. 2 is a flowchart of an online training method for an artificial intelligence model according to an embodiment of the present invention.
  • an embodiment of the present invention provides an artificial intelligence model online training system, including: an inference system, a terminal, and a training pre-issue system.
  • the reasoning system belongs to the business system of AI model application, and can be represented as a network management system in the telecommunications field, responsible for model orchestration and inference operation, including AI orchestrator, service-level AI engine and service-level AI data pool;
  • the terminal is the object of inference system management and coordination.
  • it can be represented as a network element, responsible for functions such as service forwarding, and at the same time, it cooperates with the network management to carry out intelligent network operation and maintenance, including terminal-level AI engine and terminal-level AI data. pool;
  • the training pre-release system is responsible for model development, training, and model validation and pre-release, including the pre-release environment, AI model development, and platform-level AI data pool.
  • the AI orchestrator is used to manage the collection strategy and application strategy of the AI model, and orchestrate the processing flow of AI application instances, wherein each type of AI application has a special strategy to define the data it should collect and the selected model. And the application strategy of AI engine inference results, etc.
  • the service-level AI engine and terminal-level AI engine are used to manage AI models, provide inference services for AI applications, and complete the processes of data processing, reasoning, online training, model evaluation, and model release for specific AI applications; service-level AI engines
  • the AI engine is deployed at the inference system level, and the terminal-level AI engine is deployed at the terminal level.
  • the service-level AI engine in the inference system processes the data required for AI applications from the terminal and the inference system.
  • the terminal-level AI engine on the terminal is based on the The management information base processes the data.
  • the AI engine can monitor the AI model and start retraining according to the retraining strategy.
  • the service-level AI data pool, terminal-level AI data pool, and platform-level AI data pool are used for AI data collection and data processing.
  • data collection is responsible for collecting the required training data from the inference system, terminal or other data systems to form a basic data set;
  • data processing includes data processing such as data cleaning and aggregation, the purpose is to process the data into the selected AI algorithm capable of Directly used training and inference data.
  • the pre-release environment is used to verify, rehearse and update the model and training results.
  • After evaluating the performance of the model in the pre-launch environment publish the validated model to the inference system and receive feedback from the inference system.
  • the model needs to be evaluated and verified at different levels, such as the verification of the model itself, and the simulation of the production environment at different levels.
  • the model and training results need to be verified, rehearsed and updated in the pre-launch system.
  • the AI model development is used to be responsible for the development and management of the entire life cycle of the AI model. Based on the capabilities of the AI data pool, complete data processing to generate training sets and validation sets, and then schedule hardware resources for single-machine or parallel distributed training of the model. After the training model is evaluated, it is stored in the knowledge base, and the best ones interact with the pre-release environment to complete the evaluation and verification of the model, and the verified model is released to the inference system (online IDE (Integrated Development Environment, Integrated development environment) to edit and debug model code).
  • online IDE Integrated Development Environment, Integrated development environment
  • an embodiment of the present invention also provides an online training method for an artificial intelligence model, the method comprising the following steps:
  • S1 Obtain the full amount of historical data corresponding to the AI model from the AI data pool, and use the full amount of historical data for model development and training.
  • the development and training process includes but is not limited to: data preprocessing, model code editing, model debugging, model single-machine training and model distributed parallel training.
  • S2 Validate the pre-trained model in the Siamese network and publish it to the inference system after meeting the requirements.
  • Models that achieve the expected goals are stored in the knowledge base of the AI model development system and published to the knowledge base in the inference system.
  • S3 Model inference and evaluation using real-time data.
  • the inference data is obtained from the AI data pool according to the collection and application strategy of the AI orchestrator, the inference results are obtained using the pre-trained model, and the model is evaluated according to the online training strategy.
  • S4 Use real-time incremental data for online model training and evaluation and release. Obtain real-time incremental training data from the AI data pool according to the collection and application strategy of the AI orchestrator, perform incremental online training and evaluation of the model according to the online training strategy, and automatically publish the knowledge of the model that meets the predetermined target to the AI engine in the library.
  • step S1 specifically includes the following steps:
  • the AI data pool can be constructed based on big data products.
  • unstructured processing can be constructed based on Hadoop
  • OLAP On-Line Analytical Processing
  • GreenPlum GreenPlum
  • S12 Process the full amount of historical data in the AI data pool to generate training sets and validation sets.
  • the AI data pool has big data processing capabilities, supports data cleaning, completion, and normalization of full historical data, and generates and stores training sets and validation sets according to model requirements.
  • the general steps of data preprocessing include: missing value processing, feature normalization, discrete and continuous, and denoising.
  • Different big data products more or less support data preprocessing methods, and unsupported processing can be handled by plug-ins or internal programming of the model.
  • the training set and the validation set are generally allocated in a ratio of 8:2, which is configurable.
  • S13 AI model development based on training set.
  • the commonly used AI model development language is python, which can be used to build AI models easily and quickly.
  • S14 AI model training is performed based on the training set. Using a suitable computing framework, input the training set to the model and train the model.
  • the AI data pool can provide service interfaces such as HTTP, RPC, or local reading for AI models to read training sets.
  • service interfaces such as HTTP, RPC, or local reading for AI models to read training sets.
  • computing frameworks such as TensorFlow and pytorch can be used, and machine learning models can use computing frameworks such as scikit-learn.
  • the accuracy of the model can include: mean, recall, precision, and precision.
  • the specific meaning is as follows,
  • Deep learning AI models based on the tensorflow framework can be stored as model files in pb format in the knowledge base.
  • step S2 specifically includes the following steps:
  • S21 Publish the pretrained model to the staging environment. Evaluate on the validation set, and publish the pre-trained model with the evaluation results as expected to the staging environment.
  • S22 Generate target network real-time data using Siamese network. Generate real-time running data of the target network using the siamese in the siamese network.
  • S23 Pre-trained model inference in a pre-launch environment using Siamese network real-time data.
  • the real-time running data of the target network generated by the twin is input into the pre-training model, and the model inference result is obtained.
  • the real-time running data of the target network generated by the twins can be loaded in the following three ways: Preloaded data (preloaded data), Feeding: (Python generates data, and then feeds the data to the backend ), Reading from file (read directly from the file);
  • the security of the model is mainly judged by the influence results of the twins.
  • the result of service interruption is expressed as unsafe, and the situation where the service traffic is greater than the port rate is also regarded as unsafe.
  • S25 Obtain the security training model verified by the Siamese network.
  • the verified models are stored in the knowledge base and released to the inference system, and the failed models are fed back to the AI model development system for parameter adjustment and retraining.
  • the reasoning system provides service interfaces in the form of RPC or FTP, allowing other systems to write AI model files to the knowledge base under authorization.
  • the training pre-launch system writes the AI model into the knowledge base of the reasoning system through the interface in the form of RPC or FTP provided by the reasoning system.
  • step S3 specifically includes the following steps:
  • the AI data pool obtains real-time operation data of the terminal according to the collection strategy.
  • the AI data pool collects the real-time running data required for training from the terminal or other data systems and stores it securely according to the collection strategy of the AI orchestrator.
  • the collection strategy can be expressed as follows, which should conform to the json schema:
  • the real-time operating data is cleaned and processed and stored as real-time raw data.
  • the collected real-time operation data generally has problems such as dirty data and missing data.
  • the data needs to be cleaned and processed according to the collection strategy of the AI orchestrator and stored as real-time raw data.
  • the real-time raw data is processed to generate inference data.
  • the AI engine uses the big data processing capabilities of the AI data pool to process the stored or real-time raw data to obtain inference data that meets the requirements of the AI model.
  • the application strategy can be expressed as follows, which should conform to the json schema:
  • S34 Apply the applicable model for inference using the inference data.
  • the AI engine inputs the inference data that meets the model requirements into the model to obtain the required inference results.
  • timing or real-time training of the application strategy can be expressed as follows, which should conform to the json schema:
  • step S4 specifically includes the following steps:
  • S41 Obtain real-time incremental data from the service-level AI data pool, and obtain stored or real-time incremental operation data according to the collection strategy of the AI orchestrator for models that require incremental training;
  • the real-time incremental data is processed to generate the real-time incremental training data set and the verification set.
  • the service-level AI engine adjusts the incremental data according to the application strategy of the AI orchestrator. process the data to generate real-time incremental training data sets and validation sets;
  • S43 Use the real-time incremental training data set to perform online incremental training on the model, the service-level AI engine runs the model incremental training code, and uses the incremental training data set to train the model online;
  • S44 Use the real-time incremental verification data set to evaluate the model online, and obtain a retrained model. After the training is completed, use the incremental verification data set to evaluate the retrained model. The retrained model that meets the predetermined target is marked as available, otherwise marked is unavailable;
  • S45 Publish the retrained model to the knowledge base for inference use, store the AI model marked as available in the knowledge base, and update the current inference model.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Physics & Mathematics (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Informatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

一种人工智能模型在线训练方法和系统，该系统包括：推理系统，属于AI模型应用的业务系统，负责模型的编排及推理运行，包括AI编排器、服务级AI引擎及服务级AI数据池；终端，属于推理系统管理以及协同的对象，同时与网管协同进行智能化的网络运维，包括终端级AI引擎及终端级AI数据池；训练预发系统，负责模型的开发、训练以及模型验证预发，包括预发环境、AI模型开发及平台级AI数据池。通过构建一种人工智能模型在线训练方法及系统，使得人工智能模型在训练时可以准确的度量真实的准确率和安全性，同时在推理时通过在线训练机制持续保持模型的准确率降低升级模型成本。

Description

一种人工智能模型在线训练方法及系统 技术领域
本发明属于人工智能技术领域，更具体地，涉及一种人工智能模型在线训练方法及系统。
背景技术
人工智能模型从应用上看，主要分为训练和推理两大阶段。在训练阶段使用特定场景下大量已发生的历史数据对模型进行调优、训练得到符合预期的预训练模型。在生产环境中推理阶段向预训练模型输入实时发生的数据得到模型的推理结果。但从实际应用来看，特定场景下的情景、数据也是时刻变化的，基于历史结构和数据的预训练模型也存在“保鲜期”的问题，如果模型长久不更新推理准确率势必越来越差。例如人脸识别模型，随着人年龄的增长，脸型特征也会逐渐发生变化，如果模型长久不更新则准确率会越来越低，最终可能达不到使用要求。同时，对于电信、金融等领域更会带来未验证的安全性等问题。
数字孪生技术的出现，为AI(Artificial Intelligence，人工智能)模型真实准确率和安全性等问题的度量提供了一种良好的解决手段。数字孪生是充分利用物理模型、传感器更新、运行历史等数据，集成多学科、多物理量、多尺度、多概率的仿真过程，在虚拟空间中完成映射，从而反映相对应的实体装备的全生命周期过程。数字孪生环境下可以产生现实由于伦理、成本、安全等因素限制难以观测到的数据。因此，在数字孪生环境中对AI预训练模型进行推理验证，就可以得到与真实场景下完全一致，甚至更加优异的模型。
保持模型持续准确率，有效提升模型真实准确率及安全性是本领域人 员亟需解决的技术问题。
发明内容
针对现有技术的以上缺陷或改进需求，本发明提供了一种人工智能模型在线训练方法及系统，解决人工智能领域模型在训练、推理过程中的推理持续准确率，训练真实准确率和安全性等问题。
为实现上述目的，按照本发明的一个方面，提供了一种人工智能模型在线训练系统，包括推理系统、终端及训练预发系统，其中：
所述推理系统，属于AI模型应用的业务系统，负责模型的编排及推理运行，包括AI编排器、服务级AI引擎及服务级AI数据池；
所述终端，属于推理系统管理以及协同的对象，同时与网管协同进行智能化的网络运维，包括终端级AI引擎及终端级AI数据池；
所述训练预发系统，负责模型的开发、训练以及模型验证预发，包括预发环境、AI模型开发及平台级AI数据池。
本发明的一个实施例中，所述AI编排器，用于管理AI模型的采集策略和应用策略，并且编排AI应用实例的处理流程，每类AI应用有专门的策略来定义其应该收集的数据、所选择的模型以及AI引擎推理结果的应用策略。
本发明的一个实施例中，所述服务级AI引擎和终端级AI引擎，用于进行AI模型管理，提供AI应用的推理服务，针对特定的AI应用完成数据处理、推理、在线训练、模型评估以及模型发布的过程；服务级AI引擎在推理系统层面部署，终端级AI引擎在终端层面部署，推理系统中的服务级AI引擎从终端以及推理系统中进行AI应用所需数据的处理，终端上的终端级AI引擎基于终端上的管理信息库处理数据。
本发明的一个实施例中，所述服务级AI数据池、终端级AI数据池和平台级AI数据池，用于进行AI数据采集和数据处理；其中数据采集负责从推理系统、终端或者其它的数据系统采集所需训练数据，形成基础的数据 集；数据处理包括数据清洗和聚合，用于将数据处理成所选定的AI算法能够直接使用的训练或推理数据。
本发明的一个实施例中，所述预发环境，用于对模型和训练结果进行验证、预演和更新，在评估模型在预发环境下的效果后，将验证后的模型发布到推理系统，并接收来自推理系统的反馈。
本发明的一个实施例中，所述AI模型开发，用于负责AI模型全生命周期的开发管理，基于AI数据池的能力完成数据处理生成训练集和验证集，然后调度硬件资源进行模型的单机或并行分布式训练，对训练的模型完成评估后存放在知识库汇中，并且择优与预发环境进行交互，完成模型的评估与验证，验证通过的模型发布到推理系统中。。
按照本发明的另一方面，还提供了一种人工智能模型在线训练方法，包括：
S1：从AI数据池中获取AI模型对应的全量历史数据，使用全量历史数据进行模型开发和训练；开发和训练过程包括数据预处理、模型代码编辑、模型调试、模型单机训练和模型分布式并行训练；所述AI数据池包括服务级AI数据池、终端级AI数据池和平台级AI数据池；
S2：在孪生网络中对预训练模型进行验证预发，符合要求后发布到推理系统；使用孪生模型模拟各种场景数据，对开发、训练得到的模型进行应用验证，评估各种场景下模型的准确性和安全性；达到预期目标的模型存储到AI模型开发系统的知识库当中，并发布到推理系统中的知识库中；
S3：使用实时数据进行模型推理及评估；在推理系统中，根据AI编排器的采集和应用策略从AI数据池中获得推理数据，使用预训练模型得到推理结果，并且根据在线训练策略对模型进行评估；
S4：使用实时增量数据进行模型在线训练及评估发布；根据AI编排器的采集和应用策略从AI数据池中获得实时的增量训练数据，根据在线训练策略对模型进行增量在线训练及评估，对满足预定目标的模型自动发布到 AI引擎中的知识库中，所述AI引擎包括服务级AI引擎和终端级AI引擎。
本发明的一个实施例中，所述步骤S1中包含以下步骤：
S11：从AI数据池中获取历史的全量原始数据；全量的原始数据规模取决于模型开发的需求；
S12：在AI数据池中对原始数据进行处理，生成训练集和验证集；AI数据池具备大数据处理能力，支持对原始数据进行数据清洗、补全和归一化处理，根据模型需求生成、存储训练集和验证集；
S13：基于训练集进行AI模型开发；在训练集的基础上，根据具体的应用场景，使用合适的开发语言进行AI模型代码的开发、调试，得到合适的模型代码；
S14：基于训练集进行AI模型训练；使用合适的计算框架，向模型输入训练集，对模型进行训练；
S15：基于验证集对AI模型进行评估，得到预训练模型；模型训练完成后，使用验证集对AI模型进行准确率的评估；符合预期目标的模型存储到知识库中供后续使用，否则继续进行参数调整和再次训练。
本发明的一个实施例中，所述步骤S2中包含以下步骤：
S21：将预训练模型发布到预发环境；在验证集上进行评估，将评估结果符合预期的预训练模型发布到预发环境中；
S22：使用孪生网络生成目标网络实时数据；使用孪生网络中的孪生体生成目标网络的实时运行数据；
S23：使用孪生网络实时数据进行预发环境中的预训练模型推理；将孪生体生成的目标网络实时运行数据输入到所述预训练模型中，得到AI模型推理结果；
S24：评估模型在孪生网络中的准确率及安全性；将各种场景下的推理数据汇总，得到最终的准确率，并且记录推理结果对孪生体的影响；对满足预定准确率和孪生体影响结果的模型做通过处理，否则做不通过处理；
S25：得到孪生网络验证后的安全训练模型；将通过验证的模型存储到知识库中并发布到推理系统中，不通过的模型将结果反馈到AI模型开发系统中，进行参数调整及重新训练。
本发明的一个实施例中，所述步骤S3中包含以下步骤：
S31：AI数据池根据采集策略，获取终端实时运行数据；在推理系统中，AI数据池根据AI编排器的采集策略，从终端或者其它的数据系统采集训练所需的实时运行数据并进行安全的存储；
S32：对实时运行数据清洗处理后存储为实时原始数据；采集到的实时运行数据一般存在脏数据、缺失数据等问题；需要根据AI编排器的采集策略对数据进行清洗处理并存储为实时的原始数据；
S33：根据应用策略，对实时原始数据进行处理，生成推理数据；根据AI编排器的应用策略，AI引擎使用AI数据池的大数据处理能力，对已存储或实时发生的原始数据进行处理，得到符合AI模型要求的推理数据；
S34：使用推理数据应用适用的模型进行推理；AI引擎将符合模型要求的推理数据输入到模型中，得到所需的推理结果；
S35：根据应用策略，定时或实时对所述模型进行精度评估；
S36：得到所述模型是否需要进行再次训练的判断；对不符合预定目标的模型做需要增量训练状态记录，否则不进行增量训练状态记录。
本发明的一个实施例中，所述步骤S4中包含以下步骤：
S41：从服务级AI数据池中获取实时增量数据；对需要做增量训练的模型，根据AI编排器的采集策略获取已存储或实时的增量运行数据；
S42：根据应用策略，对实时增量数据进行处理，生成实时增量训练数据集及验证集；基于服务级AI数据池的大数据处理能力，服务级AI引擎根据AI编排器的应用策略对增量数据进行处理，生成实时的增量训练数据集及验证集；
S43：使用实时增量训练数据集对模型进行在线增量训练；服务级AI引 擎运行模型增量训练代码，使用增量训练数据集对模型进行在线训练；
S44：使用实时增量验证数据集对模型进行在线评估，得到重训练模型；训练完成后，使用增量验证数据集对重训练模型进行评估；符合预定目标的重训练模型标记为可用，否则标记为不可用；
S45：将重训练模型发布到知识库，供推理使用；将标记为可用的模型存储到知识库中，并更新当前的推理模型。
总体而言，通过本发明所构思的以上技术方案与现有技术相比，具有如下有益效果：
本发明通过构建一种人工智能模型在线训练方法及系统，使得人工智能模型在训练时可以准确的度量真实的准确率和安全性，同时在推理时通过在线训练机制持续保持模型的准确率降低升级模型成本。
附图说明
图1为本发明实施例中一种人工智能模型在线训练系统的结构框图；
图2为本发明实施例中一种人工智能模型在线训练方法的流程图。
具体实施方式
为了使本发明的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本发明进行进一步详细说明。应当理解，此处所描述的具体实施例仅仅用以解释本发明，并不用于限定本发明。此外，下面所描述的本发明各个实施方式中所涉及到的技术特征只要彼此之间未构成冲突就可以相互组合。
如图1所示，本发明实施例提供一种人工智能模型在线训练系统，包括：推理系统、终端及训练预发系统。
所述推理系统，属于AI模型应用的业务系统，在电信领域可以表现为网管系统，负责模型的编排及推理运行，包括AI编排器、服务级AI引擎及服务级AI数据池；
所述终端，属于推理系统管理以及协同的对象，在电信领域可以表现为 网元，负责业务转发等功能，同时与网管协同进行智能化的网络运维，包括终端级AI引擎及终端级AI数据池；
所述训练预发系统，负责模型的开发、训练以及模型验证预发，包括预发环境、AI模型开发及平台级AI数据池。
其中，所述AI编排器，用于管理AI模型的采集策略和应用策略，并且编排AI应用实例的处理流程，其中每类AI应用有专门的策略来定义其应该收集的数据、所选择的模型以及AI引擎推理结果的应用策略等。
所述服务级AI引擎和终端级AI引擎，用于进行AI模型管理，提供AI应用的推理服务，针对特定的AI应用完成数据处理、推理、在线训练、模型评估以及模型发布的过程；服务级AI引擎在推理系统层面部署，终端级AI引擎在终端层面部署，推理系统中的服务级AI引擎从终端以及推理系统中进行AI应用所需数据的处理，终端上的终端级AI引擎基于终端上的管理信息库处理数据。特别的，在推理系统中，AI引擎可以根据重训练策略对AI模型进行监控并启动再训练。
所述服务级AI数据池、终端级AI数据池和平台级AI数据池，用于进行AI数据采集和数据处理。其中数据采集负责从推理系统、终端或者其它的数据系统采集所需训练数据，形成基础的数据集；数据处理包括数据清洗、聚合等数据处理，目的是将数据处理成所选定的AI算法能够直接使用的训练和推理数据。
所述预发环境，用于对模型和训练结果进行验证、预演和更新。在评估模型在预发环境下的效果后，将验证后的模型发布到推理系统，并接收来自推理系统的反馈。为了保证推理系统的稳健运行，在将算法(模型)训练结果正式部署到推理系统上之前，需要对模型进行不同等级的评估与验证，例如模型本身的验证，以及与不同等级的生产环境仿真相结合，需要在预发系统对模型和训练结果进行验证、预演和更新。
所述AI模型开发，用于负责AI模型全生命周期的开发管理。基于AI 数据池的能力完成数据处理生成训练集和验证集，然后调度硬件资源进行模型的单机或并行分布式训练。对训练的模型完成评估后存放在知识库汇中，并且择优与预发环境进行交互，完成模型的评估与验证，验证通过的模型发布到推理系统中(可提供在线的IDE(Integrated Development Environment，集成开发环境)进行模型代码的编辑和调试等工作)。
参见图2所示，本发明的实施例还提供了一种人工智能模型在线训练方法，该方法包括以下步骤：
S1：从AI数据池中获取AI模型对应的全量历史数据，使用全量历史数据进行模型开发和训练。开发和训练过程包括但不限于：数据预处理、模型代码编辑、模型调试、模型单机训练和模型分布式并行训练。
S2：在孪生网络中对预训练模型进行验证预发，符合要求后发布到推理系统。使用孪生模型模拟各种场景数据，对开发、训练得到的模型进行应用验证，评估各种场景下模型的准确性和安全性。达到预期目标的模型存储到AI模型开发系统的知识库当中，并发布到推理系统中的知识库中。
S3：使用实时数据进行模型推理及评估。在推理系统中，根据AI编排器的采集和应用策略从AI数据池中获得推理数据，使用预训练模型得到推理结果，并且根据在线训练策略对模型进行评估。
S4：使用实时增量数据进行模型在线训练及评估发布。根据AI编排器的采集和应用策略从AI数据池中获得实时的增量训练数据，根据在线训练策略对模型进行增量在线训练及评估，对满足预定目标的模型自动发布到AI引擎中的知识库中。
在上述方案的基础上，所述步骤S1中，具体包含以下步骤：
S11：从AI数据池中获取全量历史数据。全量历史数据规模取决于模型开发的需求。
其中，AI数据池可基于大数据产品进行构建，例如，非结构化的处理可以基于Hadoop构建，面向OLAP(On-Line Analytical Processing，联机分 析处理)类型的数据可以基于GreenPlum构建。
S12：在AI数据池中对全量历史数据进行处理，生成训练集和验证集。AI数据池具备大数据处理能力，支持对全量历史数据进行数据清洗、补全和归一化等处理，根据模型需求生成、存储训练集和验证集。
其中，数据预处理一般步骤包括：缺失值处理、特征规范化、离散与连续化及去噪。不同的大数据产品或多或少支持数据预处理方法，不支持的处理可以通过插件化或模型内部编程处理。训练集和验证集一般按8:2的比例进行分配，该分配比例可配置。
S13：基于训练集进行AI模型开发。在训练集的基础上，根据具体的应用场景，使用合适的开发语言进行AI模型代码的开发与调试，得到合适的模型代码。常用的AI模型开发语言为python，使用python可以方便、快捷的构建AI模型。
S14：基于训练集进行AI模型训练。使用合适的计算框架，向模型输入训练集，对模型进行训练。
AI数据池可以提供HTTP、RPC或本地读取等方式的服务接口，供AI模型读取训练集。对于深度学习模型，可以使用TensorFlow、pytorch等计算框架，机器学习模型可以使用scikit-learn等计算框架。
S15：基于验证集对AI模型进行评估，得到预训练模型。模型训练完成后，使用验证集对AI模型进行准确率的评估。符合预期目标的模型存储到知识库中供后续使用，否则继续进行参数调整和再次训练。
模型的准确率可以包括：平均数、召回率、精确率及准确率。具体含义如下，
Figure PCTCN2022071937-appb-000001
Figure PCTCN2022071937-appb-000002
基于tensorflow框架的深度学习AI模型可以在知识库中存储为pb格式的模型文件。
在上述方案的基础上，所述步骤S2中，具体包含以下步骤：
S21：将预训练模型发布到预发环境。在验证集上进行评估，将评估结果符合预期的预训练模型发布到预发环境中。
对符合预期(如满足模型的准确率要求)的预训练模型，如果基于tensorflow计算框架，则将pb格式的模型文件在预发环境中重新载入。
S22：使用孪生网络生成目标网络实时数据。使用孪生网络中的孪生体生成目标网络的实时运行数据。
S23：使用孪生网络实时数据进行预发环境中的预训练模型推理。将孪生体生成的目标网络实时运行数据输入到所述预训练模型中，得到模型推理结果。
在使用tensorflow计算框架的情况下，将孪生体生成的目标网络实时运行数据可以通过如下3中方式加载数据：Preloaded data(预加载数据)、Feeding:(Python产生数据，再把数据喂给后端)、Reading from file(从文件中直接读取)；
S24：评估模型在孪生网络中的准确率及安全性。将各种场景下的推理数据汇总，得到最终的准确率，并且记录推理结果对孪生体的影响。对满足预定准确率和孪生体影响结果的模型做通过处理，否则做不通过处理。
模型的安全性主要通过孪生体的影响结果来判断。例如：对于电信网络 业务孪生体，业务中断的结果则表示为不安全，业务流量大于端口速率的情况也视为不安全。
S25：得到孪生网络验证后的安全训练模型。将通过验证的模型存储到知识库中并发布到推理系统中，不通过的模型将结果反馈到AI模型开发系统中，进行参数调整及重新训练。
推理系统提供RPC或FTP等形式的服务接口，允许其他系统在授权的情况下向知识库中写入AI模型文件。训练预发系统通过推理系统提供的RPC或FTP等形式的接口，将AI模型写入推理系统知识库中。
在上述方案的基础上，所述步骤S3中，具体包含以下步骤：
S31：AI数据池根据采集策略，获取终端实时运行数据。在推理系统中，AI数据池根据AI编排器的采集策略，从终端或者其它的数据系统采集训练所需的实时运行数据并进行安全的存储。
其中，采集策略可表示如下，应符合json schema：
Figure PCTCN2022071937-appb-000003
Figure PCTCN2022071937-appb-000004
S32：对实时运行数据清洗处理后存储为实时原始数据。采集到的实时运行数据一般存在脏数据、缺失数据等问题。需要根据AI编排器的采集策略对数据进行清洗处理并存储为实时的原始数据。
S33：根据应用策略，对实时原始数据进行处理，生成推理数据。根据AI编排器的应用策略，AI引擎使用AI数据池的大数据处理能力，对已存储或实时发生的原始数据进行处理，得到符合AI模型要求的推理数据。
其中，应用策略可表示如下，应符合json schema：
Figure PCTCN2022071937-appb-000005
S34：使用推理数据应用适用的模型进行推理。AI引擎将符合模型要求的推理数据输入到模型中，得到所需的推理结果。
S35：根据应用策略，定时或实时对所述模型进行精度评估。
其中，应用策略的定时或实时训练可表示如下，应符合json schema：
Figure PCTCN2022071937-appb-000006
Figure PCTCN2022071937-appb-000007
S36：得到所述模型是否需要进行再次训练的判断。对不符合预定目标的模型做需要增量训练状态记录，否则不进行增量训练状态记录。
在上述方案的基础上，所述步骤S4中，具体包含以下步骤：
S41：从服务级AI数据池中获取实时增量数据，对需要做增量训练的模型，根据AI编排器的采集策略获取已存储或实时的增量运行数据；
S42：根据应用策略，对实时增量数据进行处理，生成实时增量训练数据集及验证集，基于服务级AI数据池的大数据处理能力，服务级AI引擎根据AI编排器的应用策略对增量数据进行处理，生成实时的增量训练数据集及验证集；
S43：使用实时增量训练数据集对模型进行在线增量训练，服务级AI引擎运行模型增量训练代码，使用增量训练数据集对模型进行在线训练；
S44：使用实时增量验证数据集对模型进行在线评估，得到重训练模型，训练完成后，使用增量验证数据集对重训练模型进行评估，符合预定目标的 重训练模型标记为可用，否则标记为不可用；
S45：将重训练模型发布到知识库，供推理使用，将标记为可用的AI模型存储到知识库中，并更新当前的推理模型。
本领域的技术人员容易理解，以上所述仅为本发明的较佳实施例而已，并不用以限制本发明，凡在本发明的精神和原则之内所作的任何修改、等同替换和改进等，均应包含在本发明的保护范围之内。

Claims (11)

  1. 一种人工智能模型在线训练系统，其特征在于，包括推理系统、终端及训练预发系统，其中：
    所述推理系统，属于AI模型应用的业务系统，负责模型的编排及推理运行，包括AI编排器、服务级AI引擎及服务级AI数据池；
    所述终端，属于推理系统管理以及协同的对象，同时与网管协同进行智能化的网络运维，包括终端级AI引擎及终端级AI数据池；
    所述训练预发系统，负责模型的开发、训练以及模型验证预发，包括预发环境、AI模型开发及平台级AI数据池。
  2. 如权利要求1所述的人工智能模型在线训练系统，其特征在于，所述AI编排器，用于管理AI模型的采集策略和应用策略，并且编排AI应用实例的处理流程，每类AI应用有专门的策略来定义其应该收集的数据、所选择的模型以及AI引擎推理结果的应用策略。
  3. 如权利要求1或2所述的人工智能模型在线训练系统，其特征在于，所述服务级AI引擎和终端级AI引擎，用于进行AI模型管理，提供AI应用的推理服务，针对特定的AI应用完成数据处理、推理、在线训练、模型评估以及模型发布的过程；服务级AI引擎在推理系统层面部署，终端级AI引擎在终端层面部署，推理系统中的服务级AI引擎从终端以及推理系统中进行AI应用所需数据的处理，终端上的终端级AI引擎基于终端上的管理信息库处理数据。
  4. 如权利要求1或2所述的人工智能模型在线训练系统，其特征在于，所述服务级AI数据池、终端级AI数据池和平台级AI数据池，用于进行AI数据采集和数据处理；其中数据采集负责从推理系统、终端或者其它的数据系统采集所需训练数据，形成基础的数据集；数据处理包括数据清洗和聚合，用于将数据处理成所选定的AI算法能够直接使用的训练或推理数据。
  5. 如权利要求1或2所述的人工智能模型在线训练系统，其特征在于，所述预发环境，用于对模型和训练结果进行验证、预演和更新，在评估模型在预发环境下的效果后，将验证后的模型发布到推理系统，并接收来自推理系统的反馈。
  6. 如权利要求1或2所述的人工智能模型在线训练系统，其特征在于，所述AI模型开发，用于负责AI模型全生命周期的开发管理，基于AI数据池的能力完成数据处理生成训练集和验证集，然后调度硬件资源进行模型的单机或并行分布式训练，对训练的模型完成评估后存放在知识库汇中，并且择优与预发环境进行交互，完成模型的评估与验证，验证通过的模型发布到推理系统中。
  7. 一种人工智能模型在线训练方法，其特征在于，包括：
    S1：从AI数据池中获取AI模型对应的全量历史数据，使用全量历史数据进行模型开发和训练；开发和训练过程包括数据预处理、模型代码编辑、模型调试、模型单机训练和模型分布式并行训练；所述AI数据池包括服务级AI数据池、终端级AI数据池和平台级AI数据池；
    S2：在孪生网络中对预训练模型进行验证预发，符合要求后发布到推理系统；使用孪生模型模拟各种场景数据，对开发、训练得到的模型进行应用验证，评估各种场景下模型的准确性和安全性；达到预期目标的模型存储到AI模型开发系统的知识库当中，并发布到推理系统中的知识库中；
    S3：使用实时数据进行模型推理及评估；在推理系统中，根据AI编排器的采集和应用策略从AI数据池中获得推理数据，使用预训练模型得到推理结果，并且根据在线训练策略对模型进行评估；
    S4：使用实时增量数据进行模型在线训练及评估发布；根据AI编排器的采集和应用策略从AI数据池中获得实时的增量训练数据，根据在线训练策略对模型进行增量在线训练及评估，对满足预定目标的模型自动发布到AI引擎中的知识库中，所述AI引擎包括服务级AI引擎和终端级AI引擎。
  8. 如权利要求7所述的人工智能模型在线训练方法，其特征在于，所述步骤S1中包含以下步骤：
    S11：从AI数据池中获取历史的全量原始数据；全量的原始数据规模取决于模型开发的需求；
    S12：在AI数据池中对原始数据进行处理，生成训练集和验证集；AI数据池具备大数据处理能力，支持对原始数据进行数据清洗、补全和归一化处理，根据模型需求生成、存储训练集和验证集；
    S13：基于训练集进行AI模型开发；在训练集的基础上，根据具体的应用场景，使用合适的开发语言进行AI模型代码的开发、调试，得到合适的模型代码；
    S14：基于训练集进行AI模型训练；使用合适的计算框架，向模型输入训练集，对模型进行训练；
    S15：基于验证集对AI模型进行评估，得到预训练模型；模型训练完成后，使用验证集对AI模型进行准确率的评估；符合预期目标的模型存储到知识库中供后续使用，否则继续进行参数调整和再次训练。
  9. 如权利要求7或8所述的人工智能模型在线训练方法，其特征在于，所述步骤S2中包含以下步骤：
    S21：将预训练模型发布到预发环境；在验证集上进行评估，将评估结果符合预期的预训练模型发布到预发环境中；
    S22：使用孪生网络生成目标网络实时数据；使用孪生网络中的孪生体生成目标网络的实时运行数据；
    S23：使用孪生网络实时数据进行预发环境中的预训练模型推理；将孪生体生成的目标网络实时运行数据输入到所述预训练模型中，得到AI模型推理结果；
    S24：评估模型在孪生网络中的准确率及安全性；将各种场景下的推理数据汇总，得到最终的准确率，并且记录推理结果对孪生体的影响；对满足 预定准确率和孪生体影响结果的模型做通过处理，否则做不通过处理；
    S25：得到孪生网络验证后的安全训练模型；将通过验证的模型存储到知识库中并发布到推理系统中，不通过的模型将结果反馈到AI模型开发系统中，进行参数调整及重新训练。
  10. 如权利要求7或8所述的人工智能模型在线训练方法，其特征在于，所述步骤S3中包含以下步骤：
    S31：AI数据池根据采集策略，获取终端实时运行数据；在推理系统中，AI数据池根据AI编排器的采集策略，从终端或者其它的数据系统采集训练所需的实时运行数据并进行安全的存储；
    S32：对实时运行数据清洗处理后存储为实时原始数据；采集到的实时运行数据一般存在脏数据、缺失数据等问题；需要根据AI编排器的采集策略对数据进行清洗处理并存储为实时的原始数据；
    S33：根据应用策略，对实时原始数据进行处理，生成推理数据；根据AI编排器的应用策略，AI引擎使用AI数据池的大数据处理能力，对已存储或实时发生的原始数据进行处理，得到符合AI模型要求的推理数据；
    S34：使用推理数据应用适用的模型进行推理；AI引擎将符合模型要求的推理数据输入到模型中，得到所需的推理结果；
    S35：根据应用策略，定时或实时对所述模型进行精度评估；
    S36：得到所述模型是否需要进行再次训练的判断；对不符合预定目标的模型做需要增量训练状态记录，否则不进行增量训练状态记录。
  11. 如权利要求7或8所述的人工智能模型在线训练方法，其特征在于，所述步骤S4中包含以下步骤：
    S41：从服务级AI数据池中获取实时增量数据；对需要做增量训练的模型，根据AI编排器的采集策略获取已存储或实时的增量运行数据；
    S42：根据应用策略，对实时增量数据进行处理，生成实时增量训练数据集及验证集；基于服务级AI数据池的大数据处理能力，服务级AI引擎 根据AI编排器的应用策略对增量数据进行处理，生成实时的增量训练数据集及验证集；
    S43：使用实时增量训练数据集对模型进行在线增量训练；服务级AI引擎运行模型增量训练代码，使用增量训练数据集对模型进行在线训练；
    S44：使用实时增量验证数据集对模型进行在线评估，得到重训练模型；训练完成后，使用增量验证数据集对重训练模型进行评估；符合预定目标的重训练模型标记为可用，否则标记为不可用；
    S45：将重训练模型发布到知识库，供推理使用；将标记为可用的模型存储到知识库中，并更新当前的推理模型。
PCT/CN2022/071937 2021-04-21 2022-01-14 一种人工智能模型在线训练方法及系统 WO2022222558A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110428701.5A CN113191502A (zh) 2021-04-21 2021-04-21 一种人工智能模型在线训练方法及系统
CN202110428701.5 2021-04-21

Publications (1)

Publication Number Publication Date
WO2022222558A1 true WO2022222558A1 (zh) 2022-10-27

Family

ID=76977877

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/071937 WO2022222558A1 (zh) 2021-04-21 2022-01-14 一种人工智能模型在线训练方法及系统

Country Status (2)

Country Link
CN (1) CN113191502A (zh)
WO (1) WO2022222558A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116519021A (zh) * 2023-06-29 2023-08-01 西北工业大学 一种惯性导航系统故障诊断方法、系统及设备

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113191502A (zh) * 2021-04-21 2021-07-30 烽火通信科技股份有限公司 一种人工智能模型在线训练方法及系统
CN113837280A (zh) * 2021-09-24 2021-12-24 中关村科学城城市大脑股份有限公司 城市大脑场景下一站式构建ai模型的方法和系统
CN116017493A (zh) * 2021-10-21 2023-04-25 维沃移动通信有限公司 模型请求方法、模型请求处理方法及相关设备
CN116668309A (zh) * 2022-02-21 2023-08-29 维沃移动通信有限公司 Ai模型的在线学习方法、装置、通信设备及可读存储介质
CN114694009A (zh) * 2022-03-31 2022-07-01 上海深至信息科技有限公司 一种人工智能模型的训练系统
CN117371943A (zh) * 2023-10-17 2024-01-09 江苏润和软件股份有限公司 一种基于数据驱动的ai中台模型管理方法及ai中台系统
CN117908847B (zh) * 2024-01-19 2024-07-05 北京三维天地科技股份有限公司 一种可编程式ai开发平台

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108809694A (zh) * 2018-04-27 2018-11-13 广州西麦科技股份有限公司 业务编排方法、系统、装置与计算机可读存储介质
CN112417358A (zh) * 2020-12-03 2021-02-26 合肥中科类脑智能技术有限公司 Ai模型训练在线实训学习系统及方法
US20210064346A1 (en) * 2019-08-30 2021-03-04 Bull Sas Support system for designing an artificial intelligence application, executable on distributed computing platforms
CN113191502A (zh) * 2021-04-21 2021-07-30 烽火通信科技股份有限公司 一种人工智能模型在线训练方法及系统

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108093030B (zh) * 2017-11-29 2022-02-22 杭州博联智能科技股份有限公司 一种基于云服务器的人工智能模型部署方法
CN110297659A (zh) * 2018-03-21 2019-10-01 北京京东尚科信息技术有限公司 算法模型部署上线方法和装置
CN110687808A (zh) * 2019-10-14 2020-01-14 苏州数言信息技术有限公司 基于物联网和机器学习的室内智慧节能系统
CN110888722B (zh) * 2019-11-15 2022-05-20 北京奇艺世纪科技有限公司 任务处理方法、装置、电子设备及计算机可读存储介质
CN112347692B (zh) * 2020-09-21 2024-02-06 深圳有电物联科技有限公司 不间断电源的电池预测性维护的实现方法、装置和电子装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108809694A (zh) * 2018-04-27 2018-11-13 广州西麦科技股份有限公司 业务编排方法、系统、装置与计算机可读存储介质
US20210064346A1 (en) * 2019-08-30 2021-03-04 Bull Sas Support system for designing an artificial intelligence application, executable on distributed computing platforms
CN112417358A (zh) * 2020-12-03 2021-02-26 合肥中科类脑智能技术有限公司 Ai模型训练在线实训学习系统及方法
CN113191502A (zh) * 2021-04-21 2021-07-30 烽火通信科技股份有限公司 一种人工智能模型在线训练方法及系统

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
HAN, BINGTAO: "uSmartInsight: an intelligent engine for autonomously evolving networks", ZTE TECHNOLOGY (NEWSLETTER), no. 7, 22 July 2019 (2019-07-22), pages 18 - 20, XP009540407 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116519021A (zh) * 2023-06-29 2023-08-01 西北工业大学 一种惯性导航系统故障诊断方法、系统及设备
CN116519021B (zh) * 2023-06-29 2023-09-15 西北工业大学 一种惯性导航系统故障诊断方法、系统及设备

Also Published As

Publication number Publication date
CN113191502A (zh) 2021-07-30

Similar Documents

Publication Publication Date Title
WO2022222558A1 (zh) 一种人工智能模型在线训练方法及系统
US20240070487A1 (en) Systems and methods for enriching modeling tools and infrastructure with semantics
US10310968B2 (en) Developing software project plans based on developer sensitivity ratings detected from monitoring developer error patterns
EP3627408A1 (en) Equipment repair management and execution
US11282035B2 (en) Process orchestration
CN112148586A (zh) 机器辅助质量保证和软件改善
US20200242540A1 (en) Predicting a work progress metric for a user
US20130024167A1 (en) Computer-Implemented Systems And Methods For Large Scale Automatic Forecast Combinations
CN105190564A (zh) 通过用线程强度分析进行季节性趋向和预报来对云服务中的sla违反进行预测性诊断
CN110457294A (zh) 一种数据处理方法和装置
WO2020087011A2 (en) Resource configuration and management system
Martens et al. From monolithic to component-based performance evaluation of software architectures: a series of experiments analysing accuracy and effort
Bozzon et al. Pattern-based specification of crowdsourcing applications
US20210311729A1 (en) Code review system
US20230353590A1 (en) Project failure reduction through generation, structuring, and/or assessment of a critical event objectt
WO2022134001A1 (zh) 基于容器化技术的机器学习模型框架的开发方法与系统
Pícha et al. Process anti-pattern detection–a case study
US20190340540A1 (en) Adaptive continuous log model learning
Omori et al. Comparing concept drift detection with process mining software
Huang et al. Reviewing rounds prediction for code patches
Estrada-Torres et al. Discovering business process simulation models in the presence of multitasking
US20080319809A1 (en) System and method of maintaining contracts in business process management
do Amaral et al. Attribute selection with filter and wrapper: an application on incident management process
Lazareva Estimating development time and complexity of programs
CN117493140B (zh) 一种针对深度学习模型的评测系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22790668

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22790668

Country of ref document: EP

Kind code of ref document: A1