CN112529023A - Configured artificial intelligence scene application research and development method and system - Google Patents
Configured artificial intelligence scene application research and development method and system Download PDFInfo
- Publication number
- CN112529023A CN112529023A CN201910867877.3A CN201910867877A CN112529023A CN 112529023 A CN112529023 A CN 112529023A CN 201910867877 A CN201910867877 A CN 201910867877A CN 112529023 A CN112529023 A CN 112529023A
- Authority
- CN
- China
- Prior art keywords
- model
- data
- artificial intelligence
- algorithm
- configuration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013473 artificial intelligence Methods 0.000 title claims abstract description 52
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000012827 research and development Methods 0.000 title claims abstract description 19
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 79
- 238000012549 training Methods 0.000 claims abstract description 55
- 238000012545 processing Methods 0.000 claims abstract description 45
- 238000007726 management method Methods 0.000 claims abstract description 31
- 238000004364 calculation method Methods 0.000 claims abstract description 19
- 238000011161 development Methods 0.000 claims abstract description 19
- 238000012544 monitoring process Methods 0.000 claims abstract description 18
- 238000005457 optimization Methods 0.000 claims abstract description 18
- 238000011160 research Methods 0.000 claims abstract description 7
- 238000013461 design Methods 0.000 claims abstract description 6
- 230000000007 visual effect Effects 0.000 claims abstract description 6
- 230000006870 function Effects 0.000 claims description 33
- 238000012360 testing method Methods 0.000 claims description 21
- 238000011156 evaluation Methods 0.000 claims description 15
- 238000004806 packaging method and process Methods 0.000 claims description 14
- 238000010276 construction Methods 0.000 claims description 11
- 238000005516 engineering process Methods 0.000 claims description 8
- 238000003860 storage Methods 0.000 claims description 8
- 230000002159 abnormal effect Effects 0.000 claims description 7
- 239000008186 active pharmaceutical agent Substances 0.000 claims description 7
- 238000013135 deep learning Methods 0.000 claims description 6
- 238000002372 labelling Methods 0.000 claims description 6
- 230000010354 integration Effects 0.000 claims description 5
- 238000013500 data storage Methods 0.000 claims description 4
- 230000008676 import Effects 0.000 claims description 4
- 238000012512 characterization method Methods 0.000 claims description 3
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 230000002452 interceptive effect Effects 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 238000009825 accumulation Methods 0.000 abstract description 4
- 230000007613 environmental effect Effects 0.000 abstract description 3
- 239000012141 concentrate Substances 0.000 abstract 1
- 230000008569 process Effects 0.000 description 11
- 230000006872 improvement Effects 0.000 description 4
- 238000010801 machine learning Methods 0.000 description 3
- 238000012821 model calculation Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000003066 decision tree Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 206010000117 Abnormal behaviour Diseases 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000007635 classification algorithm Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000002790 cross-validation Methods 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000007477 logistic regression Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007637 random forest analysis Methods 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
- G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Medical Informatics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
One or more embodiments of the present specification provide a method and a system for artificial intelligence configuration research and development, which are applied to artificial intelligence research and development including multiple fields of scenes, and the configured artificial intelligence research and development system includes: the system research and development tool is a function suite integrated by various tools, and comprises a data processing tool, a model training tool and a configuration tool, wherein the data processing tool, the model training tool and the configuration tool provide visual interface operation; secondly, an algorithm engine integrates and customizes various algorithm frameworks of artificial intelligence mainstream and self-research; and thirdly, infrastructure provides calculation optimization and operation monitoring for environmental management of system operation. The method and the system can help non-artificial intelligence professionals to concentrate on the business scheme, and can realize the data sorting, the model establishment, the model deployment and the operation monitoring through the configuration function without code development, thereby quickly realizing the delivery of the artificial intelligence application system. In the system, the artificial intelligence application scenes are subjected to classification management, and the field specialized design and the accumulation of data and knowledge are performed on the artificial intelligence application, so that the artificial intelligence application system based on the scene field can be developed and delivered more quickly, and has higher professional level.
Description
1 technical field
The invention belongs to the technical field of artificial intelligence, and particularly relates to a method and a system for rapidly researching, developing and delivering artificial intelligence application through configuration.
2 background of the invention
The artificial intelligence technology is rapidly developed, various technical frameworks and algorithms are continuously proposed, the requirements of various artificial intelligence applications are increased at present, but the artificial intelligence technology has high threshold and high research and development cost, and the practical accumulation of industrial field data and models is lacked. The existing tools need code development, need deep understanding of a bottom-layer algorithm, lack of interactive development, lack of scene management and industry accumulation of a model, lack of a correction optimization function of an expert on the model and the like, low development efficiency, high implementation threshold, high cost and long implementation period, and the problems become artificial intelligence development of most enterprises. No effective solution has been proposed at present.
Disclosure of the invention
In view of this, the present specification provides a method and a system for configuring an artificial intelligence rapid research and development delivery application system, and an implementer can implement data arrangement, model establishment, model deployment and operation monitoring in the system through a configuration function without code development by only concentrating on a business scheme, so that delivery of the artificial intelligence application system can be rapidly implemented.
In order to achieve the above object, the present specification provides a method for artificial intelligence configuration research and development, which is applied to artificial intelligence research and development including scenes in various fields. The method comprises the following steps:
for different data sources, data import is carried out;
through data processing, the data imported into the data source is cleaned, integrated and the like;
performing characteristic processing on the cleaned data through data characterization, such as realizing standardization, and adjusting various characteristic parameters and the like;
establishing a new model set or selecting an existing model set according to an application scene through model set management, establishing project engineering under the model set, and managing parameters of the model set;
selecting a data source, and dividing the characterized data into training data and testing data according to a certain rule;
selecting a calculation frame, selecting an algorithm, combining the algorithms, adjusting the algorithm sequence and the logic relationship, and establishing a model;
adjusting model algorithm parameters;
selecting a target function and a model operation parameter;
training the model to obtain a training result and testing the model to obtain a testing result through model training;
evaluating the test result by selecting an evaluation method through model evaluation;
the expert checks the training data and the test data, removes abnormal data and adjusts the model parameters;
releasing the model by one key to generate service;
managing and analyzing the parameters of the model set, and generating the application range/optimal value and the like of the scene application, thereby realizing the continuous optimization of the scene application;
packaging the algorithm framework, packaging the algorithm and providing a use interface and an interface;
packaging the operating environment into a container, and optimizing the computing power of a GPU/CPU and the like;
packaging the operation/deployment instruction of each algorithm frame to form deployment service;
and monitoring the model and the service, displaying indexes such as state/performance and the like, and executing operations such as starting, suspending/stopping and the like.
Accordingly, in order to achieve the above object, the present invention provides a platform system for research and development of configured artificial intelligence. The system comprises three subsystems of a system tool, an algorithm engine and infrastructure:
a system tool: providing visual interface operation for a function suite integrated by various tools;
an algorithm engine: various algorithm frameworks of mainstream and self-research of artificial intelligence are integrated and customized;
basic implementation: and the calculation optimization and the operation monitoring are provided for the environmental management of the system operation.
As a further improvement of the invention, the system tool comprises a data processing tool, a model training tool and a configuration tool:
10: the data processing tool consists of a data access tool, a data processing module, a sample marking tool and a characteristic processing module.
11: and the data access tool is used for accessing different data sources.
12: and the data processing module is used for performing data processing functions such as format conversion and the like on the data source data.
13: and the sample labeling tool is used for generating sample labeling data serving as basic data for training.
14: a feature processing tool: the method is used for preprocessing data before model training and converting business data into machine-learned digital feature data.
As a further improvement of the present invention, the model training tool comprises:
21: and the scene management module is used for accumulating and managing the knowledge of the scene model, and achieves the aims of continuously optimizing the scene model and supporting a specific service scene scheme.
22: a model construction module: the method can be used for single model construction and complex model construction and supports various algorithms.
23: a model training module: for training the model.
24: a model evaluation module: the method is used for determining an evaluation standard, and analyzing and evaluating the quality of the model.
25: a model group management module: the method is used for classification management of the scene set.
26: an expert system: the method is used for analyzing, adjusting and optimizing the domain model by experts, and achieves the purpose of optimizing the model by experts.
27: a model release module: the method is used for model publishing and generating consumable services.
28: and a version management module: for version management of the trained and published models.
As a further refinement of the invention, the configuration tool comprises:
31: a configuration module: the method is used for providing an interactive manual interface for each function, and comprises a dragging graphic process configuration function.
32: a code editor: for advanced design of data, models, services, etc. by code.
33: and (3) SDK: and the API is used for providing calling API functions, including data input, data processing, model operation, model release, calculation results and the like.
As a further improvement of the present invention, the unified algorithm engine comprises:
41: and (3) an algorithm framework: and the AI framework is used for integrating and packaging the main stream.
42: an algorithm library: the method is used for introducing algorithms in various frameworks and integrating self-research algorithms.
43: and the algorithm integration module performs upper-layer packaging on the algorithms in the algorithm library and realizes the parameter configuration function.
44: an algorithm tuning module: the method is used for carrying out scene algorithm optimization at the algorithm level.
45: the calculation force optimization module: the system is used for providing distributed parallel computing for a running environment and providing computing power.
As a further development of the invention, the infrastructure comprises:
51: operating the container: the method is used for encapsulating middleware, class libraries and the like operated by the mainstream deep learning framework to generate a directly deployable container.
52: deploying a tool: the method is used for publishing the training model and realizing the online deployment of the model and the service.
53: data storage: the method is used for storing various data in various calculation processes and supports various storage media and storage schemes.
54: operating the console: and the control console is used for scheduling and monitoring the operation of servers, models, services and the like.
According to the technical scheme, the method and the system for configurating artificial intelligence research and development, provided by the specification, realize configurated delivery and management of artificial intelligence research and development processes such as data processing, model training, service release, operation monitoring and the like, realize scene model management, and achieve the technical effect of rapid configuration research and development of artificial intelligence scene application.
4 description of the drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
FIG. 1 is a schematic diagram of a technical framework for developing a configured artificial intelligence scenario provided in the present specification;
the numbers in the figures correspond to module names in the summary and the detailed description, and are as follows:
10: data processing means, 11: data access means, 12: data processing module, 13: sample labeling tool, 14: a feature processing tool; 20: model training tool, 21: scene management module, 22: model building module, 23: model training module, 24: model evaluation module, 25: model set management module, 26: expert system, 27: model publishing module, 28: a version management module; 30: configuration tool, 31 configuration module, 32: code editor, 33: SDK; 40: algorithm engine, 41: algorithm framework, 42: algorithm library, 43: algorithm integration module, 44: algorithm tuning module, 45: a calculation power optimization module; 50: infrastructure, 51: operating the vessel, 52: deployment tool, 53: data storage, 54: and operating the console.
FIG. 2 is a diagram of a supportable artificial intelligence technology framework and algorithm described in this specification.
Detailed description of the preferred embodiments
To make the objects, aspects and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the invention are shown, and which, instead, are merely examples of systems and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
It should be noted that: in other embodiments, the steps of the corresponding methods are not necessarily performed in the order shown and described herein. In some other embodiments, the method may include more or fewer steps than those described herein. Moreover, a single step described in this specification may be broken down into multiple steps for description in other embodiments; multiple steps described in this specification may be combined into a single step in other embodiments.
The invention is described in further detail below with reference to the attached drawing figures:
the specification provides a set of configurable artificial intelligence research and development method and system, implementers can realize data arrangement, model establishment, model deployment and operation monitoring in the system through configuration functions without code development only by concentrating on a business scheme, and accordingly delivery of an artificial intelligence application system can be quickly realized. In the method and the system, the artificial intelligence application scenes are subjected to classification management, and the field specialized design and the accumulation of data and knowledge are performed on the artificial intelligence application, so that the artificial intelligence application system based on the scene field can be developed and delivered more quickly and has higher professional level.
The rapid research and development system comprises a system tool, an algorithm engine and an infrastructure, wherein the system tool is a function suite integrated by various tools, comprises a data processing tool, a model training tool and a configuration tool and provides visual interface operation; the algorithm engine integrates and customizes various algorithm frameworks of mainstream and self-research of artificial intelligence; the basic implementation provides for environmental management of system operation, computational optimization and operational monitoring.
A system tool: the system for providing the front-end operation interface for the user consists of a series of research and development tools.
10: a data processing tool: the system is composed of a data access tool, a data processing module, a sample marking tool and a characteristic processing module (11-14).
11: data access tools, including but not limited to the following data access modes:
the method comprises the following steps of (1) importing a database, and supporting conventional database products such as mysql, oracle and KV types, ETL tools and the like;
file import, which supports the import of file data in formats such as csv, txt and the like;
in a crawler mode, data access is acquired through a crawler;
and the API interface is used for interfacing data through the interface and supporting formats such as json and xml.
12: and the data processing module is responsible for performing functions of format conversion, default value processing, abnormal data processing, duplicate removal and the like.
13: and the sample labeling tool can select the picture sample by a user, label the picture area and name the label. And marking samples with large data volume as training basic data for visual identification function.
14: a feature processing tool: the method comprises the steps of preprocessing acquired data before model training, converting the acquired data into digital features of machine learning, and performing functions of digitalization, discretization, normalization (scaling according to the minimum and maximum values), standardization (processing the data to a specified range), dimension reduction processing (reducing the number of dimensions during high-dimensional calculation, improving the training speed), feature selection and the like on the features.
20: model training tool comprising the following modules (21-28):
21: the scene management module can set different application scenes, and each application scene has a training model and an algorithm set which are most suitable for the application scene, such as image recognition, voice recognition, an environment monitoring scene, anomaly monitoring, an estimation scene, a prediction scene, a wind control scene, a recommendation scene, a knowledge graph and the like, and is continuously expanded;
for example, in a prediction scene with relatively complete and ordered data, algorithms such as logistic regression and time sequence are adopted, in a prediction scene with divergent data, an optimized random forest machine learning algorithm, an LSTM algorithm and the like are adopted for deep learning training, a stack of decision trees with random data subsets is created, a model is trained on random samples of a data set for multiple times, and a good prediction result is obtained from the algorithms;
in the abnormal behavior monitoring model, a Recursive Bayesian Estimation (RBE) algorithm is applied, the behavior pattern of each device and each person is obtained by unsupervised learning, the estimated probability of an event is continuously updated along with the discovery of new characteristics, and whether behaviors are abnormal or not is automatically judged.
22: a model construction module: single model construction and complex model construction can be performed. Selecting a specific deep learning algorithm to create a model in the single model construction; in the construction of the complex model, a plurality of algorithms can be selected to carry out further model calculation methods such as sequential processing, composite calculation, weighting, interference resistance and the like. For example, clustering analysis (unsupervised machine learning algorithm) can be performed through K-means, a given data set is operated through a preset number of clusters K, and model operation of the next step is performed on different groups of data after classification, so that the accuracy of operation of each group is improved.
23: a model training module: selecting training data, setting parameters corresponding to an algorithm, setting various parameters such as training step length, output requirements and the like for training. The training system has the functions of starting the training of a certain model, monitoring a training process log, stopping the training and the like.
24: a model evaluation module: and determining an evaluation standard and an evaluation target, comparing the test data with the test result output after model calculation, and evaluating by methods such as deviation, variance, confidence interval, Holdout, cross validation, ROC curve, AUC and the like. And the results are recorded.
25: a model group management module: the models are classified and managed according to two dimensions of scenes and projects, field experience is accumulated aiming at typical scenes, and models belonging to a certain project can be managed in each project.
26: an expert system: for the model of the industry field, a field expert can analyze the test result on an interface, analyze training data and test data, find data abnormality, remove abnormal data or correct errors according to expert experience, and also can adjust parameter weight to perform training test to achieve the purpose of optimizing the model by the expert.
27: a model release module: and releasing the trained model to the running environment in a one-key mode. The callable service may be generated by specifying an IP address, a port number, defining a service name, a service URL, access rights, etc.
28: and a version management module: and performing version management on the trained model and the released model, grouping and marking the models, and simultaneously performing online on multiple versions.
30: a configuration tool comprising the following modules (31-33):
31: a configuration module: the method is used for configuring functions such as data processing, model creation and the like, and comprises a dragging graphic process configuration function, so that a user can operate the whole modeling process on an interface without encoding.
32: a code editor: the method is used for high-level design, and can be used for programming a model design which cannot be met through interface configuration through a python language and a system self-developed function in a code editor, and performing functions such as specific feature processing, model calculation, parameter setting and the like.
33: and (3) SDK: and providing API functions called externally, including data input, data processing, model operation, model release, calculation result and other packaged APIs.
40: unified algorithm engine, comprising the following modules (41-45):
41: and (3) an algorithm framework: the module integrates mainstream AI frameworks including but not limited to Tensorflow, Caffe, Mxnet, Dlib, NLP and the like, and can perform functions of selection and scheduling, parameter optimization and the like at the front end.
42: an algorithm library: algorithms incorporated into the above framework include, but are not limited to, regression, classification, clustering, decision trees, neural networks (convolutional neural networks, recurrent neural networks, etc.), visual algorithms, etc., such as LR, LFC, GBDT, GBRT, FR-CNN, Densenet, YOLO, BLSTM, CTC, ASR, NLU, KMeans, SVM, LSTM, GRU, GAN, etc.
43: and the algorithm integration module performs upper-layer packaging on the algorithms in the algorithm library and realizes the parameter configuration function.
44: an algorithm tuning module: according to the operation effect of the model in each scene, various parameters such as sigmoid functions and the like are set, and scene experience values are recorded for automatic learning and can also be manually adjusted on a front-end interface for testing, so that the optimization of the algorithm on each scene is achieved. Setting an objective function, calculating loss of the estimated value and the actual value, and supporting optimization by using different optimization algorithms, including but not limited to the following algorithms: momentum, Nesterov, RMSProp, AdaGrad, Adam, SGD, GD, and the like.
45: the calculation force optimization module: the operating environment of model training is designed based on spark and other technologies, distributed parallel computing is provided, and computing efficiency is improved by dozens of times.
50: infrastructure comprising the following modules (51-54):
51: operating the container: the method comprises the steps of carrying out standardized encapsulation on middleware, class libraries and the like operated by a mainstream deep learning framework, wherein the middleware, class libraries and the like comprise CNTK, Tensorflow, PyTorch, Caffe2, MXNet and the like, carrying out calculation optimization on GPU operation based on technologies such as Yarn, K8 and the like, and encapsulating the GPU operation into a docker container, so that one-time deployment and operation can be realized, the complex processes of installation and configuration at each time are reduced, the GPU can be fully utilized, and the load of a server is more balanced;
when the container is started on a GPU or a CPU server, all the specified models are loaded into a memory from a model version library, a plurality of same or different models can be scheduled on the GPU or the CPU in a concurrent mode, and the utilization rate of the GPU or the CPU can be automatically improved.
52: deploying a tool: and issuing a training model, realizing multi-model online deployment, monitoring the running model, the log and the performance of the model running, and performing error reporting and alarming.
53: data storage: data such as source data, in-process data, characteristic data, training data, test data, operation results, expert data, project engineering, audio and video and the like of data acquisition are stored by adopting a resource server, a relational database, HDFS distributed storage and the like respectively.
54: operating the console: the running condition of each product can be checked on the console, the resource use conditions of a GPU/CPU, a memory and the like are monitored, the resource scheduling of the server is carried out, and the running scheduling and monitoring of the model service are carried out.
Hereinafter, an example of a configured artificial intelligence development method and system provided in the present specification will be described in detail by taking an application scenario of intelligent pricing for financial assets in the field of financial technology as an example.
The automobile assets are high-value assets in financial scenes, but the automobile assets are opaque, one automobile has one condition and one price, the price influence factors are numerous, and the problems of data shortage, inaccurate estimation and the like exist in the traditional mode of estimation. Through artificial intelligence code development, the threshold is high, and the cycle is long. Therefore, in the embodiment, the configuration development and delivery of the artificial intelligence scene can be performed through the following steps;
in the scene management module, a vehicle asset valuation scene is established, and a vehicle asset valuation project is established;
cleaning and importing various calculation factor data such as automobile transaction data, automobile condition data, automobile basic data and the like by using a data importing tool;
the data processing tool removes abnormal data and standardizes information such as vehicle types and the like;
in the sample marking tool, the characteristic marking can be carried out on the automobile picture, so that the functions of automatically identifying the automobile type/frame number and the like through the picture can be achieved;
in the characteristic processing module, irregular data such as vehicle types and the like are processed (such as normalization) and the like to be converted into numerical data, and dimension reduction operation and the like are performed;
the data processing work is realized;
in the model training tool, a model creating module divides a transaction data selectable rule (such as a random rule) into training data and test data, selects a classification algorithm (such as K-means) and an estimation prediction algorithm (such as LSTM), selects a loss function, and can adjust certain parameters (such as sigmoid) if necessary;
defining training times, step length and the like in a model training module, and executing training; selecting test data to test;
after the test is finished, the training result can be seen in the evaluation module, and the accuracy of the model estimation can be checked/analyzed and judged by selecting methods such as an ROC curve and the like;
the evaluation expert can analyze whether abnormal transaction records exist in the expert system and remove or correct the data;
the version difference can be compared in the version module, the optimal version can be selected to be released to the production environment in the model release module, the vehicle valuation service is generated, and the access link for providing the service to the outside is generated;
at the operation console, the operation state and the performance of the vehicle estimation model in the production environment can be monitored;
optimal parameters can be recommended by comparing/accumulating different items in the same scene, so that model parameters can be set and adjusted in an expert system, and a mechanism for continuously and automatically optimizing a field scene model is realized;
in the above example, the entire artificial intelligence application delivery process need not be encoded, and is accomplished entirely through interface configuration.
The above description is only a preferred embodiment of the present disclosure, and should not be taken as limiting the present disclosure, and any modifications, equivalents, improvements, etc. made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.
Claims (8)
1. A configuration-based artificial intelligence scene application development platform system, comprising:
the system tool is a functional suite integrated by various artificial intelligence research and development tools, provides visual interface operation, and realizes research, development and management of an artificial intelligence model from data processing to model creation, training, evaluation, optimization, release and complete operation flow through configuration;
the algorithm engine integrates and customizes various algorithm frames and algorithms of artificial intelligence mainstream and self-research, and provides algorithm service and parameter management for the outside;
and basic implementation provides functions of deployment, calculation optimization, operation monitoring and the like for environment management of the operation of the artificial intelligent application system.
2. The configuration-based artificial intelligence scenario application development platform system of claim 1, wherein the system tools comprise:
the data processing tool is used for acquiring various data sources and finally converting the data sources into data meeting deep learning characteristics and requirements;
the model training tool is used for realizing the tool for managing the whole life cycle of the artificial intelligent model;
and the configuration tool is used for carrying out configuration research and development.
3. The configuration-based artificial intelligence scenario application development platform system of claim 1 or 2, wherein the data processing tool comprises:
the data access tool is used for accessing data of different data sources;
the data processing module is used for performing data processing functions such as conversion, duplicate removal, exception checking, default value processing, integration, simple calculation and the like on data source data;
the sample labeling tool is used for generating sample labeling data serving as basic data for training;
and the feature processing tool is used for preprocessing data before model training and converting the source data into machine-learned digital feature data.
4. The configuration-based artificial intelligence scenario application development platform system of claim 1 or 2, wherein the model training tool comprises:
the scene management module is used for accumulating and managing the knowledge of the scene model, and achieves the aims of continuously optimizing the scene model and supporting a service scene scheme;
the model construction module can carry out single model construction or complex model construction and supports various algorithms;
the model set management module is used for classified management of the scene sets;
the model training module is used for training a model;
the model evaluation module is used for determining an evaluation standard, and analyzing and evaluating the quality of the model;
the expert system is used for analyzing, adjusting and optimizing the domain model by experts to realize the purpose of optimizing the model by experts; the model issuing module is used for issuing a model and generating consumable service;
and the version management module is used for carrying out version management on the trained model and the released model.
5. The configuration-based artificial intelligence scenario application development platform system of claim 1 or 2, wherein the configuration tool comprises:
the configuration module is used for providing an interactive manual interface for system functions, and comprises a dragging graphic flow configuration function;
the code editor is used for realizing high-level design of data, models, services and the like through code compiling;
and the SDK is used for providing called API functions, including data input, data processing, model operation, model release, calculation result and other packaged APIs.
6. The configuration-based artificial intelligence scenario application development platform system of claim 1, wherein the algorithm engine comprises:
the algorithm framework is used for integrating, packaging and managing a mainstream or self-developed artificial intelligence technology framework;
the algorithm library is used for introducing and managing the algorithms in the technical framework or the self-research algorithms;
the algorithm integration module performs upper-layer packaging on the algorithms in the algorithm library to realize functions such as parameter configuration and the like;
the algorithm tuning module is used for optimizing the algorithm;
and the calculation power optimization module provides distributed parallel calculation and improves the calculation capability.
7. The configuration-based artificial intelligence scenario application development platform system of claim 1, wherein the infrastructure comprises:
the operation container is used for packaging middleware, class libraries and the like operated by the mainstream deep learning framework to generate a container capable of being directly deployed;
the deployment tool is used for releasing the model, realizing the online deployment of the model and the service and managing the operating environment;
the data storage is used for storing data used by the system such as source data, marking data, model data, result data and the like, and supporting various storage media and storage schemes;
and the operation console is used for scheduling and monitoring the operation of resources such as servers, models, services and the like.
8. A method for researching and developing a configuration type artificial intelligence scene application is characterized by comprising the following steps:
data import is carried out on different kinds of data sources;
through data processing, data of a data source are cleaned, converted, integrated and the like;
through data characterization, the cleaned data is subjected to feature processing and converted into artificial intelligent easily-processed feature data, so that various characterization parameters can be adjusted and the like, such as standardization, normalization and the like;
establishing a new model set or selecting an existing model set according to an application scene through scene management, establishing project engineering under the model set, and managing parameters of the model set;
selecting a data source, and dividing the characterized data into training data and testing data according to a certain rule;
selecting an artificial intelligence technical frame, selecting an algorithm or a combined algorithm, adjusting the algorithm sequence or the logic relationship, and establishing a model through configuration on an interface;
or establishing a model by writing codes;
adjusting model algorithm parameters;
selecting a target function and a model operation parameter;
training the model to obtain a training result and testing the model to obtain a testing result through model training;
evaluating the test result by selecting an evaluation method through model evaluation;
the expert analyzes and checks the training data and the test data, removes abnormal data and adjusts the model parameters;
issuing a model to generate a callable service;
managing and analyzing the parameters of the model set, and generating the application range, the optimal value and the like of the scene application, thereby realizing the continuous optimization of the scene application;
packaging the artificial intelligence technology framework, packaging the algorithm and providing a use interface and an interface;
packaging resources required by operation, generating a container, and optimizing the computing power of a GPU (graphics processing unit) or a CPU (central processing unit) and the like;
packaging the instructions of each algorithm frame to form deployment service;
and monitoring the model and the service, displaying indexes such as state, performance and the like, and executing operations such as starting, pausing, stopping and the like.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910867877.3A CN112529023A (en) | 2019-09-18 | 2019-09-18 | Configured artificial intelligence scene application research and development method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910867877.3A CN112529023A (en) | 2019-09-18 | 2019-09-18 | Configured artificial intelligence scene application research and development method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112529023A true CN112529023A (en) | 2021-03-19 |
Family
ID=74974269
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910867877.3A Pending CN112529023A (en) | 2019-09-18 | 2019-09-18 | Configured artificial intelligence scene application research and development method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112529023A (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113112247A (en) * | 2021-05-11 | 2021-07-13 | 云从科技集团股份有限公司 | System, method, electronic device and storage medium for rapidly constructing industry application |
| CN113592017A (en) * | 2021-08-10 | 2021-11-02 | 菲特（天津）检测技术有限公司 | Deep learning model standardization training method, management system and processing terminal |
| CN113792029A (en) * | 2021-09-22 | 2021-12-14 | 电信科学技术第十研究所有限公司 | Rapid development framework for big data processing and analysis model and construction method thereof |
| CN113812851A (en) * | 2021-09-09 | 2021-12-21 | 熊猫智慧水务有限公司 | Water age control system for direct drinking water purification equipment |
| CN113837280A (en) * | 2021-09-24 | 2021-12-24 | 中关村科学城城市大脑股份有限公司 | Method and system for building AI model in one-stop manner under urban brain scene |
| CN114066629A (en) * | 2021-11-15 | 2022-02-18 | 深圳华云信息系统有限公司 | Model management AI platform and model management method |
| CN114268661A (en) * | 2021-11-19 | 2022-04-01 | 科大讯飞股份有限公司 | Service scheme deployment method, device, system and equipment |
| CN114421628A (en) * | 2022-03-30 | 2022-04-29 | 北京全路通信信号研究设计院集团有限公司 | Debugging system and method for auxiliary monitoring system of substation |
| CN114501164A (en) * | 2021-12-28 | 2022-05-13 | 海信视像科技股份有限公司 | Method and device for labeling audio and video data and electronic equipment |
| CN115329670A (en) * | 2022-08-11 | 2022-11-11 | 深圳朗道智通科技有限公司 | Data acquisition method for unmanned vehicle |
| CN116187507A (en) * | 2022-12-07 | 2023-05-30 | 华润三九（枣庄）药业有限公司 | Traditional chinese medicine production system of adjusting based on artificial intelligence |
| WO2023115570A1 (en) * | 2021-12-24 | 2023-06-29 | 深圳晶泰科技有限公司 | Management method and apparatus for machine learning model, computer device and storage medium |
| CN116521778A (en) * | 2023-03-29 | 2023-08-01 | 山东省国土空间数据和遥感技术研究院(山东省海域动态监视监测中心) | One-stop data analysis tool, method and terminal equipment based on artificial intelligence |
| CN116933062A (en) * | 2023-09-18 | 2023-10-24 | 中孚安全技术有限公司 | Intelligent file judgment system and method |
| CN117635083A (en) * | 2024-01-26 | 2024-03-01 | 之江实验室 | Intelligent hardware chip model integrated development system and method for population intelligent scene |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103489053A (en) * | 2013-05-13 | 2014-01-01 | 湖南云控科技有限公司 | Intelligent water resource control platform based on cloud computing and expert system |
| CN108881446A (en) * | 2018-06-22 | 2018-11-23 | 深源恒际科技有限公司 | A kind of artificial intelligence plateform system based on deep learning |
| CN109523316A (en) * | 2018-11-16 | 2019-03-26 | 杭州珞珈数据科技有限公司 | The automation modeling method of commerce services model |
| CN109739922A (en) * | 2019-01-10 | 2019-05-10 | 江苏徐工信息技术股份有限公司 | A kind of industrial data intelligent analysis system |
| CN110163233A (en) * | 2018-02-11 | 2019-08-23 | 陕西爱尚物联科技有限公司 | A method of so that machine is competent at more complex works |
-
2019
- 2019-09-18 CN CN201910867877.3A patent/CN112529023A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103489053A (en) * | 2013-05-13 | 2014-01-01 | 湖南云控科技有限公司 | Intelligent water resource control platform based on cloud computing and expert system |
| CN110163233A (en) * | 2018-02-11 | 2019-08-23 | 陕西爱尚物联科技有限公司 | A method of so that machine is competent at more complex works |
| CN108881446A (en) * | 2018-06-22 | 2018-11-23 | 深源恒际科技有限公司 | A kind of artificial intelligence plateform system based on deep learning |
| CN109523316A (en) * | 2018-11-16 | 2019-03-26 | 杭州珞珈数据科技有限公司 | The automation modeling method of commerce services model |
| CN109739922A (en) * | 2019-01-10 | 2019-05-10 | 江苏徐工信息技术股份有限公司 | A kind of industrial data intelligent analysis system |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113112247A (en) * | 2021-05-11 | 2021-07-13 | 云从科技集团股份有限公司 | System, method, electronic device and storage medium for rapidly constructing industry application |
| CN113112247B (en) * | 2021-05-11 | 2024-06-11 | 云从科技集团股份有限公司 | System, method, electronic device and storage medium for quickly constructing industry application |
| CN113592017A (en) * | 2021-08-10 | 2021-11-02 | 菲特（天津）检测技术有限公司 | Deep learning model standardization training method, management system and processing terminal |
| CN113592017B (en) * | 2021-08-10 | 2024-05-31 | 菲特（天津）检测技术有限公司 | Deep learning model standardized training method, management system and processing terminal |
| CN113812851A (en) * | 2021-09-09 | 2021-12-21 | 熊猫智慧水务有限公司 | Water age control system for direct drinking water purification equipment |
| CN113792029A (en) * | 2021-09-22 | 2021-12-14 | 电信科学技术第十研究所有限公司 | Rapid development framework for big data processing and analysis model and construction method thereof |
| CN113792029B (en) * | 2021-09-22 | 2023-12-01 | 电信科学技术第十研究所有限公司 | Rapid development framework for big data processing and analysis model and construction method thereof |
| CN113837280A (en) * | 2021-09-24 | 2021-12-24 | 中关村科学城城市大脑股份有限公司 | Method and system for building AI model in one-stop manner under urban brain scene |
| CN114066629A (en) * | 2021-11-15 | 2022-02-18 | 深圳华云信息系统有限公司 | Model management AI platform and model management method |
| CN114268661A (en) * | 2021-11-19 | 2022-04-01 | 科大讯飞股份有限公司 | Service scheme deployment method, device, system and equipment |
| CN114268661B (en) * | 2021-11-19 | 2024-04-30 | 科大讯飞股份有限公司 | Service scheme deployment method, device, system and equipment |
| WO2023115570A1 (en) * | 2021-12-24 | 2023-06-29 | 深圳晶泰科技有限公司 | Management method and apparatus for machine learning model, computer device and storage medium |
| CN114501164A (en) * | 2021-12-28 | 2022-05-13 | 海信视像科技股份有限公司 | Method and device for labeling audio and video data and electronic equipment |
| CN114421628A (en) * | 2022-03-30 | 2022-04-29 | 北京全路通信信号研究设计院集团有限公司 | Debugging system and method for auxiliary monitoring system of substation |
| CN115329670A (en) * | 2022-08-11 | 2022-11-11 | 深圳朗道智通科技有限公司 | Data acquisition method for unmanned vehicle |
| CN116187507A (en) * | 2022-12-07 | 2023-05-30 | 华润三九（枣庄）药业有限公司 | Traditional chinese medicine production system of adjusting based on artificial intelligence |
| CN116521778A (en) * | 2023-03-29 | 2023-08-01 | 山东省国土空间数据和遥感技术研究院(山东省海域动态监视监测中心) | One-stop data analysis tool, method and terminal equipment based on artificial intelligence |
| CN116933062B (en) * | 2023-09-18 | 2023-12-15 | 中孚安全技术有限公司 | Intelligent file judgment system and method |
| CN116933062A (en) * | 2023-09-18 | 2023-10-24 | 中孚安全技术有限公司 | Intelligent file judgment system and method |
| CN117635083A (en) * | 2024-01-26 | 2024-03-01 | 之江实验室 | Intelligent hardware chip model integrated development system and method for population intelligent scene |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112529023A (en) | Configured artificial intelligence scene application research and development method and system | |
| US10877863B2 (en) | Automatic prediction system for server failure and method of automatically predicting server failure | |
| US10311044B2 (en) | Distributed data variable analysis and hierarchical grouping system | |
| CN113792825B (en) | Fault classification model training method and device for electricity information acquisition equipment | |
| US20190065343A1 (en) | Automated Log Analysis and Problem Solving Using Intelligent Operation and Deep Learning | |
| US20170109657A1 (en) | Machine Learning-Based Model for Identifying Executions of a Business Process | |
| US20170109676A1 (en) | Generation of Candidate Sequences Using Links Between Nonconsecutively Performed Steps of a Business Process | |
| US20170109668A1 (en) | Model for Linking Between Nonconsecutively Performed Steps in a Business Process | |
| US20180007145A1 (en) | Graphically managing data classification workflows in a social networking system with directed graphs | |
| US20180004835A1 (en) | Data classification workflows implemented with dynamically modifiable directed graphs | |
| US20170109667A1 (en) | Automaton-Based Identification of Executions of a Business Process | |
| US20170109636A1 (en) | Crowd-Based Model for Identifying Executions of a Business Process | |
| CN112579728B (en) | Behavior abnormity identification method and device based on mass data full-text retrieval | |
| US20200286095A1 (en) | Method, apparatus and computer programs for generating a machine-learning system and for classifying a transaction as either fraudulent or genuine | |
| CN110705598A (en) | Intelligent model management method and device, computer equipment and storage medium | |
| CN113626241B (en) | Abnormality processing method, device, equipment and storage medium for application program | |
| Lugaresi et al. | Generation and tuning of discrete event simulation models for manufacturing applications | |
| CN109542737A (en) | Platform alert processing method, device, electronic device and storage medium | |
| CN115641162A (en) | Prediction data analysis system and method based on construction project cost | |
| CN113704389A (en) | Data evaluation method and device, computer equipment and storage medium | |
| CN113326177A (en) | Index anomaly detection method, device, equipment and storage medium | |
| CN115689334A (en) | Efficiency analysis method and system of warehouse management system and computer equipment | |
| CN116089218A (en) | Dynamic baseline alarm method and system based on Kubernetes historical data and trend analysis | |
| CN115358481A (en) | Early warning and identification method, system and device for enterprise ex-situ migration | |
| US20170109637A1 (en) | Crowd-Based Model for Identifying Nonconsecutive Executions of a Business Process |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |