CN110363280A - Algorithm model training analysis system - Google Patents
Algorithm model training analysis system Download PDFInfo
- Publication number
- CN110363280A CN110363280A CN201910430820.7A CN201910430820A CN110363280A CN 110363280 A CN110363280 A CN 110363280A CN 201910430820 A CN201910430820 A CN 201910430820A CN 110363280 A CN110363280 A CN 110363280A
- Authority
- CN
- China
- Prior art keywords
- module
- algorithm model
- model
- data
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a kind of algorithm model training analysis systems, including data access management module, algorithm model management module, model visualization load module, algorithm model training module, visual control module and algorithm model application module.Ability for algorithm model training neural network based, debugging, monitoring is provided, data access management, model management, model visualization assembly, model training, visual control and the model application function of algorithm model needs are provided.
Description
Technical field
The present invention relates to algorithm model training field more particularly to algorithm model training analysis systems.
Background technique
Artificial intelligence is the trend studied now, and algorithm model training analysis is the core of artificial intelligence platform, existing skill
Lack complete set, system algorithm model training analysis tool in art, is trained and analyzes to be directed to algorithm model.It calculates
Method model system should have the function to algorithm model training neural network based, debugging, monitoring, and providing algorithm model needs
Data access management, model management, model visualization assembly, model training, visual control and the model application function wanted.
Summary of the invention
The purpose of the present invention is to overcome the shortcomings of the existing technology, provides a kind of algorithm model training analysis system, can needle
To algorithm model training, debugging and monitoring neural network based, data access management, the model pipe of algorithm model needs are provided
Reason, model visualization assembly, model training, visual control and model application function.
The purpose of the present invention is achieved through the following technical solutions: algorithm model training analysis system, including data
Access Management, algorithm model management module, model visualization load module, algorithm model training module, visual control
Module and algorithm model application module；
Data access management module is the data of algorithm model access training and test；
Algorithm model management module provides the management function to algorithm model；
Each process in algorithm model is encapsulated as component by model visualization load module, and by way of visualization pulls
It is assembled into a complete algorithm model；
Algorithm model training module is used for the training of algorithm model；
Visual control module is checked for model training state；
Algorithm model application module is for applying trained model.
Further, data access module data source include local data, api interface data, database data and
HDFS data, data access obtain initial data, and the file cached as csv format from each data source, are used for follow-up data
Cleaning, refining or model training, test.
Further, local data, api interface data, database data and HDFS data pass through interface encapsulation dependency number
According to operation.
Further, algorithm model management module includes algorithm model creation module, algorithm model update module, algorithm mould
Type removing module, algorithm model import modul, algorithm model export module and algorithm model version management module.
Further, model visualization load module include Component encapsulating module based on job stream and based on Web can
Depending on changing load module；It is disassembled and execute that algorithm model logical definition is converted system by Component encapsulating module based on job stream
Job stream, the Visual assembly module based on Web provides model development tool for user.
Further, algorithm model training module includes distributed scheduling module and conditioning training module；Distribution is adjusted
Degree module includes scheduling system, training actuator and registration center, and conditioning training module includes shape in newly-built state, training
State, physical training condition, halted state and halted state.
Further, model training state includes the summary information, learning rate and iterative relation figure, weight of "current" model
Updated value and the ratio between parameter with the number of iterations relational graph, change over time the mark of situation activation primitive, gradient and updated value
The histogram of quasi- difference and parameter and updated value.
Further, visual control module further includes in the system to business algorithm model actuator place server
It deposits, the monitoring of JVM memory, JVM heap memory, CPU usage variation diagram and hardware information and software information.
Further, algorithm model application module includes modelling effect evaluation module and model application release module；Model
Recruitment evaluation module uses trained model, carries out prediction application to data-oriented, and returns to the data after prediction, with user
Assessment data be compared, and computation model accuracy rate；Model after training is issued as one by model application release module
A RESTFul interface, user are passed to specified parameter, can carry out far call by calling the RESTFul interface.
The beneficial effects of the present invention are: the algorithm model training analysis system of complete set is provided, for based on nerve net
Algorithm model training, debugging, the monitoring of network, provide data access management, the model management, model visualization of algorithm model needs
Assembly, model training, visual control and model application function.
Detailed description of the invention
Fig. 1 is this system structural block diagram；
Fig. 2 is overall architecture of the present invention；
Fig. 3 is data access flow chart；
Fig. 4 is algorithm model management activity figure；
Fig. 5 is algorithm model and Component encapsulating schematic diagram；
Fig. 6 is distributed scheduling schematic diagram；
Fig. 7 is algorithm model training state diagram；
Fig. 8 is model evaluation flow chart.
Specific embodiment
Technical solution of the present invention is described in further detail with reference to the accompanying drawing, but protection scope of the present invention is not limited to
It is as described below.
As shown in figures 1-8, algorithm model training analysis system, including data access management module, algorithm model manage mould
Block, model visualization load module, algorithm model training module, visual control module and algorithm model application module；
Data access management module is the data of algorithm model access training and test；
Algorithm model management module provides the management function to algorithm model；
Each process in algorithm model is encapsulated as component by model visualization load module, and by way of visualization pulls
It is assembled into a complete algorithm model；
Algorithm model training module is used for the training of algorithm model；
Visual control module is checked for model training state；
Algorithm model application module is for applying trained model.
Data access module data source includes local data, api interface data, database data and HDFS data, number
According to access from each data source obtain initial data, and cache be csv format file, for follow-up data cleaning, refining or
Model training, test.
Data access management module provides cleaning after data access, refining tool, so that access data are from non-knot
Structure, nonstandard numbers are according to the normal data for switching to structuring.Including following functions:
File format conversion: supporting to switch to csv file from formats such as txt, xls, xlsx, supports customized separator, support more
Kind character encoding format.
Column count: basic calculating is carried out to column, it is made to meet user demand.It supports to carry out four fundamental rules to single-row or two column datas
Operation, including add, subtract, multiplication and division, support power operation and exponent arithmetic, support logic operation, including be greater than, be less than, being equal to, greatly
In be equal to, be less than or equal to, be not equal to, between etc..As a result certain column can be covered or increased as new column.
Statistics: carrying out basic statistics to each column (feature), facilitates user to be based on statistical result and carries out decision.Including with it is flat
Mean value, median, maximum value, minimum value, variance, standard deviation, relative standard deviation, mean difference, opposite mean difference etc..
Filtering: missing data, the abnormal data, the data for meeting or being unsatisfactory for condition in filter data are crossed.Support filtering number
According to the data that there is missing are concentrated, abnormal data meet the data of logical condition；Support direction filtered model.
Sequence: according to the entire data set of ascending or descending order sequence.It supports to sort to single-row and multi-column data collection；It supports by spy
The formula that fixes (such as time) is ranked up
Standardization: in terms of weather or model training usually requires that data are after standardizing, and standardized tool is provided to data mark
The function of standardization.Support min-max standardization, z-score standardization.
File separator: the function by a file separator for multiple files is provided.It supports to separate by file size.
File mergences: it provides to multiple file mergencess into the function of a file.It supports to merge by row and by column.
Column split: if user with customized separator group organization data, can be according to specified separation by this tool
Symbol or regular expression, which separate, is divided into multi-column data for single-row data.
Column merge: it is corresponding with column split, multi-column data can be merged into a column data.
Data type conversion: the function that data conversion is carried out to single-row or multiple row is provided.Support character string, integer, floating-point
Conversion between type.
Column operation: data column operation is provided.It supports to increase column, delete column, train value conversion.
Local data, api interface data, database data and HDFS data pass through interface encapsulation associated data operation.
Algorithm model management module includes algorithm model creation module, algorithm model update module, algorithm model deletion mould
Block, algorithm model import modul, algorithm model export module and algorithm model version management module.
Algorithm model creation: algorithm model is the basic unit of algorithm model training tool, is the object of model training, is
The concentrated reflection of algorithm.Algorithm model creates the information such as the title of clear algorithm model, description.
Algorithm model updates: user edits an already present model, the information such as modification model name, description.
Algorithm model is deleted: user passes through after Authority Verification, deletes an already present model.
Algorithm model export: the model after a training can be exported to local by user.Derived model remains model
The information such as parameter.
Algorithm model imports: the model after export can be imported into system by user again, the model of importing keep with
The identical parameter information of model before export.
Version management: algorithm model editor provides historical record function, and modification can all automatically record the content of modification every time,
User can fall back on specified old version at any time.
Model visualization load module includes the Component encapsulating module based on job stream and the Visual assembly mould based on Web
Block；Component encapsulating module based on job stream converts algorithm model logical definition to the job stream that system is disassembled and executes,
Visual assembly module based on Web provides model development tool for user.
Component encapsulating based on job stream is to convert algorithm model logical definition to the operation that system is disassembled and executes
Stream, this process are referred to as job stream definition.It is defined by the job stream, algorithm can be bound, divide job stream, from job stream
Deng.Unified component standard can be provided for the design and manufacturing method of algorithm model based on this mode, improve algorithm model
Reusability forms the capitalization of software function feature.
One algorithm model is usually made of multiple general algorithms and logic process flow, the component based on job stream
Encapsulation is that these general-purpose algorithms and logical process encapsulation are independent component, is trainable algorithm model for assembling.
The definition of job stream by the process that encapsulates step by step, by fine granularity to coarseness, by surround and watch it is macroscopical, by bottom to
Using come the interaction that carries out data.
Visual assembly based on Web provides the model development tool of friendly interface for user.What a user creating
After algorithm model, into web Visual assembly interface, algorithm component is pulled, each component, formation algorithm model are connected.
Algorithm model training module includes distributed scheduling module and conditioning training module；Distributed scheduling module includes
Scheduling system, training actuator and registration center, conditioning training module include newly-built state, state in training, have trained shape
State, halted state and halted state.
Scheduling system, that is, operation system, user initiate each order of model training by operation system；Training actuator
It is also known as server in a distributed system, provides service for model training；Distributed registry center is by Zookeeper cluster structure
At the main registration and discovery for completing service.When operation, scheduling system first accesses registration center, inquires available server,
That is algorithm model training actuator, registration center return to available actuator, and control centre initiates model instruction to the actuator again
Practice request.
Distributed scheduling passes through the communication between RPC agreement and RESTFul protocol realization difference process.RPC can be used for same
The high performance communication of language, and RESTFul then meets the communication between different language type.
Visual control module further include the Installed System Memory to server where business algorithm model actuator, JVM memory,
The monitoring of JVM heap memory, CPU usage variation diagram and hardware information and software information.
Algorithm model application module includes modelling effect evaluation module and model application release module；Modelling effect assesses mould
Block uses trained model, carries out prediction application to data-oriented, and returns to the data after prediction, the assessment data with user
It is compared, and computation model accuracy rate；Model after training is issued as a RESTFul and connect by model application release module
Mouthful, user is passed to specified parameter, can carry out far call by calling the RESTFul interface.
The interface of publication provides following parameter:
Apply Names: the model that defining interface calls also can customize.
The address REST: the address that defining interface calls is the calling interface of service.
Parameter: the parameter of defining interface is divided into optional and necessary.
Request type: include the types such as common GET, POST, PUT, DELETE.
Returned data format: including JSON, file type, general character string type
Request example: friendly calling example is provided for user.
Algorithm model training analysis system proposed by the invention, can for algorithm model neural network based training,
Debugging and monitoring, provide algorithm model needs data access management, model management, model visualization assembly, model training, can
Depending on changing monitoring and model application function.
Claims (9)
1. algorithm model training analysis system, it is characterised in that: including data access management module, algorithm model management module,
Model visualization load module, algorithm model training module, visual control module and algorithm model application module；
Data access management module is the data of algorithm model access training and test；
Algorithm model management module provides the management function to algorithm model；
Each process in algorithm model is encapsulated as component by model visualization load module, and by way of visualization pulls
It is assembled into a complete algorithm model；
Algorithm model training module is used for the training of algorithm model；
Visual control module is checked for model training state；
Algorithm model application module is for applying trained model.
2. algorithm model training analysis system according to claim 1, it is characterised in that: the data access module data
Source includes local data, api interface data, database data and HDFS data, and data access obtains former from each data source
Beginning data, and the file cached as csv format, for follow-up data cleaning, refining or model training, test.
3. algorithm model training analysis system according to claim 2, it is characterised in that: the local data, api interface
Data, database data and HDFS data pass through interface encapsulation associated data operation.
4. algorithm model training analysis system according to claim 1, it is characterised in that: the algorithm model management module
Including algorithm model creation module, algorithm model update module, algorithm model removing module, algorithm model import modul, algorithm
Model export module and algorithm model version management module.
5. algorithm model training analysis system according to claim 1, it is characterised in that: the model visualization fit drawing die
Block includes the Component encapsulating module based on job stream and the Visual assembly module based on Web；Component encapsulating based on job stream
Module converts algorithm model logical definition to the job stream that system is disassembled and executes, the Visual assembly module based on Web
Model development tool is provided for user.
6. algorithm model training analysis system according to claim 1, it is characterised in that: the algorithm model training module
Including distributed scheduling module and conditioning training module；Distributed scheduling module includes scheduling system, training actuator and note
Volume center, conditioning training module include state, physical training condition, halted state and halted state in newly-built state, training.
7. algorithm model training analysis system according to claim 1, it is characterised in that: the model training state includes
The pass of the ratio between summary information, learning rate and the iterative relation figure of "current" model, the updated value of weight and parameter with the number of iterations
System's figure changes over time the standard deviation of situation activation primitive, gradient and updated value and the histogram of parameter and updated value.
8. algorithm model training analysis system according to claim 1, it is characterised in that: the visual control module is also
Including changing to the Installed System Memory of server, JVM memory, JVM heap memory, CPU usage where business algorithm model actuator
The monitoring of figure and hardware information and software information.
9. algorithm model training analysis system according to claim 1, it is characterised in that: the algorithm model application module
Including modelling effect evaluation module and model application release module；Modelling effect evaluation module uses trained model, to giving
Fixed number returns to the data after prediction according to carrying out prediction application, is compared with the assessment data of user, and computation model is accurate
Rate；Model after training is issued as a RESTFul interface by model application release module, and user should by calling
RESTFul interface is passed to specified parameter, can carry out far call.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910430820.7A CN110363280A (en) | 2019-09-02 | 2019-09-02 | Algorithm model training analysis system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910430820.7A CN110363280A (en) | 2019-09-02 | 2019-09-02 | Algorithm model training analysis system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110363280A true CN110363280A (en) | 2019-10-22 |
Family
ID=68215328
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910430820.7A Pending CN110363280A (en) | 2019-09-02 | 2019-09-02 | Algorithm model training analysis system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110363280A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110955470A (en) * | 2019-12-06 | 2020-04-03 | 深圳前海环融联易信息科技服务有限公司 | Algorithm model interfacing method, apparatus, computer device and storage medium |
| CN111259064A (en) * | 2020-01-10 | 2020-06-09 | 同方知网(北京)技术有限公司 | Visual natural language analysis mining system and modeling method thereof |
| CN111898742A (en) * | 2020-08-05 | 2020-11-06 | 上海眼控科技股份有限公司 | Method and equipment for monitoring training state of neural network model |
| CN111913715A (en) * | 2020-07-30 | 2020-11-10 | 上海数策软件股份有限公司 | Micro-service based machine learning automation process management and optimization system and method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106202192A (en) * | 2016-06-28 | 2016-12-07 | 浪潮软件集团有限公司 | Workflow-based big data analysis method |
| CN108073582A (en) * | 2016-11-08 | 2018-05-25 | 中移(苏州)软件技术有限公司 | A kind of Computational frame selection method and device |
| CN109189750A (en) * | 2018-09-06 | 2019-01-11 | 北京九章云极科技有限公司 | Operation method, data analysis system and the storage medium of data analysis workflow |
| CN109213482A (en) * | 2018-06-28 | 2019-01-15 | 清华大学天津高端装备研究院 | The graphical application platform of artificial intelligence and application method based on convolutional neural networks |
| CN109710383A (en) * | 2018-12-29 | 2019-05-03 | 上海晏鼠计算机技术股份有限公司 | A kind of method of intelligent algorithm containerization application |
| CN109840111A (en) * | 2019-02-26 | 2019-06-04 | 广州衡昊数据科技有限公司 | A kind of patterned transaction processing system and method |
-
2019
- 2019-09-02 CN CN201910430820.7A patent/CN110363280A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106202192A (en) * | 2016-06-28 | 2016-12-07 | 浪潮软件集团有限公司 | Workflow-based big data analysis method |
| CN108073582A (en) * | 2016-11-08 | 2018-05-25 | 中移(苏州)软件技术有限公司 | A kind of Computational frame selection method and device |
| CN109213482A (en) * | 2018-06-28 | 2019-01-15 | 清华大学天津高端装备研究院 | The graphical application platform of artificial intelligence and application method based on convolutional neural networks |
| CN109189750A (en) * | 2018-09-06 | 2019-01-11 | 北京九章云极科技有限公司 | Operation method, data analysis system and the storage medium of data analysis workflow |
| CN109710383A (en) * | 2018-12-29 | 2019-05-03 | 上海晏鼠计算机技术股份有限公司 | A kind of method of intelligent algorithm containerization application |
| CN109840111A (en) * | 2019-02-26 | 2019-06-04 | 广州衡昊数据科技有限公司 | A kind of patterned transaction processing system and method |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110955470A (en) * | 2019-12-06 | 2020-04-03 | 深圳前海环融联易信息科技服务有限公司 | Algorithm model interfacing method, apparatus, computer device and storage medium |
| CN110955470B (en) * | 2019-12-06 | 2024-01-19 | 深圳前海环融联易信息科技服务有限公司 | Algorithm model interfacing method, device, computer equipment and storage medium |
| CN111259064A (en) * | 2020-01-10 | 2020-06-09 | 同方知网(北京)技术有限公司 | Visual natural language analysis mining system and modeling method thereof |
| CN111913715A (en) * | 2020-07-30 | 2020-11-10 | 上海数策软件股份有限公司 | Micro-service based machine learning automation process management and optimization system and method |
| CN111898742A (en) * | 2020-08-05 | 2020-11-06 | 上海眼控科技股份有限公司 | Method and equipment for monitoring training state of neural network model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110363280A (en) | Algorithm model training analysis system | |
| CN104954453B (en) | Data mining REST service platform based on cloud computing | |
| CN111355606B (en) | Web application-oriented container cluster self-adaptive expansion and contraction system and method | |
| CN107103064B (en) | Data statistical method and device | |
| CN111061788A (en) | Multi-source heterogeneous data conversion integration system based on cloud architecture and implementation method thereof | |
| CN108092813A (en) | Data center's total management system server hardware Governance framework and implementation method | |
| CN103390066A (en) | Database overall automation optimizing early warning device and processing method thereof | |
| CN108038239A (en) | A kind of heterogeneous data source method of standardization management, device and server | |
| US11119989B1 (en) | Data aggregation with schema enforcement | |
| CN106557470A (en) | data extraction method and device | |
| CN107590181A (en) | A kind of intelligent analysis system of big data | |
| CN108733532A (en) | Health degree management-control method, device, medium and the electronic equipment of big data platform | |
| CN108777637A (en) | A kind of data center's total management system and method for supporting server isomery | |
| CN108108986A (en) | A kind of design method of CRM system, device and electronic equipment | |
| CN109669976A (en) | Data service method and equipment based on ETL | |
| CN109308309B (en) | Data service quality assessment method and terminal | |
| Lundberg et al. | Quality attributes in software architecture design | |
| WO2016036386A1 (en) | Dynamically generating an aggregation routine | |
| CN103198099A (en) | Cloud-based data mining application method facing telecommunication service | |
| CN115408381A (en) | Data processing method and related equipment | |
| CN102411757B (en) | Method and system for forecasting capacity of large host central processing unit (CPU) | |
| CN108763323A (en) | Meteorological lattice point file application process based on resource set and big data technology | |
| CN106657282B (en) | Method and device for integrating running state information of converter station equipment | |
| CN109358842A (en) | A kind of service implementing method, electronic equipment and storage medium | |
| Ribeiro et al. | A data integration architecture for smart cities |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191022 |
|
| RJ01 | Rejection of invention patent application after publication |