CN108664540A - Big data machine learning system and method - Google Patents
Big data machine learning system and method Download PDFInfo
- Publication number
- CN108664540A CN108664540A CN201810148259.9A CN201810148259A CN108664540A CN 108664540 A CN108664540 A CN 108664540A CN 201810148259 A CN201810148259 A CN 201810148259A CN 108664540 A CN108664540 A CN 108664540A
- Authority
- CN
- China
- Prior art keywords
- module
- data
- controller
- machine learning
- model training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Electrically Operated Instructional Devices (AREA)
Abstract
The invention discloses big data machine learning system and methods, including power supply, controller, model training and data feedback module, the model training includes assessment system and test sample, the assessment system includes index evaluation module and visualization evaluation and test module, the test sample is connected with sample collection, the sample collection is connected with external data, the controller is connected with sensor, the sensor is connected with browsing module, the data feedback module is connected with data processing module and filter module, the filter module includes amplification module and module for reading and writing, the data processing module includes data storage module, data transmission module, converter and data acquisition module.The present invention improves the development deployment efficiency of standardization big data, and is capable of providing succinct unified interface, so that algorithm development, application and development and framework exploitation can realize modular operation.
Description
Technical field
The present invention relates to a kind of big data technical fields, specially big data machine learning system and method.
Background technology
The technical solution of machine learning algorithm model based on big data in the prior art, sufficiently complex, the technology of design
Level covers the every aspects such as Distributed Calculation, framework deployment, model calculating, data mining, spends prodigious manpower and materials
It can complete, each flow is required for continually carrying out exchanging visit operation with file system, substantially reduces the performance of whole system, from
And the reliability of modeling, prediction and application is caused to reduce, it is often more important that the flexibility programmed, ease for maintenance, code can be caused
Or the reusability of component is a greater impact, therefore cause user experience poor.
Invention content
The technical problem to be solved by the present invention is to overcome the defects of the prior art, provide big data machine learning system and side
Method realizes adaptability and versatility to various application scenarios, realizes to being arrived involved in standardization big data development process
Data mining, machine learning Effec-tive Function, simplify standardization big data development process, improve standardization big data
Development deployment efficiency, and succinct unified interface is capable of providing, so that algorithm development, application and development and framework develop energy
Enough realize modular operation, the efficiency for greatly improving the deployment efficiency of big data platform and being excavated to data.
To solve the above problems, the present invention provides the following technical solutions：Big data machine learning system and method, including electricity
Source, controller, model training and data feedback module, the model training include assessment system and test sample, the assessment
System includes index evaluation module and visualization evaluation and test module, and the test sample is connected with sample collection, and the sample is received
Collection is connected with external data, and the controller is connected with sensor, and the sensor is connected with browsing module, the number
It is connected with data processing module and filter module according to feedback module, the filter module includes amplification module and module for reading and writing,
The data processing module includes data storage module, data transmission module, converter and data acquisition module.
As a kind of preferred embodiment of the present invention, the power supply is powered for entire circuit, the power supply and the control
Device processed is connected, and the model training is connected with the controller, and the data feedback module is connected with the controller,
The controller is that whole system handles and convey information.
As a kind of preferred embodiment of the present invention, the mark evaluation module passes through power cord and the model training phase
Connection, the test sample are connected by power cord with the model training, and the index evaluation module is with the assessment
System is connected, and the visualization evaluation and test module is connected with the assessment system.
As a kind of preferred embodiment of the present invention, the amplification module is connected with the filter module, the reading
Writing module is connected with the amplification module.
As a kind of preferred embodiment of the present invention, the data processing module is connected with the data feedback module
It connects, the data transmission module is connected with the data processing module, and the converter is connected with the data transmission module
It connects, the data acquisition module is connected with the converter.
As a kind of preferred embodiment of the present invention, the method for the big data machine learning system, including following
Step：
A. Client-initiated is received by data acquisition module first and asks included executable file；
B. then the data received are stored and is fed back, to be first filtered in feedback, using letter
The amplification and read-write of breath；
C. after the completion of waiting for step b, then by data feedback to controller, then by controller by data feedback to sensor, by
Sensor feedback carries out checking for data to browsing module；
D. it after the completion of waiting for step c, is handled by controller and is analyzed, data are then subjected to model training.
As a kind of preferred embodiment of the present invention, it is with word and picture that the browsing module, which carries out checking for data,
Form show.
As a kind of preferred embodiment of the present invention, the model training is evaluated and tested with index evaluation and visualization later
Two kinds of forms are evaluated and tested.
Compared with prior art, beneficial effects of the present invention are as follows：
Big data machine learning system of the present invention and method realize adaptability and versatility to various application scenarios,
The Effec-tive Function to the data mining, machine learning arrived involved in standardization big data development process is realized, standard is simplified
The development process for changing big data, improves the development deployment efficiency of standardization big data, and is capable of providing succinct unified interface,
So that algorithm development, application and development and framework exploitation can realize modular operation, big data platform is greatly improved
Deployment efficiency and efficiency that data are excavated.
Description of the drawings
Fig. 1 is big data machine learning system of the present invention and the flow chart of method.
In figure：Power supply 1, controller 2, data feedback module 3, model training 4, assessment system 5, test sample 6, index is commented
Estimate module 7, visualizes evaluation and test module 8, sample collection 9, external data 10, sensor 11, browsing module 12, data processing module
13, filter module 14, amplification module 15, module for reading and writing 16, data storage module 17, data transmission module 18, converter 19, number
According to acquisition module 20.
Specific implementation mode
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation describes.
Referring to Fig. 1, the present invention provides a kind of technical solution：Big data machine learning system and method, including power supply 1,
Controller 2, data feedback module 3 and model training 4, the model training 4 include assessment system 5 and test sample 6, institute's commentary
It includes index evaluation module 7 and visualization evaluation and test module 8 to estimate system 5, and the test sample 6 is connected with sample collection 9, described
Sample collection 9 is connected with external data 10, and the controller 2 is connected with sensor 11, the sensor 11 and browsing mould
Block 12 is connected, and the data feedback module 3 is connected with data processing module 13 and filter module 14, the filter module 14
Including amplification module 15 and module for reading and writing 16, the data processing module 13 includes data storage module 17, data transmission module
18, converter 19 and data acquisition module 20.
The power supply 1 is the power supply of entire circuit, and the power supply 1 is connected with the controller 2, the model training 4 and
The controller 2 is connected, and the data feedback module 3 is connected with the controller 2, and the controller 2 is whole system
Processing and reception and registration information.
The mark evaluation module is connected by 1 line of power supply with the model training 4, and the test sample 6 passes through power supply 1
Line is connected with the model training 4, and the index evaluation module 7 is connected with the assessment system 5, the visualization evaluation and test
Module 8 is connected with the assessment system 5.
The amplification module 15 is connected with the filter module 14, the module for reading and writing 16 and 15 phase of the amplification module
Connection.
The data processing module 13 is connected with the data feedback module 3, the data transmission module 18 with it is described
Data processing module 13 is connected, and the converter 19 is connected with the data transmission module 18, the data acquisition module
20 are connected with the converter 19.
The method of the big data machine learning system, includes the following steps：
A. Client-initiated is received by data acquisition module 20 first and asks included executable file；
B. then the data received are stored and is fed back, to be first filtered in feedback, using letter
The amplification and read-write of breath；
C. after the completion of waiting for step b, then by data feedback to controller 2, then by controller 2 by data feedback to sensor
11, browsing module 12 is fed back to by sensor 11 and carries out checking for data；
D. it after the completion of waiting for step c, is handled and is analyzed by controller 2, data are then subjected to model training 4.
The browsing module 12, which carries out checking for data, to be shown in the form of word and picture.
The model training 4 is evaluated and tested in the form of two kinds of index evaluation and visualization evaluation and test later.
During big data machine learning, the reception of data is carried out by data acquisition module 20 first, then by turning
Parallel operation 19 then carries out data into being about to data and convert be sent to data transmission module 18 to program by data storage module 17 again
Storage, data storage feeds back information to filter module 14 by data feedback module 3 again and is filtered interference ripple, connect later
It and information is transmitted to by module for reading and writing 16 by amplification module 15 is again read out, finally carried out by controller control browsing module 12
Data checking and again by model training 4, being assessed again by assessment system 5 after the completion of training.
The power supply 1 of the present invention, controller 2, data feedback module 3, model training 4, assessment system 5, test sample 6 refer to
Evaluation module 7 is marked, evaluation and test module 8, sample collection 9, external data 10, sensor 11, browsing module 12, data processing are visualized
Module 13, filter module 14, amplification module 15, module for reading and writing 16, data storage module 17, data transmission module 18, converter
19, the equal components of data acquisition module 20 are universal standard part or the component that those skilled in the art know, structure and principle
All it is this technology personnel can learn by technical manual or by routine experiment method know, realizes to various application scenarios
Adaptability and versatility, realize to standardization big data development process involved in arrive data mining, machine learning height
Effect operation, simplifies the development process of standardization big data, improves the development deployment efficiency of standardization big data, and can carry
For succinct unified interface, so that algorithm development, application and development and framework exploitation can realize modular operation, greatly
The efficiency for improving the deployment efficiency of big data platform and data being excavated.
The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention, for this field skill
For art personnel, it is clear that invention is not limited to the details of the above exemplary embodiments, and without departing substantially from the present invention spirit or
In the case of essential characteristic, the present invention can be realized in other specific forms.Therefore, in all respects, should all incite somebody to action
Embodiment regards exemplary as, and is non-limiting, the scope of the present invention by appended claims rather than on state
Bright restriction, it is intended that including all changes that come within the meaning and range of equivalency of the claims in the present invention
It is interior.Any reference signs in the claims should not be construed as limiting the involved claims.
The foregoing is only a preferred embodiment of the present invention, is not intended to restrict the invention, although with reference to aforementioned reality
Applying example, invention is explained in detail, for those skilled in the art, still can be to aforementioned each implementation
Technical solution recorded in example is modified or equivalent replacement of some of the technical features.All essences in the present invention
With within principle, any modification, equivalent replacement, improvement and so on should all be included in the protection scope of the present invention god.
Claims (8)
1. big data machine learning system, including power supply (1), controller (2), data feedback module (3) and model training (4),
It is characterized in that：The model training (4) includes assessment system (5) and test sample (6), and the assessment system (5) includes referring to
Evaluation module (7) and visualization evaluation and test module (8) are marked, the test sample (6) is connected with sample collection (9), the sample
Collect (9) be connected with external data (10), the controller (2) is connected with sensor (11), the sensor (11) and
Browsing module (12) is connected, and the data feedback module (3) is connected with data processing module (13) and filter module (14),
The filter module (14) includes amplification module (15) and module for reading and writing (16), and the data processing module (13) includes data storage
Storing module (17), data transmission module (18), converter (19) and data acquisition module (20).
2. big data machine learning system according to claim 1, it is characterised in that：The power supply (1) is entire circuit
Power supply, the power supply (1) are connected with the controller (2), and the model training (4) is connected with the controller (2), institute
It states data feedback module (3) with the controller (2) to be connected, the controller (2) is that whole system handles and convey information.
3. big data machine learning system according to claim 1, it is characterised in that：The mark evaluation module passes through power supply
(1) line is connected with the model training (4), and the test sample (6) passes through power supply (1) line and the model training (4) phase
Connection, the index evaluation module (7) are connected with the assessment system (5), the visualization evaluation and test module (8) and institute's commentary
Estimate system (5) to be connected.
4. big data machine learning system according to claim 1, it is characterised in that：The amplification module (15) with it is described
Filter module (14) is connected, and the module for reading and writing (16) is connected with the amplification module (15).
5. big data machine learning system according to claim 1, it is characterised in that：The data processing module (13) with
The data feedback module (3) is connected, and the data transmission module (18) is connected with the data processing module (13), institute
It states converter (19) with the data transmission module (18) to be connected, the data acquisition module (20) and the converter (19)
It is connected.
6. according to the method for the big data machine learning system described in claim 1 to 5, which is characterized in that include the following steps：
A. Client-initiated is received by data acquisition module (20) first and asks included executable file；
B. then the data received are stored and is fed back, to be first filtered in feedback, using information
Amplification and read-write；
C. after the completion of waiting for step b, then by data feedback to controller (2), then by controller (2) by data feedback to sensor
(11), browsing module (12) is fed back to by sensor (11) and carries out checking for data；
D. it after the completion of waiting for step c, is handled and is analyzed by controller (2), data are then subjected to model training (4).
7. big data machine learning system according to claim 6 and method, it is characterised in that：The browsing module (12)
Carrying out checking for data is shown in the form of word and picture.
8. big data machine learning system according to claim 6 and method, it is characterised in that：The model training (4)
Later evaluated and tested in the form of two kinds of index evaluation and visualization evaluation and test.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810148259.9A CN108664540A (en) | 2018-02-13 | 2018-02-13 | Big data machine learning system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810148259.9A CN108664540A (en) | 2018-02-13 | 2018-02-13 | Big data machine learning system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108664540A true CN108664540A (en) | 2018-10-16 |
Family
ID=63784124
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810148259.9A Pending CN108664540A (en) | 2018-02-13 | 2018-02-13 | Big data machine learning system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108664540A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110321112A (en) * | 2019-07-02 | 2019-10-11 | 北京百度网讯科技有限公司 | AI ability research/development platform and data processing method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105677615A (en) * | 2016-01-04 | 2016-06-15 | 北京邮电大学 | Distributed machine learning method based on weka interface |
| US20160232457A1 (en) * | 2015-02-11 | 2016-08-11 | Skytree, Inc. | User Interface for Unified Data Science Platform Including Management of Models, Experiments, Data Sets, Projects, Actions and Features |
| CN106250987A (en) * | 2016-07-22 | 2016-12-21 | 无锡华云数据技术服务有限公司 | A kind of machine learning method, device and big data platform |
| CN106447046A (en) * | 2016-10-17 | 2017-02-22 | 深圳市彬讯科技有限公司 | House type design scheme evaluation method based on machine learning |
| CN106878409A (en) * | 2017-02-09 | 2017-06-20 | 深圳市莫二科技有限公司 | A kind of game data processing system and processing method |
| CN107229976A (en) * | 2017-06-08 | 2017-10-03 | 郑州云海信息技术有限公司 | A kind of distributed machines learning system based on spark |
-
2018
- 2018-02-13 CN CN201810148259.9A patent/CN108664540A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160232457A1 (en) * | 2015-02-11 | 2016-08-11 | Skytree, Inc. | User Interface for Unified Data Science Platform Including Management of Models, Experiments, Data Sets, Projects, Actions and Features |
| CN105677615A (en) * | 2016-01-04 | 2016-06-15 | 北京邮电大学 | Distributed machine learning method based on weka interface |
| CN106250987A (en) * | 2016-07-22 | 2016-12-21 | 无锡华云数据技术服务有限公司 | A kind of machine learning method, device and big data platform |
| CN106447046A (en) * | 2016-10-17 | 2017-02-22 | 深圳市彬讯科技有限公司 | House type design scheme evaluation method based on machine learning |
| CN106878409A (en) * | 2017-02-09 | 2017-06-20 | 深圳市莫二科技有限公司 | A kind of game data processing system and processing method |
| CN107229976A (en) * | 2017-06-08 | 2017-10-03 | 郑州云海信息技术有限公司 | A kind of distributed machines learning system based on spark |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110321112A (en) * | 2019-07-02 | 2019-10-11 | 北京百度网讯科技有限公司 | AI ability research/development platform and data processing method |
| US11693624B2 (en) | 2019-07-02 | 2023-07-04 | Beijing Baidu Netcom Science Technology Co., Ltd. | AI capability research and development platform and data processing method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103530826B (en) | BIM (Building Information Modeling)-based shield method tunnel construction management method | |
| CN104077140B (en) | Automation Compilation Method and compilation device for continuous integrating | |
| CN107390110A (en) | A kind of method, apparatus and system tested automatically PCBA | |
| CN202305756U (en) | Relay service life tester and relay service life test system | |
| CN103019939B (en) | Method and system for testing driver | |
| CN108897240A (en) | Automatic Pilot emulation testing cloud platform and automatic Pilot emulation test method | |
| CN102591780B (en) | Device driver software test method and test system | |
| CN103425584A (en) | Large-scale application regression testing information processing method based on Java bytecode | |
| CN102938713A (en) | 1553B data bus testing simulation system | |
| CN106776294A (en) | Android mobile phone automated testing method and system | |
| CN101275987A (en) | Equipment test system and method | |
| CN108664540A (en) | Big data machine learning system and method | |
| Madhavan et al. | Benchmarking and standardization of intelligent robotic systems | |
| CN110135289A (en) | A kind of underground coal mine intelligent use cloud service platform based on deep learning | |
| CN107819793A (en) | Collecting method and device for robot operating system | |
| CN106844211A (en) | A kind of method for testing software, apparatus and system | |
| CN102097017A (en) | System for testing automatic transmission of automobile | |
| CN110045133A (en) | A kind of consumptive material monitoring device and monitoring method | |
| CN115686988A (en) | RISC-V architecture Debug protocol function verification method and system | |
| CN109592525A (en) | Elevator frequency converter fault diagnosis system and method | |
| CN116089257A (en) | RPA robot software safety penetration test technology based on artificial intelligence algorithm | |
| CN109508204A (en) | A kind of front-end code quality determining method and device | |
| Russo | Finding a way in the model driven jungle: Invited keynote talk | |
| CN208938094U (en) | A kind of universal serial bus test macro | |
| CN108318752A (en) | Detecting system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181016 |