CN106250987A - A kind of machine learning method, device and big data platform - Google Patents
A kind of machine learning method, device and big data platform Download PDFInfo
- Publication number
- CN106250987A CN106250987A CN201610587879.3A CN201610587879A CN106250987A CN 106250987 A CN106250987 A CN 106250987A CN 201610587879 A CN201610587879 A CN 201610587879A CN 106250987 A CN106250987 A CN 106250987A
- Authority
- CN
- China
- Prior art keywords
- machine learning
- data
- data base
- module
- logic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Electrically Operated Instructional Devices (AREA)
Abstract
A kind of machine learning device of disclosure of the invention, a kind of machine learning method based on machine learning device, and using a kind of big data platform of above-mentioned machine learning device and machine learning method thereof, this machine learning device includes: User Defined process module, configuration module, data base；Event server；User Defined process module comprises a logic, and this logic is able to receive that Client-initiated asks the executable file comprised, and is called by event server；Front end exploitation application, by configuring the configuration file that module is write, is bound by data base with described executable file.The component of finishing service logic is carried out by User-defined template, achieve the adaptability to various application scenarios and versatility, the data mining that relates to during achieving data mining big to standardization, the Effec-tive Function of machine learning, simplify the development process of the big data of standardization, improve the development deployment efficiency of the big data of standardization.
Description
Technical field
The present invention relates to big data technique field, particularly relate to a kind of machine learning method, machine learning device, Yi Jiji
A kind of big data platform in this machine learning device.
Background technology
Spark was that the big data calculating that Databricks increases income processes engine, became the top project of Apache in 2010,
Its core calculations is elasticity distribution formula data set (RDD), it is provided that the MapReduce model more enriched than Hadoop, it is possible to
Quickly to data set iterative computation in internal memory, support complicated machine learning algorithm and graph-theoretical algorithm.
Machine learning method of the prior art is as follows.
First, perform step 1) collection of initial data: data producer can generate polytype data, such as log literary composition
Part, view data, text data etc., the quality of data can produce along with some problems of the improper department of user or system
A lot of noise datas, are difficult to avoid the generation of wrong data；It is special that other multi-medium data such as text, image also need to some
Other instrument carries out data loading.
Then, perform step 2) data prediction: from step 1) in the data collected containing a lot of dirty datas, invalid
Data and some multi-medium datas, it is necessary to process through a series of measure, frequently with Hive, MR and
Data are gone dirty process, Missing Data Filling etc. by the preprocess module of Spark；Place for multi-medium data
Reason, has a third-party tool kit for being converted to the data that computer can process, such as OpenCV, Word2vec.
Then, step 3 is performed) Feature Engineering: Feature Engineering mainly includes again the processing of preprocessed data, data
Format, sampling, the conversion of data and the design of feature and selection, often use MR, Spark features module and
Data are processed by third party's instrument of a little specialties, output characteristic data after process, are used for doing the training of model.
Then, step 4 is performed) model training: practical business problem is mainly entered by model training by data method
Row modeling, conventional Spark MLlib, Mahout and third party bag such as sklearn etc. carry out business model to data,
The model trained carries out persistence preservation.
Then, step 5 is performed) model reaches the standard grade: model is reached the standard grade and is mainly said data on the model tie-in line trained, and is
User provide conventional classification, return, the machine learning service such as recommendation.
Finally, framework relies on: whole machine learning techniques scheme based on spark relies on data literary composition big with HDFS, S3 etc.
Part system, and various big data including hadoop, spark process and deployment tool.
As can be seen here, the whole framework in machine learning method based on the big data of spark of the prior art supports and relates to
And wide, business difficulty is big.Therefore, the technical scheme of machine learning algorithm models based on the big data of spark in prior art,
Sufficiently complex, the technological layer of design covers the every aspects such as Distributed Calculation, framework deployment, model calculating, data mining, flower
Take the biggest manpower and materials just can complete.Each flow process is required for carrying out continually operation of exchanging visits with file system, significantly drops
The performance of low whole system, thus the reliability reduction causing modeling, predicting and apply；The more important thing is the spirit that can cause programming
The reusability of activity, ease for maintenance, code or assembly is a greater impact, and therefore causes Consumer's Experience poor.
Summary of the invention
It is an object of the invention to open a kind of machine learning device, a kind of machine learning based on this machine learning device
Method, and use a kind of big data platform of above-mentioned machine learning device and machine learning method thereof, in order to realize standard
The data mining that relates to during changing big data mining, the Effec-tive Function of machine learning, simplify the exploitation of the big data of standardization
Flow process, improves the development deployment efficiency of the big data of standardization, and provides the most unified interface.
For realizing above-mentioned first goal of the invention, the invention provides a kind of machine learning device, comprising:
User Defined process module, configuration module, data base；And
Event server；Wherein,
User Defined process module comprises a logic, this logic be able to receive that Client-initiated request comprised can
Perform file, and called by event server；
Front end exploitation application, by configuring the configuration file that module is write, is carried out by data base with described executable file
Binding.
As a further improvement on the present invention, described User Defined process module includes interface module, service logic mould
Block, service module and performance estimation module.
As a further improvement on the present invention, described business logic modules comprises executable file execution logical operation
At least one rule, described rule includes that machine learning algorithm rule, text data process rule, graphic user interface processes rule
Then.
As a further improvement on the present invention, described performance estimation module is according to the rule included in this business logic modules
The executable file then comprised Client-initiated request obtains machine learning algorithm model, and is specifying according to user's scene
Hyper parameter adjusting and optimizing is carried out, to obtain machine learning algorithm model parameter in model parameter.
As a further improvement on the present invention, described business logic modules comprises data prediction logic, Feature Engineering is patrolled
Volume, model algorithm logic and model reach the standard grade logic；Wherein, described model logic of reaching the standard grade includes RESTfull API and data stock
Storage index.
As a further improvement on the present invention, this machine learning device also includes encrypting module, and it passes through access key
Binding RESTfull API, to bind configuration file with described executable file.
As a further improvement on the present invention, described access key includes Access key or Secret key.
As a further improvement on the present invention, described data base is by creating different tables of data, and during by service request
Between type data base is divided into first data base, the second data base and the 3rd data base；Wherein,
Described first data base, is used for storing metadata；
Described second data base, is used for storing event type, configuration parameter, model training parameter；
Described 3rd data base, for having stored the model of training.
As a further improvement on the present invention, described data base supports that Hbase interactive mode, Elasticsearch are mutual
Pattern or Mysql interactive mode.
As a further improvement on the present invention, event server include import engine, process engine, model training engine and
Service provides engine.
As a further improvement on the present invention, described executable file includes that executable program, computer module, system are inserted
The application of part, visualization interface or computer can perform document.
For realizing above-mentioned second goal of the invention, present invention also offers a kind of machine learning method, comprise the following steps:
S1, the executable file comprised by the reception Client-initiated request of User Defined process module；
S2, executable file is called to event server；
S3, build configuration file according to the environmental variable of user；
S4, in data base content according to configuration file, front end exploitation application is tied up with described executable file
Fixed.
For realizing above-mentioned 3rd goal of the invention, present invention also offers a kind of big data platform, including above-mentioned any one
Machine learning device and at least one platform engine, described platform engine include spark engine, tensorflow engine or
Person's mxnet engine.
Compared with prior art, the invention has the beneficial effects as follows: carry out finishing service logic by User-defined template
Component, it is achieved that adaptability and the versatility to various application scenarios, it is achieved that relate to during data mining big to standardization
The data mining arrived, the Effec-tive Function of machine learning, simplify the development process of the big data of standardization, improve the big number of standardization
According to development deployment efficiency, and the most unified interface can be provided, so that algorithm development, application and development are developed with framework
It is capable of modular operation, the deployment efficiency greatly improving big data platform and the efficiency that data are excavated.
Accompanying drawing explanation
Fig. 1 is the structure chart of the present invention a kind of machine learning device；
Fig. 2 is the structure chart of the event server in the machine learning device in Fig. 1；
Fig. 3 is the present invention a kind of machine learning device structure chart in a kind of variation；
Fig. 4 is the flow chart of a kind of machine learning method of the present invention.
Detailed description of the invention
The present invention is described in detail for each embodiment shown in below in conjunction with the accompanying drawings, but it should explanation, these
Embodiment not limitation of the present invention, those of ordinary skill in the art according to these embodiment institute work energy, method,
Or the equivalent transformation in structure or replacement, within belonging to protection scope of the present invention.
Embodiment one:
Please join a kind of detailed description of the invention of a kind of machine learning device of the present invention shown in Fig. 1 Yu Fig. 2.
In the present embodiment, a kind of machine learning device, comprising: User Defined process module 1, configuration module 4,
Data base 3；And event server 2.User Defined process module 1 comprises a logic, and this logic is able to receive that user sends out
The executable file that the request risen is comprised, and called by event server institute 2.Data base 3 is by configuring what module 4 was write
Configuration file, binds front end exploitation application with described executable file.Concrete, this executable file includes performing
The application of program, computer module, system plugin, visualization interface or computer can perform document.
This User Defined process module 1 includes that interface module 11, business logic modules 12, service module 13 and performance are commented
Estimate module 14.Concrete, shown in ginseng Fig. 2, in the present embodiment, event server 2 includes importing engine 21, processing engine
22, model training engine 23 and service provide engine 24.Import engine 21 to be responsible for disposition data source parameter, pending data are entered
The basic handling such as row read operation/write operation, and support to carry out data interaction between data base 3.Process engine 22 to be responsible for treating
Process data and perform text data process.Model training engine 23, is responsible for being carried out practical business problem by data method
Modeling, commonly uses, with Spark MLlib, Mahout and third party's bag sklearn etc. such as, data is carried out business model,
The model trained carries out persistence preservation, and preserves to the second data base 32.Concrete, model training engine 23 is supported
Line model training is trained with off-line model, thus improves user's convenience when big data are disposed.Service provides engine 24,
It accepts the model in service module 13, and directly provides a user with online service operation by network.
This business logic modules 12 comprises at least one rule that executable file performs logical operation, described rule bag
Include machine learning algorithm rule, text data processes rule, graphic user interface processes rule.In the present embodiment, business
Logic module 12 is by adding above-mentioned four kinds of logics so that whole User Defined process module 1 has possessed centralization and processed industry
The logic of business.
Client-initiated request is comprised by performance estimation module 14 according to the rule included in this business logic modules
Executable file obtain machine learning algorithm model, and in designated model parameter, carry out hyper parameter adjustment according to user's scene
Optimize, to obtain machine learning algorithm model parameter.The rule that performance estimation module 14 can allow user on-demand or set by oneself
Then input rule, and dispose on the line of implementation model.
Business logic modules 12 comprise data prediction logic 121, Feature Engineering logic 122, model algorithm logic 123 and
Model is reached the standard grade logic 124；Wherein, described model reach the standard grade logic 124 include RESTfull API and database purchase index.
RESTfull API, a kind of software architecture interface, it provides one group of design principle and constraints.It is mainly used in client
Software with server interactive class.
Data base 3 is by creating different tables of data, and by service request time type, data base 3 is divided into first
Data base the 31, second data base 32 and the 3rd data base 33, wherein, the first data base 31, it is used for storing metadata；Second data
Storehouse 32, is used for storing event type, configuration parameter, model training parameter；3rd data base 33, for having stored the mould of training
Type.
The executable file generated from User Defined process module 1 can be realized, the most also by RESTfull API
User can be received or data inquiry request that manager is sent, and event server 2 can be preserved in the second data base 32
The model exported or service or application.
User or manager, can be by app, system plugin, program, graphic user interfaces when building big data platform
(GUI), all data that can be readable by a computer such as text data are captured by User Defined process module 1, and form use
Family self-defined template.The configuration file that this User-defined template can be imported with configuration module 4, with front end in data base 3
Web application, app or the service of exploitation realize encapsulation, and are stored in the 3rd data base 33, thus for follow-up service or should
By the big data, services providing integration.The web application of front end exploitation, app or service can pass through JAVA, Python, PHP or
The language such as person Ruby are write and are formed.
Concrete, in the present embodiment, this data base 3 supports the mutual mould of Hbase interactive mode, Elasticsearch
Formula or Mysql interactive mode, and preferably Elasticsearch interactive mode.Elasticsearch be one based on
The search server of Lucene.It provide the full-text search engine of a distributed multi-user ability, based on RESTful web
Interface.Elasticsearch Java develops, and issues as the open source code under Apache license terms, and can be real
Existing distributed full-text search.
In the present embodiment, the component of finishing service logic is carried out by User-defined template 1, it is achieved that answer various
By adaptability and the versatility of scene, it is achieved that the data mining that relates to during data mining big to standardization, engineering
The Effec-tive Function practised, simplifies the development process of the big data of standardization, improves the development deployment efficiency of the big data of standardization, and
The most unified interface can be provided, so that algorithm development, application and development are developed with framework is capable of modular operation,
The deployment efficiency greatly improving big data platform and the efficiency that data are excavated.
Simultaneously, it is also possible to realize the adaptation exploitation of back end business logic, design and set, and pass through to event server 2
Submit to newly-built app to ask, determine the relevant informations such as app ID, app NAME, Access Key.Determine these information complete it
After, directly can carry out structure and the deployment of template in data base 3, to complete the binding of template and app；Then taken by event
The operations such as the template built and deployment is complete is compiled by business device 2 successively, training, thus generate template.Template after generation
Can be bound by Access Key with the various application of front end exploitation, program, the computer executable file such as plug-in unit, it is achieved
Business separates with framework.
In service deployment and development process, business separates with framework, algorithm engineering teacher only need to pay close attention to algorithm logic work,
It is responsible for the development of algorithm logic template；CLP AD only need to participate in app, web development, it is provided that data access
Logic is presented with data；Framework Developmental Engineer only need to pay close attention to framework details, and all business all have event to trigger, and event is by business
Personnel are self-defined, and the configuration file that all model related works are write in data base 3 by configuration template 4 is controlled, whole
Individual big data platform is divided the work clearly, is disposed simply, and the bottom at spark calculates and with the help of big data ecology instrument, can be big
The earth reduces the data storage redundancy in conventional machines learning device, degraded performance, a difficult problem for development process complexity.
Embodiment two:
In conjunction with reference to shown in Fig. 3, the present embodiment differs primarily in that with embodiment one, in the present embodiment, and this machine
Device learning device also include encrypt mould 5 pieces, its by access key bind RESTfull API, with by configuration file with described
Executable file is bound.Preferably, this access key is Access key, it is possible to for Secret key.Front end applications
Interacted with model time server 6 by binding Access Key binding RESTful API service, complete the inquiry of data
Service.
The technical scheme that the present embodiment is identical with embodiment one please be joined described in embodiment one, does not repeats them here.
Embodiment three:
Shown in ginseng Fig. 4, the present embodiment discloses a kind of machine learning method, comprises the following steps:
S1, the executable file comprised by the reception Client-initiated request of User Defined process module；
S2, executable file is called to event server；
S3, build configuration file according to the environmental variable of user；
S4, in data base content according to configuration file, front end exploitation application is tied up with described executable file
Fixed.
Embodiment four:
Present embodiment discloses a kind of big data platform, it includes one or more machine learning device and at least one
Individual platform engine, described platform engine includes spark engine, tensorflow engine or mxnet engine, concrete, and this is put down
Platform engine selects different computing engines according to business demand, such as in scene based on the multimedia data service such as image and video
Under, we provide tensorflow or mxnet engine to do the Computational frame of platform, at business scenario based on structural data
Under, use spark as platform computing engines.
Machine learning device in the present embodiment coordinates with reference to described in this specification embodiment one and/or embodiment two.
Spark is a data processing platform (DPP) increased income, and is made up of one group of storehouse powerful, high level, at present these
Storehouse mainly includes Spark SQL, Spark Streaming, MLlib, GraphX, supports to include Scala, Java, Python, R
In interior API Calls, it is possible to carry out the most integrated with Hadoop ecosystem and data source.
Spark mainly includes structured data query and analysis engine (SparkSQL), distributed machines learning database
(MLlib), parallel figure Computational frame (GraphX), stream calculation framework (Spark Streaming), third party's sub-project are (such as
BlinkDB, Tachyon, Mesos etc.).MLlib is the assembly being responsible for machine learning in Spark, and conventional module includes
Classification、Regression、Clustering、Collaborativefiltering、
Frequentpatternmining and conventional data prediction and Feature Engineering module.MLlib is offer machine under Spark
The module of device learning algorithm, built-in multiple machine learning algorithm.
Mxnet and tensorflow is that the degree of depth learns calculating instrument, is commonly used to build the degree of depth based on multi-medium data
Practise framework, have an advantage in that without engineer's feature, given business demand, build multilayer neural network, changing by magnanimity
In generation, calculates, and carrys out the multi-medium data demand that digging user is interested.Common business scenario include security protection, critical point abnormality detection,
Target recognition etc..
The a series of detailed description of those listed above is only for the feasibility embodiment of the present invention specifically
Bright, they also are not used to limit the scope of the invention, all equivalent implementations made without departing from skill of the present invention spirit
Or change should be included within the scope of the present invention.
It is obvious to a person skilled in the art that the invention is not restricted to the details of above-mentioned one exemplary embodiment, Er Qie
In the case of the spirit or essential attributes of the present invention, it is possible to realize the present invention in other specific forms.Therefore, no matter
From the point of view of which point, all should regard embodiment as exemplary, and be nonrestrictive, the scope of the present invention is by appended power
Profit requires rather than described above limits, it is intended that all by fall in the implication of equivalency and scope of claim
Change is included in the present invention.Should not be considered as limiting involved claim by any reference in claim.
Although moreover, it will be appreciated that this specification is been described by according to embodiment, but the most each embodiment only wraps
Containing an independent technical scheme, this narrating mode of description is only that for clarity sake those skilled in the art should
Description can also be formed those skilled in the art through appropriately combined as an entirety, the technical scheme in each embodiment
May be appreciated other embodiments.
Claims (13)
1. a machine learning device, it is characterised in that comprising:
User Defined process module, configuration module, data base；And
Event server；Wherein,
User Defined process module comprises a logic, and this logic is able to receive that Client-initiated asks comprised to perform
File, and called by event server；
Front end exploitation application, by configuring the configuration file that module is write, is tied up by data base with described executable file
Fixed.
Machine learning device the most according to claim 1, it is characterised in that described User Defined process module includes connecing
Mouth die block, business logic modules, service module and performance estimation module.
Machine learning device the most according to claim 2, it is characterised in that described business logic modules comprises performing
File perform logical operation at least one rule, described rule include machine learning algorithm rule, text data process rule,
Graphic user interface processes rule.
Machine learning device the most according to claim 3, it is characterised in that described performance estimation module is patrolled according to this business
Collect the rule included in module and the executable file that Client-initiated request is comprised is obtained machine learning algorithm model, and
In designated model parameter, hyper parameter adjusting and optimizing is carried out, to obtain machine learning algorithm model parameter according to user's scene.
Machine learning device the most according to claim 1, it is characterised in that described business logic modules comprises data and locates in advance
Reason logic, Feature Engineering logic, model algorithm logic and model are reached the standard grade logic；Wherein, described model logic of reaching the standard grade includes
RESTfull API and database purchase index.
Machine learning device the most according to claim 5, it is characterised in that also include encrypting module, it is closed by access
Key word binding RESTfull API, to bind configuration file with described executable file.
Machine learning device the most according to claim 6, it is characterised in that described access key includes Access key
Or Secret key.
Machine learning device the most according to claim 1, it is characterised in that described data base is by creating different data
Table, and by servicing request time type, data base is divided into first data base, the second data base and the 3rd data base；Its
In,
Described first data base, is used for storing metadata；
Described second data base, is used for storing event type, configuration parameter, model training parameter；
Described 3rd data base, for having stored the model of training.
Machine learning device the most according to claim 1, it is characterised in that described data base support Hbase interactive mode,
Elasticsearch interactive mode or Mysql interactive mode.
Machine learning device the most according to claim 1, it is characterised in that described event server include import engine,
Process engine, model training engine and service and engine is provided.
11. machine learning devices as claimed in any of claims 1 to 10, it is characterised in that described perform literary composition
Part includes that the application of executable program, computer module, system plugin, visualization interface or computer can perform document.
12. 1 kinds of machine learning methods, it is characterised in that comprise the following steps:
S1, the executable file comprised by the reception Client-initiated request of User Defined process module；
S2, executable file is called to event server；
S3, build configuration file according to the environmental variable of user；
S4, in data base content according to configuration file, front end exploitation application is bound with described executable file.
13. 1 kinds of big data platforms, it is characterised in that include the machine learning as described in any one in claim 1 to 10
Device and at least one platform engine, described platform engine includes that spark engine, tensorflow engine or mxnet draw
Hold up.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610587879.3A CN106250987B (en) | 2016-07-22 | 2016-07-22 | A kind of machine learning method, device and big data platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610587879.3A CN106250987B (en) | 2016-07-22 | 2016-07-22 | A kind of machine learning method, device and big data platform |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106250987A true CN106250987A (en) | 2016-12-21 |
| CN106250987B CN106250987B (en) | 2019-03-01 |
Family
ID=57603542
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610587879.3A Active CN106250987B (en) | 2016-07-22 | 2016-07-22 | A kind of machine learning method, device and big data platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106250987B (en) |
Cited By (40)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106821376A (en) * | 2017-03-28 | 2017-06-13 | 南京医科大学 | A kind of epileptic attack early warning system and method based on deep learning algorithm |
| CN106951751A (en) * | 2017-03-24 | 2017-07-14 | 电子科技大学 | A kind of sensor-based smart mobile phone unlocking recognition methods |
| CN107169575A (en) * | 2017-06-27 | 2017-09-15 | 北京天机数测数据科技有限公司 | A kind of modeling and method for visualizing machine learning training pattern |
| CN107562859A (en) * | 2017-08-29 | 2018-01-09 | 武汉斗鱼网络科技有限公司 | A kind of disaggregated model training system and its implementation |
| CN107577708A (en) * | 2017-07-31 | 2018-01-12 | 北京北信源软件股份有限公司 | Class base construction method and system based on SparkMLlib document classifications |
| CN107977397A (en) * | 2017-09-08 | 2018-05-01 | 华瑞新智科技（北京）有限公司 | Internet user's notice index calculation method and system based on deep learning |
| CN108268598A (en) * | 2017-12-18 | 2018-07-10 | 苏州航天系统工程有限公司 | A kind of analysis system and analysis method based on vedio data |
| CN108427709A (en) * | 2018-01-25 | 2018-08-21 | 朗新科技股份有限公司 | A kind of multi-source mass data processing system and method |
| CN108510082A (en) * | 2018-03-27 | 2018-09-07 | 苏宁易购集团股份有限公司 | The method and device that machine learning model is handled |
| CN108628669A (en) * | 2018-04-25 | 2018-10-09 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus of scheduling machine learning algorithm task |
| CN108664540A (en) * | 2018-02-13 | 2018-10-16 | 贵州财经大学 | Big data machine learning system and method |
| CN108710949A (en) * | 2018-04-26 | 2018-10-26 | 第四范式（北京）技术有限公司 | The method and system of template are modeled for creating machine learning |
| CN109086038A (en) * | 2018-07-10 | 2018-12-25 | 千寻位置网络有限公司 | Big data development approach and device, terminal based on Spark |
| CN109284298A (en) * | 2018-11-09 | 2019-01-29 | 上海晏鼠计算机技术股份有限公司 | A kind of contents production system handled based on machine learning and big data |
| CN109299785A (en) * | 2018-09-17 | 2019-02-01 | 浪潮软件集团有限公司 | Method and device for realizing machine learning model |
| CN109389143A (en) * | 2018-06-19 | 2019-02-26 | 北京九章云极科技有限公司 | A kind of Data Analysis Services system and method for automatic modeling |
| CN109408592A (en) * | 2018-10-12 | 2019-03-01 | 北京聚云位智信息科技有限公司 | The Feature Engineering knowledge base and its implementation of AI in a kind of decision type distributed data base system |
| CN109582294A (en) * | 2018-12-28 | 2019-04-05 | 中国科学院电子学研究所苏州研究院 | A kind of Software Architecture Design Method of embedded machine learning system |
| CN109656922A (en) * | 2018-12-19 | 2019-04-19 | 国网北京市电力公司 | Data processing method and device |
| CN109685089A (en) * | 2017-10-18 | 2019-04-26 | 北京京东尚科信息技术有限公司 | The system and method for assessment models performance |
| CN109800277A (en) * | 2018-12-18 | 2019-05-24 | 合肥天源迪科信息技术有限公司 | A kind of machine learning platform and the data model optimization method based on the platform |
| CN110048905A (en) * | 2019-03-26 | 2019-07-23 | 清华大学 | The recognition methods of internet of things equipment communication pattern and device |
| CN110119271A (en) * | 2018-12-19 | 2019-08-13 | 厦门渊亭信息科技有限公司 | A kind of model across machine learning platform defines agreement and adaption system |
| CN110324185A (en) * | 2019-06-28 | 2019-10-11 | 京东数字科技控股有限公司 | Hyper parameter tuning method, apparatus, server, client and medium |
| CN110659261A (en) * | 2019-09-19 | 2020-01-07 | 成都数之联科技有限公司 | Data mining model publishing method, model and model service management method |
| CN110766163A (en) * | 2018-07-10 | 2020-02-07 | 第四范式（北京）技术有限公司 | System for implementing a machine learning process |
| CN110895718A (en) * | 2018-09-07 | 2020-03-20 | 第四范式（北京）技术有限公司 | Method and system for training machine learning model |
| CN110942155A (en) * | 2019-11-29 | 2020-03-31 | 广西电网有限责任公司 | Research method of machine learning engine |
| CN111399853A (en) * | 2020-02-20 | 2020-07-10 | 四川新网银行股份有限公司 | Templated deployment method of machine learning model and custom operator |
| CN111444170A (en) * | 2018-12-28 | 2020-07-24 | 第四范式（北京）技术有限公司 | Automatic machine learning method and device based on predicted business scene |
| CN111681158A (en) * | 2020-08-14 | 2020-09-18 | 支付宝(杭州)信息技术有限公司 | Preprocessing method for executing front-end model |
| CN111832735A (en) * | 2019-04-18 | 2020-10-27 | 第四范式（北京）技术有限公司 | Method and system for performing a machine learning process based on a template |
| CN111913715A (en) * | 2020-07-30 | 2020-11-10 | 上海数策软件股份有限公司 | Micro-service based machine learning automation process management and optimization system and method |
| CN112099848A (en) * | 2020-09-11 | 2020-12-18 | 杭州海康威视数字技术股份有限公司 | Service processing method, device and equipment |
| CN112199896A (en) * | 2020-10-26 | 2021-01-08 | 云中芯半导体技术(苏州)有限公司 | Chip logic comprehensive optimization acceleration method based on machine learning |
| CN112348022A (en) * | 2020-10-28 | 2021-02-09 | 富邦华一银行有限公司 | Free-form document identification method based on deep learning |
| CN112685010A (en) * | 2020-12-21 | 2021-04-20 | 福建新大陆软件工程有限公司 | AI application development method and system |
| CN113129049A (en) * | 2019-12-31 | 2021-07-16 | 上海哔哩哔哩科技有限公司 | File configuration method and system for model training and application |
| CN113971032A (en) * | 2021-12-24 | 2022-01-25 | 百融云创科技股份有限公司 | Full-process automatic deployment method and system of machine learning model for code generation |
| TWI795375B (en) * | 2017-01-06 | 2023-03-11 | 香港商阿里巴巴集團服務有限公司 | Component release and component construction method based on graphical machine learning algorithm platform, graphical machine learning algorithm platform |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110282812A1 (en) * | 2010-05-17 | 2011-11-17 | Microsoft Corporation | Dynamic pattern matching over ordered and disordered data streams |
| CN103838617A (en) * | 2014-02-18 | 2014-06-04 | 河海大学 | Method for constructing data mining platform in big data environment |
| CN105335132A (en) * | 2014-06-13 | 2016-02-17 | 阿里巴巴集团控股有限公司 | Method, apparatus and system for user-defined application function |
| US20160110502A1 (en) * | 2014-10-17 | 2016-04-21 | Betterpath, Inc. | Human and Machine Assisted Data Curation for Producing High Quality Data Sets from Medical Records |
-
2016
- 2016-07-22 CN CN201610587879.3A patent/CN106250987B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110282812A1 (en) * | 2010-05-17 | 2011-11-17 | Microsoft Corporation | Dynamic pattern matching over ordered and disordered data streams |
| CN103838617A (en) * | 2014-02-18 | 2014-06-04 | 河海大学 | Method for constructing data mining platform in big data environment |
| CN105335132A (en) * | 2014-06-13 | 2016-02-17 | 阿里巴巴集团控股有限公司 | Method, apparatus and system for user-defined application function |
| US20160110502A1 (en) * | 2014-10-17 | 2016-04-21 | Betterpath, Inc. | Human and Machine Assisted Data Curation for Producing High Quality Data Sets from Medical Records |
Non-Patent Citations (1)
| Title |
|---|
| 赵薇 等: "基于组件的大数据分析服务平台", 《计算机科学》 * |
Cited By (57)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI795375B (en) * | 2017-01-06 | 2023-03-11 | 香港商阿里巴巴集團服務有限公司 | Component release and component construction method based on graphical machine learning algorithm platform, graphical machine learning algorithm platform |
| CN106951751A (en) * | 2017-03-24 | 2017-07-14 | 电子科技大学 | A kind of sensor-based smart mobile phone unlocking recognition methods |
| CN106821376A (en) * | 2017-03-28 | 2017-06-13 | 南京医科大学 | A kind of epileptic attack early warning system and method based on deep learning algorithm |
| CN106821376B (en) * | 2017-03-28 | 2019-12-06 | 南京医科大学 | epileptic seizure early warning system based on deep learning algorithm |
| CN107169575A (en) * | 2017-06-27 | 2017-09-15 | 北京天机数测数据科技有限公司 | A kind of modeling and method for visualizing machine learning training pattern |
| CN107577708A (en) * | 2017-07-31 | 2018-01-12 | 北京北信源软件股份有限公司 | Class base construction method and system based on SparkMLlib document classifications |
| CN107562859A (en) * | 2017-08-29 | 2018-01-09 | 武汉斗鱼网络科技有限公司 | A kind of disaggregated model training system and its implementation |
| CN107977397A (en) * | 2017-09-08 | 2018-05-01 | 华瑞新智科技（北京）有限公司 | Internet user's notice index calculation method and system based on deep learning |
| CN109685089A (en) * | 2017-10-18 | 2019-04-26 | 北京京东尚科信息技术有限公司 | The system and method for assessment models performance |
| CN109685089B (en) * | 2017-10-18 | 2020-12-22 | 北京京东尚科信息技术有限公司 | System and method for evaluating model performance |
| CN108268598A (en) * | 2017-12-18 | 2018-07-10 | 苏州航天系统工程有限公司 | A kind of analysis system and analysis method based on vedio data |
| CN108427709A (en) * | 2018-01-25 | 2018-08-21 | 朗新科技股份有限公司 | A kind of multi-source mass data processing system and method |
| CN108427709B (en) * | 2018-01-25 | 2020-10-16 | 朗新科技集团股份有限公司 | Multi-source mass data processing system and method |
| CN108664540A (en) * | 2018-02-13 | 2018-10-16 | 贵州财经大学 | Big data machine learning system and method |
| CN108510082B (en) * | 2018-03-27 | 2022-11-11 | 苏宁易购集团股份有限公司 | Method and device for processing machine learning model |
| CN108510082A (en) * | 2018-03-27 | 2018-09-07 | 苏宁易购集团股份有限公司 | The method and device that machine learning model is handled |
| CN108628669A (en) * | 2018-04-25 | 2018-10-09 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus of scheduling machine learning algorithm task |
| CN108710949A (en) * | 2018-04-26 | 2018-10-26 | 第四范式（北京）技术有限公司 | The method and system of template are modeled for creating machine learning |
| CN109389143A (en) * | 2018-06-19 | 2019-02-26 | 北京九章云极科技有限公司 | A kind of Data Analysis Services system and method for automatic modeling |
| CN113935434A (en) * | 2018-06-19 | 2022-01-14 | 北京九章云极科技有限公司 | Data analysis processing system and automatic modeling method |
| CN109086038A (en) * | 2018-07-10 | 2018-12-25 | 千寻位置网络有限公司 | Big data development approach and device, terminal based on Spark |
| CN110766163B (en) * | 2018-07-10 | 2023-08-29 | 第四范式（北京）技术有限公司 | System for implementing machine learning process |
| CN110766163A (en) * | 2018-07-10 | 2020-02-07 | 第四范式（北京）技术有限公司 | System for implementing a machine learning process |
| CN110895718A (en) * | 2018-09-07 | 2020-03-20 | 第四范式（北京）技术有限公司 | Method and system for training machine learning model |
| CN109299785B (en) * | 2018-09-17 | 2022-04-26 | 浪潮软件股份有限公司 | Method and device for realizing machine learning model |
| CN109299785A (en) * | 2018-09-17 | 2019-02-01 | 浪潮软件集团有限公司 | Method and device for realizing machine learning model |
| CN109408592B (en) * | 2018-10-12 | 2021-09-24 | 北京聚云位智信息科技有限公司 | AI characteristic engineering knowledge base in decision-making type distributed database system and implementation method thereof |
| CN109408592A (en) * | 2018-10-12 | 2019-03-01 | 北京聚云位智信息科技有限公司 | The Feature Engineering knowledge base and its implementation of AI in a kind of decision type distributed data base system |
| CN109284298A (en) * | 2018-11-09 | 2019-01-29 | 上海晏鼠计算机技术股份有限公司 | A kind of contents production system handled based on machine learning and big data |
| CN109800277A (en) * | 2018-12-18 | 2019-05-24 | 合肥天源迪科信息技术有限公司 | A kind of machine learning platform and the data model optimization method based on the platform |
| CN109656922A (en) * | 2018-12-19 | 2019-04-19 | 国网北京市电力公司 | Data processing method and device |
| CN110119271A (en) * | 2018-12-19 | 2019-08-13 | 厦门渊亭信息科技有限公司 | A kind of model across machine learning platform defines agreement and adaption system |
| CN111444170A (en) * | 2018-12-28 | 2020-07-24 | 第四范式（北京）技术有限公司 | Automatic machine learning method and device based on predicted business scene |
| CN111444170B (en) * | 2018-12-28 | 2023-10-03 | 第四范式（北京）技术有限公司 | Automatic machine learning method and equipment based on predictive business scene |
| CN109582294A (en) * | 2018-12-28 | 2019-04-05 | 中国科学院电子学研究所苏州研究院 | A kind of Software Architecture Design Method of embedded machine learning system |
| CN109582294B (en) * | 2018-12-28 | 2022-02-22 | 中国科学院电子学研究所苏州研究院 | Software architecture design method of embedded machine learning system |
| CN110048905B (en) * | 2019-03-26 | 2021-01-15 | 清华大学 | Internet of things equipment communication mode identification method and device |
| CN110048905A (en) * | 2019-03-26 | 2019-07-23 | 清华大学 | The recognition methods of internet of things equipment communication pattern and device |
| CN111832735A (en) * | 2019-04-18 | 2020-10-27 | 第四范式（北京）技术有限公司 | Method and system for performing a machine learning process based on a template |
| CN110324185A (en) * | 2019-06-28 | 2019-10-11 | 京东数字科技控股有限公司 | Hyper parameter tuning method, apparatus, server, client and medium |
| CN110324185B (en) * | 2019-06-28 | 2022-12-27 | 京东科技控股股份有限公司 | Hyper-parameter tuning method, device, server, client and medium |
| CN110659261A (en) * | 2019-09-19 | 2020-01-07 | 成都数之联科技有限公司 | Data mining model publishing method, model and model service management method |
| CN110942155A (en) * | 2019-11-29 | 2020-03-31 | 广西电网有限责任公司 | Research method of machine learning engine |
| CN113129049A (en) * | 2019-12-31 | 2021-07-16 | 上海哔哩哔哩科技有限公司 | File configuration method and system for model training and application |
| CN113129049B (en) * | 2019-12-31 | 2023-07-28 | 上海哔哩哔哩科技有限公司 | File configuration method and system for model training and application |
| CN111399853A (en) * | 2020-02-20 | 2020-07-10 | 四川新网银行股份有限公司 | Templated deployment method of machine learning model and custom operator |
| CN111399853B (en) * | 2020-02-20 | 2023-06-06 | 四川新网银行股份有限公司 | Templated deployment method for machine learning model and custom operator |
| CN111913715A (en) * | 2020-07-30 | 2020-11-10 | 上海数策软件股份有限公司 | Micro-service based machine learning automation process management and optimization system and method |
| CN111681158A (en) * | 2020-08-14 | 2020-09-18 | 支付宝(杭州)信息技术有限公司 | Preprocessing method for executing front-end model |
| CN112099848A (en) * | 2020-09-11 | 2020-12-18 | 杭州海康威视数字技术股份有限公司 | Service processing method, device and equipment |
| CN112099848B (en) * | 2020-09-11 | 2024-03-05 | 杭州海康威视数字技术股份有限公司 | Service processing method, device and equipment |
| CN112199896A (en) * | 2020-10-26 | 2021-01-08 | 云中芯半导体技术(苏州)有限公司 | Chip logic comprehensive optimization acceleration method based on machine learning |
| CN112348022A (en) * | 2020-10-28 | 2021-02-09 | 富邦华一银行有限公司 | Free-form document identification method based on deep learning |
| CN112348022B (en) * | 2020-10-28 | 2024-05-07 | 富邦华一银行有限公司 | Free-form document identification method based on deep learning |
| CN112685010B (en) * | 2020-12-21 | 2022-06-07 | 福建新大陆软件工程有限公司 | AI application development method and system |
| CN112685010A (en) * | 2020-12-21 | 2021-04-20 | 福建新大陆软件工程有限公司 | AI application development method and system |
| CN113971032A (en) * | 2021-12-24 | 2022-01-25 | 百融云创科技股份有限公司 | Full-process automatic deployment method and system of machine learning model for code generation |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106250987B (en) | 2019-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106250987A (en) | A kind of machine learning method, device and big data platform | |
| US10762422B2 (en) | Wide and deep machine learning models | |
| Vedaldi et al. | Matconvnet: Convolutional neural networks for matlab | |
| Bures et al. | Internet of things: Current challenges in the quality assurance and testing methods | |
| CN104298496B (en) | data analysis type software development framework system | |
| Falah et al. | Design of virtual engineering and digital twin platform as implementation of cyber-physical systems | |
| CN108121742A (en) | The generation method and device of user's disaggregated model | |
| CN110489630A (en) | Processing method, device, computer equipment and the storage medium of resource data | |
| CN109947811A (en) | Generic features library generating method and device, storage medium, electronic equipment | |
| CN110348109A (en) | The method and terminal device of three-dimensional artificial training data processing | |
| CN109074378A (en) | Modular electrical subdata analytical calculation system | |
| CN116245670A (en) | Method, device, medium and equipment for processing financial tax data based on double-label model | |
| Ejaz | Implementation of industry 4.0 enabling technologies from smart manufacturing perspective | |
| Zehe et al. | Tutorial on a modeling and simulation cloud service | |
| Sunkle et al. | Intentional Modeling for Problem Solving in Enterprise Architecture. | |
| Gierej | Big data in the industry-overview of selected issues | |
| CN109710890B (en) | Method and system for identifying false material in real time based on constructed behavior portrait model | |
| Schina et al. | Virtual reality for product development in manufacturing industries | |
| CN114417161B (en) | Virtual article time sequence recommendation method, device, medium and equipment based on special-purpose map | |
| Kappel et al. | Internet of production: entering phase two of industry 4.0 | |
| CN109120509A (en) | A kind of method and device that information is collected | |
| Rohrer et al. | Predictive Object-Centric Process Monitoring | |
| CN106998350A (en) | The method and system of framework are used based on the function items across user message | |
| Flügel et al. | Development and implementation of an integrated water resources management system (IWRMS) | |
| CN109214474A (en) | Behavioural analysis, information coding risk analysis method and device based on information coding |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 214000, science and software park, Binhu District, Jiangsu, Wuxi 6 Patentee after: Huayun data holding group Co.,Ltd. Address before: 214000, science and software park, Binhu District, Jiangsu, Wuxi 6 Patentee before: WUXI CHINAC DATA TECHNICAL SERVICE Co.,Ltd. |
|
| CP01 | Change in the name or title of a patent holder | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20221102 Address after: Room 316, Government Affairs Service Center, No. 1, Renmin Road, Pingshang Town, Lingang Economic Development Zone, Linyi City, Shandong Province, 276000 Patentee after: Huayun Industrial Internet Co.,Ltd. Address before: No. 6 Science and Education Software Park, Binhu District, Wuxi City, Jiangsu Province Patentee before: Huayun data holding group Co.,Ltd. |
|
| TR01 | Transfer of patent right |