CN103631953A - Large data analysis method and large data analysis terminal based on internal error checking - Google Patents

Large data analysis method and large data analysis terminal based on internal error checking Download PDF

Info

Publication number
CN103631953A
CN103631953A CN201310683366.9A CN201310683366A CN103631953A CN 103631953 A CN103631953 A CN 103631953A CN 201310683366 A CN201310683366 A CN 201310683366A CN 103631953 A CN103631953 A CN 103631953A
Authority
CN
China
Prior art keywords
data
error
analytics server
input data
investigating
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201310683366.9A
Other languages
Chinese (zh)
Inventor
黄彤元
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
DONGGUAN FOOCAA NETWORK TECHNOLOGY Co Ltd
Original Assignee
DONGGUAN FOOCAA NETWORK TECHNOLOGY Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by DONGGUAN FOOCAA NETWORK TECHNOLOGY Co Ltd filed Critical DONGGUAN FOOCAA NETWORK TECHNOLOGY Co Ltd
Priority to CN201310683366.9A priority Critical patent/CN103631953A/en
Publication of CN103631953A publication Critical patent/CN103631953A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention discloses a large data analysis method based on internal error checking. The method comprises the following steps: receiving a starting instruction from a data analysis server; performing internal error checking on input data from a data platform according to the starting instruction; selectively transmitting the input data or a sampling time label of the input data to the data analysis server according to a checking result. Compared with the prior art, the method has the characteristics that the input data from the data platform is subjected to internal error checking according to the starting instruction of the data analysis server and is selectively transmitted to the data analysis server according to the checking result, so that the analysis calculation quantity of the data analysis server to large data is reduced and the accuracy of analysis results is improved. Meanwhile, the invention discloses a large data analysis terminal based on internal error checking.

Description

Large data analytical approach and terminal thereof based on investigating in error
Technical field
The present invention relates to large data analysis and applied technical field, relate more specifically to a kind of large data analytical approach and terminal thereof based on investigating in error.
Background technology
At present, the analytic product of large data mostly adopts raw data Direct Sampling, afterwards raw data is sent to data analytics server and carries out analyzing and processing.In raw data, have a large amount of repetitions, tediously long data, and the data of these repetitions are often contributed not quite to analysis result, or even do not had resultful operation.Therefore, directly raw data is sent to data analytics server, because data are huge, huge computational burden will be brought to data analytics server, and along with the increase day by day of data, often cause hardware device to catch up with, when serious, can directly have influence on precision of analysis.In addition, directly raw data is sent to data analytics server, raw data is not carried out to any error investigation, understand equally the accuracy of impact analysis result.
Therefore, be badly in need of a kind of improved large data analytical approach and terminal thereof based on investigating in error and overcome above-mentioned defect.
Summary of the invention
The object of this invention is to provide a kind of large data analytical approach based on investigating in error, to reduce the analytic operation amount of data analytics server to large data, improve precision of analysis.
Another object of the present invention is to provide a kind of large data analysing terminal based on investigating in error, to reduce the analytic operation amount of data analytics server to large data, improves precision of analysis.
For achieving the above object, the invention provides a kind of large data analytical approach based on investigating in error, comprising:
Reception is from the enabled instruction of data analytics server;
According to described enabled instruction, the input data from data platform are carried out investigating in error;
According to investigation result, optionally the sampling time label of described input data or described input data is sent to described data analytics server.
Compared with prior art, method of the present invention carries out investigating in error to the input data from data platform according to the enabled instruction of data analytics server, and optionally input data are sent to data analytics server according to investigation result, thereby reduced the analytic operation amount of data analytics server to large data, improved precision of analysis.
Preferably, the enabled instruction receiving from data analytics server also comprises before:
Reception is from the input data of described data platform.
Preferably, the input data that receive from described data also comprise afterwards:
Reception is from error judgment value and the data model of described data analytics server.
Particularly, according to investigation result, optionally the sampling time label of described input data or described input data being sent to described data analytics server specifically comprises:
Judge whether described investigation result is greater than described error judgment value;
When described investigation result is greater than described error judgment value, described input data are sent to described data analytics server;
When described investigation result is less than or equal to described error judgment value, the sampling time label of described input data is sent to described data analytics server.
Preferably, according to investigation result, optionally described input data being sent to described data analytics server also comprises afterwards:
Described data analytics server is according to error judgment value and data model described in described input Data Update, and described error judgment value after renewal and error model are outwards transmitted.
Correspondingly, the present invention also provides a kind of large data analysing terminal based on investigating in error, comprising:
Instruction module, for receiving the enabled instruction from data analytics server;
Investigation module, for carrying out investigating in error to the input data from data platform according to described enabled instruction;
Output module, for being optionally sent to described data analytics server by the sampling time label of described input data or described input data according to investigation result.
Preferably, this terminal also comprises:
Load module, for receiving error judgment value and the data model from the input data of described data platform, described data analytics server.
Preferably, this terminal also comprises:
Judge module, for judging whether investigation result is greater than described error judgment value.
Preferably, this terminal also comprises:
Sampling module, for extract the sampling time label of the input data of the described load module of input according to the investigation result of described investigation module, and is sent to described output module by described sampling time label.
By following description also by reference to the accompanying drawings, it is more clear that the present invention will become, and these accompanying drawings are used for explaining embodiments of the invention.
Accompanying drawing explanation
Fig. 1 is the process flow diagram that the present invention is based on large data analytical approach one embodiment of investigation in error.
Structured flowchart when Fig. 2 applies for the large data analysing terminal based on investigating in error.
Fig. 3 is the theory diagram of Fig. 2.
Embodiment
With reference now to accompanying drawing, describe embodiments of the invention, in accompanying drawing, similarly element numbers represents similar element
Please refer to Fig. 1, the large data analytical approach that the present invention is based on investigation in error comprises the following steps:
S101, receives the input data P from data platform i(data 1, data 2data n);
S102, receives error judgment value and data model from data analytics server; Particularly, data analytics server completes error judgment value e and data model [e, P according to the data that receive before s(data 1, data 2data n)] structure;
S103, receives the enabled instruction Command[start from data analytics server]; It should be noted that data analytics server can according to before the Data Update error judgment value and the data model that receive, when error judgment value and data model upgrade to some extent, data analytics server will outwards send update instruction Command[update];
S104, carries out investigating in error computing Delta[(P according to enabled instruction to input data s(data 1, data 2data n), P i(data 1, data 2data n)];
S105, whether judgement investigation result is greater than error judgment value, if so, carries out S106, otherwise, carry out S107;
S106, is sent to data analytics server by input data;
S107, is sent to data analytics server by the sampling time label of input data, or does not export any data.
Correspondingly, please refer to Fig. 2 and Fig. 3, the present invention also provides a kind of large data analysing terminal based on investigating in error, comprising:
Load module 10, for receiving the input data from data platform, error judgment value and the data model of data analytics server;
Instruction module 11, for receiving the enabled instruction from data analytics server; It should be noted that data analytics server can according to before the Data Update error judgment value and the data model that receive, when error judgment value and data model upgrade to some extent, instruction module receives the update instruction of data analytics server;
Investigation module 12, for carrying out investigating in error to input data according to enabled instruction;
Judge module 13, for judging whether investigation result is greater than error judgment value;
Sampling module 14, for extracting the sampling time label of input data, and transmits sampling time label;
Output module 15, for being optionally sent to data analytics server by the sampling time label of input data or input data according to investigation result.
Particularly, when judge module is judged investigation result while being greater than error judgment value, indication output is touched input data is sent to data analytics server, otherwise, indicate sampling module to extract the sampling time label of input data, and sampling time label is sent to output module, and by output module, sampling time label is sent to data analytics server.
As can be seen from the above description, the present invention is based on large data analytical approach and the terminal thereof of investigation in error, there is following beneficial effect:
(1) particularly for the higher plateform system of repeated data, because the method and terminal carry out investigating in error to the input data from data platform according to the enabled instruction of data analytics server, and optionally input data are sent to data analytics server according to investigation result, thereby reduced the analytic operation amount of data analytics server to large data, improved precision of analysis;
(2) data analytics server is according to input Data Update error judgment value and data model, and error judgment value after renewal and error model are outwards transmitted, be dynamically alignment error value e and data model Ps of data analytics server, so can dynamically adjust operand, greatly strengthen dirigibility and the conformability of data analytics server (system) self.
(3) strengthened the trace ability of historical data, when applications client whenever necessary, can provide strong data history to case study, investigation operating personnel.
(4) for the data of each different platform, the present invention can adopt for the personality data of these platforms different error computational algorithms (Delta), has effectively improved algorithm dirigibility and diversity for whole huge data analysis.
Invention has been described for above combination most preferred embodiment, but the present invention is not limited to the embodiment of above announcement, and should contain the various modifications of carrying out according to essence of the present invention, equivalent combinations.

Claims (9)

1. the large data analytical approach based on investigating in error, is characterized in that, comprising:
Reception is from the enabled instruction of data analytics server;
According to described enabled instruction, the input data from data platform are carried out investigating in error;
According to investigation result, optionally the sampling time label of described input data or described input data is sent to described data analytics server.
2. the large data analytical approach based on investigating in error as claimed in claim 1, is characterized in that, the enabled instruction receiving from data analytics server also comprises before:
Reception is from the input data of described data platform.
3. the large data analytical approach based on investigating in error as claimed in claim 2, is characterized in that, the input data that receive from described data also comprise afterwards:
Reception is from error judgment value and the data model of described data analytics server.
4. the large data analytical approach based on investigating in error as claimed in claim 3, is characterized in that, optionally the sampling time label of described input data or described input data is sent to described data analytics server specifically comprises according to investigation result:
Judge whether described investigation result is greater than described error judgment value;
When described investigation result is greater than described error judgment value, described input data are sent to described data analytics server;
When described investigation result is less than or equal to described error judgment value, the sampling time label of described input data is sent to described data analytics server.
5. the large data analytical approach based on investigating in error as described in claim 3 or 4, is characterized in that, optionally described input data is sent to described data analytics server also comprises afterwards according to investigation result:
Described data analytics server is according to error judgment value and data model described in described input Data Update, and described error judgment value after renewal and error model are outwards transmitted.
6. the large data analysing terminal based on investigating in error, is characterized in that, comprising:
Instruction module, for receiving the enabled instruction from data analytics server;
Investigation module, for carrying out investigating in error to the input data from data platform according to described enabled instruction;
Output module, for being optionally sent to described data analytics server by the sampling time label of described input data or described input data according to investigation result.
7. the large data analysing terminal based on investigating in error as claimed in claim 6, is characterized in that, also comprises:
Load module, for receiving error judgment value and the data model from the input data of described data platform, described data analytics server.
8. the large data analysing terminal based on investigating in error as claimed in claim 7, is characterized in that, also comprises:
Judge module, for judging whether investigation result is greater than described error judgment value.
9. the large data analysing terminal based on investigating in error as claimed in claim 8, is characterized in that, also comprises:
Sampling module, for extract the sampling time label of the input data of the described load module of input according to the investigation result of described judge module, and is sent to described output module by described sampling time label.
CN201310683366.9A 2013-12-13 2013-12-13 Large data analysis method and large data analysis terminal based on internal error checking Pending CN103631953A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201310683366.9A CN103631953A (en) 2013-12-13 2013-12-13 Large data analysis method and large data analysis terminal based on internal error checking

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201310683366.9A CN103631953A (en) 2013-12-13 2013-12-13 Large data analysis method and large data analysis terminal based on internal error checking

Publications (1)

Publication Number Publication Date
CN103631953A true CN103631953A (en) 2014-03-12

Family

ID=50212994

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201310683366.9A Pending CN103631953A (en) 2013-12-13 2013-12-13 Large data analysis method and large data analysis terminal based on internal error checking

Country Status (1)

Country Link
CN (1) CN103631953A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017013518A1 (en) * 2015-07-23 2017-01-26 International Business Machines Corporation Identifying errors in medical data
CN106446696A (en) * 2015-08-10 2017-02-22 联想(北京)有限公司 Information processing method and electronic device

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004075011A2 (en) * 2003-02-14 2004-09-02 Test Advantage, Inc. Methods and apparatus for data analysis
US20090031018A1 (en) * 2000-02-22 2009-01-29 Smartsignal Corporation Web based fault detection architecture
CN102044135A (en) * 2009-10-22 2011-05-04 宝钢集团上海梅山有限公司 Method for acquiring data of underground mine
CN102184267A (en) * 2011-04-14 2011-09-14 上海同岩土木工程科技有限公司 Abnormal data filtration method for interference elimination of automatic data acquisition system
CN102810251A (en) * 2012-08-01 2012-12-05 重庆大学 Simplified road net model orientated real-time road condition information acquisition system on basis of GPS (global positioning system) terminal

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090031018A1 (en) * 2000-02-22 2009-01-29 Smartsignal Corporation Web based fault detection architecture
WO2004075011A2 (en) * 2003-02-14 2004-09-02 Test Advantage, Inc. Methods and apparatus for data analysis
CN102044135A (en) * 2009-10-22 2011-05-04 宝钢集团上海梅山有限公司 Method for acquiring data of underground mine
CN102184267A (en) * 2011-04-14 2011-09-14 上海同岩土木工程科技有限公司 Abnormal data filtration method for interference elimination of automatic data acquisition system
CN102810251A (en) * 2012-08-01 2012-12-05 重庆大学 Simplified road net model orientated real-time road condition information acquisition system on basis of GPS (global positioning system) terminal

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017013518A1 (en) * 2015-07-23 2017-01-26 International Business Machines Corporation Identifying errors in medical data
US9754076B2 (en) 2015-07-23 2017-09-05 International Business Machines Corporation Identifying errors in medical data
US9858385B2 (en) 2015-07-23 2018-01-02 International Business Machines Corporation Identifying errors in medical data
GB2559055A (en) * 2015-07-23 2018-07-25 Ibm Identifying errors in medical data
CN106446696A (en) * 2015-08-10 2017-02-22 联想(北京)有限公司 Information processing method and electronic device
CN106446696B (en) * 2015-08-10 2020-06-23 联想(北京)有限公司 Information processing method and electronic equipment

Similar Documents

Publication Publication Date Title
US11163991B2 (en) Method and apparatus for detecting body
KR101752251B1 (en) Method and device for identificating a file
CN109743311B (en) WebShell detection method, device and storage medium
EP3026519A1 (en) A processor-implemented method for determining an expected overall performance value of a substation automation system
WO2013102192A3 (en) Real-time dynamic data validation apparatus, system, program code, computer readable medium, and methods for intelligent fields
CN106815524B (en) Malicious script file detection method and device
US20180212986A1 (en) Network attack detection method and device
US10635520B2 (en) Monitoring method and monitoring device of deep learning processor
WO2012100079A3 (en) Apparatus and method for enhancing security of data on a host computing device and a peripheral device
CN113326767A (en) Video recognition model training method, device, equipment and storage medium
US11410062B2 (en) Method and system for reducing risk values discrepancies between categories
WO2014182787A3 (en) Systems and methods for high fidelity multi-modal out-of-band biometric authentication
WO2018031097A1 (en) Systems and methods for multi-instance learning-based classification for streaming inputs
CN103631953A (en) Large data analysis method and large data analysis terminal based on internal error checking
CN113641804A (en) Pre-training model obtaining method and device, electronic equipment and storage medium
CN113204695A (en) Website identification method and device
CN111159009A (en) Pressure testing method and device for log service system
CN107037262B (en) Big data spectrum analysis system and method thereof
CN108312179B (en) Elastic part testing method and device based on mechanical arm and mechanical arm
CN114371863A (en) Data processing method and device, electronic equipment and computer readable storage medium
CN114078274A (en) Face image detection method and device, electronic equipment and storage medium
WO2009158406A8 (en) Methods and systems for increasing protein food safety
US10599845B2 (en) Malicious code deactivating apparatus and method of operating the same
JPWO2020105179A1 (en) Information processing equipment, control methods, and programs
CN114218069B (en) Regression testing method, regression testing device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
CB02 Change of applicant information
CB02 Change of applicant information

Address after: 523808 Guangdong province Dongguan Songshan Lake high tech Industrial Development Zone No. 9 Keyuan pine building 203, Room 204

Applicant after: Dongguan Foocaa Information Technology Co., Ltd

Address before: 523808 Guangdong province Dongguan Songshan Lake high tech Industrial Development Zone No. 9 Keyuan pine building 203, Room 204

Applicant before: Dongguan Foocaa Network Technology Co., Ltd.

COR Change of bibliographic data

Free format text: CORRECT: APPLICANT; FROM: DONGGUAN FOOCAA NETWORK TECHNOLOGY CO., LTD. TO: DONGGUAN FOOCAA INFORMATION TECHNOLOGY CO., LTD.

C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20140312