CN111352929A - Data processing method - Google Patents
Data processing method Download PDFInfo
- Publication number
- CN111352929A CN111352929A CN202010287005.2A CN202010287005A CN111352929A CN 111352929 A CN111352929 A CN 111352929A CN 202010287005 A CN202010287005 A CN 202010287005A CN 111352929 A CN111352929 A CN 111352929A
- Authority
- CN
- China
- Prior art keywords
- data
- processing method
- data processing
- cloud database
- channels
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 19
- 230000007547 defect Effects 0.000 claims description 2
- 238000000034 method Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1464—Management of the backup or restore process for networked environments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The invention provides a data processing method, which comprises the following steps: s1: splitting a plurality of plaintext data packets into X channels of N data M through a data separator, wherein X, N, M is a positive integer; s2: merging the data M of the X channels N, classifying all the data according to the channels and storing the data into a primary cloud database; s3: extracting data in a certain time period of a certain channel from the primary cloud database, and judging the validity of the data; s4: if the data in the step S3 is valid, light curtain validity judgment, fault judgment and hidden danger judgment are carried out according to the data; if the data in the step S3 is invalid, discarding the data; s5: and storing the light curtain effectiveness judgment result, the fault judgment result and the hidden danger judgment result into a secondary cloud database.
Description
Technical Field
The invention relates to the technical field of communication, in particular to a data processing method.
Background
At present, many application fields such as elevator operation abnormity, rocket launching, satellite attitude control and the like need to acquire data based on high frequency, and generation equipment of the high frequency data is often dispersed in a wide physical space and even places which cannot be reached by personnel, so that the data cannot be processed in a traditional mode that a specially-assigned person acquires the data and then stores the data in a local host database.
Since the devices generating high frequency data are distributed in different places and in different environments, erroneous data and lost data are often caused, resulting in abnormal final data, requiring relatively high calculation power and being difficult to scale.
In addition, the number of devices (hereinafter referred to as data sources) generating data is often huge, and the amount of data generated by each data source is also extremely huge, so that the traditional server mode cannot meet the requirement, or the cluster processing cost is high.
Disclosure of Invention
In order to achieve the purpose, the technical scheme adopted by the invention is as follows: a method of data processing, comprising:
s1: splitting a plurality of plaintext data packets into X channels of N data M through a data separator, wherein X, N, M is a positive integer;
s2: merging the data M of the X channels N, classifying all the data according to the channels and storing the data into a primary cloud database;
s3: extracting data in a certain time period of a certain channel from the primary cloud database, and judging the validity of the data;
s4: if the data in the step S3 is valid, light curtain validity judgment, fault judgment and hidden danger judgment are carried out according to the data; if the data in the step S3 is invalid, discarding the data;
s5: and storing the light curtain effectiveness judgment result, the fault judgment result and the hidden danger judgment result into a secondary cloud database.
Further, in the data processing method, the method further includes the steps of: and storing the data of the primary cloud database in a local database.
Further, in the data processing method, the method further includes the steps of: and displaying the data in the secondary cloud database through a display platform.
Further, in the data processing method, in step S3, the time period is at most one month.
Further, in the data processing method, in step S4, the failure determination includes: the elevator door has the defects of door opening failure when arriving at a station, people and objects stuck in the elevator door, door opening failure during operation and failure of normal response of a key.
Further, in the data processing method, when it is determined that a failure occurs, an alarm is issued to warn.
Further, in the data processing method, in step S4, a hidden danger determination is performed according to a preset algorithm, and an impending failure is determined in advance before the failure occurs.
Compared with the prior art, the invention has the advantages that: the invention acquires the data in the high-frequency data source and packs the data, so that the original relatively frequent server communication is changed into relatively low-frequency inter-server communication.
Drawings
Fig. 1 is a flow chart of a data processing method of the present invention.
Detailed Description
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following description will be made with reference to the accompanying drawings. It is obvious that the drawings in the following description are only some examples of the invention, and that for a person skilled in the art, other drawings and embodiments can be derived from them without inventive effort.
For the sake of simplicity, the drawings only schematically show the parts relevant to the present invention, and they do not represent the actual structure as a product. In addition, in order to make the drawings concise and understandable, components having the same structure or function in some of the drawings are only schematically illustrated or only labeled. In this document, "one" means not only "only one" but also a case of "more than one".
As shown in fig. 1, the present invention provides a data processing method, including:
s1: splitting a plurality of plaintext data packets into X channels of N data M through a data separator, wherein X, N, M is a positive integer;
s2: merging the data M of the X channels N, classifying all the data according to the channels and storing the data into a primary cloud database; and meanwhile, the data of the primary cloud database is stored in the local database, so that if the data of the cloud database is lost, backup still exists.
S3: extracting data in a certain time period of a certain channel from the primary cloud database, and judging the validity of the data; in this embodiment, the duration of the time period is at most selected to be one month.
S4: if the data in the step S3 is valid, light curtain validity judgment, fault judgment and hidden danger judgment are carried out according to the data; if the data in the step S3 is invalid, discarding the data;
the failure determination includes: when the fault occurs, the alarm is sent out to warn.
Further, hidden danger judgment is carried out according to a preset algorithm, and the impending fault is judged in advance before the fault does not occur.
S5: and storing the light curtain effectiveness judgment result, the fault judgment result and the hidden danger judgment result into a secondary cloud database, and displaying the data in the secondary cloud database through a display platform, so that the trend of the data can be more visually seen.
The above description is only a preferred embodiment of the present invention, and does not limit the present invention in any way. It will be understood by those skilled in the art that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (7)
1. A data processing method, comprising:
s1: splitting a plurality of plaintext data packets into X channels of N data M through a data separator, wherein X, N, M is a positive integer;
s2: merging the data M of the X channels N, classifying all the data according to the channels and storing the data into a primary cloud database;
s3: extracting data in a certain time period of a certain channel from the primary cloud database, and judging the validity of the data;
s4: if the data in the step S3 is valid, light curtain validity judgment, fault judgment and hidden danger judgment are carried out according to the data; if the data in the step S3 is invalid, discarding the data;
s5: and storing the light curtain effectiveness judgment result, the fault judgment result and the hidden danger judgment result into a secondary cloud database.
2. The data processing method of claim 1, further comprising the steps of: and storing the data of the primary cloud database in a local database.
3. The data processing method of claim 1, further comprising the steps of: and displaying the data in the secondary cloud database through a display platform.
4. The data processing method of claim 1, wherein in step S3, the period of time is at most one month in duration.
5. The data processing method according to claim 1, wherein in step S4, the failure determination includes: the elevator door has the defects of door opening failure when arriving at a station, people and objects stuck in the elevator door, door opening failure during operation and failure of normal response of a key.
6. The data processing method of claim 5, wherein an alarm is issued to warn when a failure is determined.
7. The data processing method of claim 1, wherein in step S4, a hidden danger determination is made according to a preset algorithm, and an impending failure is predicted before it occurs.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010287005.2A CN111352929A (en) | 2020-04-13 | 2020-04-13 | Data processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010287005.2A CN111352929A (en) | 2020-04-13 | 2020-04-13 | Data processing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111352929A true CN111352929A (en) | 2020-06-30 |
Family
ID=71196454
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010287005.2A Pending CN111352929A (en) | 2020-04-13 | 2020-04-13 | Data processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111352929A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2013118489A (en) * | 2011-12-02 | 2013-06-13 | Nippon Telegr & Teleph Corp <Ntt> | Network failure detection method and network device |
| CN106452881A (en) * | 2016-10-21 | 2017-02-22 | 用友网络科技股份有限公司 | Operation and maintenance data processing system and method based on cloud + terminal mode |
| CN106790367A (en) * | 2016-11-15 | 2017-05-31 | 山东省科学院自动化研究所 | The vehicle safety hidden danger early warning of big data treatment and accident reproduction system and method |
| CN109753499A (en) * | 2018-12-17 | 2019-05-14 | 云南电网有限责任公司信息中心 | A kind of O&M monitoring data administering method |
| CN110333962A (en) * | 2019-05-16 | 2019-10-15 | 上海精密计量测试研究所 | A kind of electronic component fault diagnosis model based on data analysis prediction |
| CN110765236A (en) * | 2019-10-09 | 2020-02-07 | 中国人民解放军国防科技大学 | Preprocessing method and system for unstructured mass data |
-
2020
- 2020-04-13 CN CN202010287005.2A patent/CN111352929A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2013118489A (en) * | 2011-12-02 | 2013-06-13 | Nippon Telegr & Teleph Corp <Ntt> | Network failure detection method and network device |
| CN106452881A (en) * | 2016-10-21 | 2017-02-22 | 用友网络科技股份有限公司 | Operation and maintenance data processing system and method based on cloud + terminal mode |
| CN106790367A (en) * | 2016-11-15 | 2017-05-31 | 山东省科学院自动化研究所 | The vehicle safety hidden danger early warning of big data treatment and accident reproduction system and method |
| CN109753499A (en) * | 2018-12-17 | 2019-05-14 | 云南电网有限责任公司信息中心 | A kind of O&M monitoring data administering method |
| CN110333962A (en) * | 2019-05-16 | 2019-10-15 | 上海精密计量测试研究所 | A kind of electronic component fault diagnosis model based on data analysis prediction |
| CN110765236A (en) * | 2019-10-09 | 2020-02-07 | 中国人民解放军国防科技大学 | Preprocessing method and system for unstructured mass data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210392152A1 (en) | Intrusion detection using robust singular value decomposition | |
| US9734343B2 (en) | Detection and prevention of sensitive information leaks | |
| CN106301823B (en) | Fault warning method and device for key component and big data management system | |
| DE102017128694A1 (en) | Multi-mode border selection for threat detection in an industrial plant control system | |
| CN110290522B (en) | Risk identification method and device for mobile equipment and computer system | |
| US20160352679A1 (en) | Identifying algorithmically generated domains | |
| US20150363518A1 (en) | Dynamic operating procedures for emergency response | |
| US20110302291A1 (en) | Methods and systems for prioritizing network assets | |
| US20110208679A1 (en) | Trouble pattern creating program and trouble pattern creating apparatus | |
| CN107733834B (en) | Data leakage protection method and device | |
| CN109889550A (en) | A kind of ddos attack determines method and device | |
| US10701031B2 (en) | Identifying algorithmically generated domains | |
| CN106022349B (en) | Method and system for device type determination | |
| CN103595807A (en) | Data synchronous updating method and device | |
| CN113783845B (en) | Method and device for determining risk level of instance on cloud server, electronic equipment and storage medium | |
| US11531676B2 (en) | Method and system for anomaly detection based on statistical closed-form isolation forest analysis | |
| CN114881708A (en) | Method, device, system, equipment and storage medium for monitoring delivered content | |
| CN110895566A (en) | Vehicle evaluation method and device | |
| CN111352929A (en) | Data processing method | |
| CN112069137A (en) | Method and device for generating information, electronic equipment and computer readable storage medium | |
| CN111046010A (en) | Log storage method, device, system, electronic equipment and computer readable medium | |
| JP6541903B2 (en) | Attack / abnormality detection device, attack / abnormality detection method, and attack / abnormality detection program | |
| CN114995914A (en) | Picture data processing method and device, computer equipment and storage medium | |
| CN105574404B (en) | A kind of method and apparatus prompting Modify password | |
| CN106648795A (en) | Updating method and device of application program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200630 |
|
| WD01 | Invention patent application deemed withdrawn after publication |