CN109344030A - The method of streaming fault data write-in processing - Google Patents
The method of streaming fault data write-in processing Download PDFInfo
- Publication number
- CN109344030A CN109344030A CN201811108916.3A CN201811108916A CN109344030A CN 109344030 A CN109344030 A CN 109344030A CN 201811108916 A CN201811108916 A CN 201811108916A CN 109344030 A CN109344030 A CN 109344030A
- Authority
- CN
- China
- Prior art keywords
- data
- zookeeper
- kafka
- consumption terminal
- fault
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of methods of streaming fault data write-in processing, comprising: Step 1: configuration Zookeeper application attribute；Creation JavaConsumer consumption terminal, directly foundation are linked with Kafka cluster service；Step 2: JavaConsumer consumption terminal reads Kafka theme flow data；Step 3: Kafka flow data is switched to object data by String class method, handled using multithreading ThreadPool: fault message is correct, and MySQL database is written；Fault message mistake is set as empty, is not written into MySQL database；It is closed Step 4: removing application.The present invention does not limit data package size in entire treatment process, and the processing of all fault datas may be implemented, and reduces and service needed for data transmission.
Description
Technical field
The present invention relates to a kind of fault data processing methods, and in particular to a kind of side of streaming fault data write-in processing
Method belongs to big data calculation processing technical field.
Background technique
In big data calculation processing technology, from distributed information system (Kafka) read failure data and MySQL is written
Database, as shown in Fig. 1, at present to the mode of fault data processing are as follows:
Step 1: the connection of configuration Zookeeper and Kafka；
Step 2: the connection of configuration ICE and Zookeeper；
Step 3: using PythonConsumer connection ICE, read failure data；
Step 4: PythonConsumer handling failure data, if fault data information is correct, just by correct failure
MySQL database is written in data；If fault data information is incorrect, it is set as empty, is not written into MySQL database.
This method is in reading process, since ICE middleware limits data package size (10M), causes to be more than size valve
The data packet of value (10M) can trigger ICE alarm exception and be dropped, and it is normal that fault data cannot be transmitted to PythonConsumer
Processing causes partial fault loss of data phenomenon to occur.
Summary of the invention
It is an object of the invention to overcome in the prior art since the limitation of ICE middleware data package size leads to super large data
The problem of packet triggering ICE alerts exception and is dropped, fault data is caused to lose provides a kind of streaming fault data write-in processing
Method.
Core of the invention thinking is:
In data calculation processing technology, from distributed information system (Kafka) read failure data, be written after processing
MySQL database, in such a way that JavaConsumer substitutes ICE middleware and PythonConsumer, to solve ICE pairs
The size of data packet limits (10M) problem, realizes the processing of all fault datas, and reduces and service needed for data transmission.
The present invention is specifically achieved in that
1, a kind of method of streaming fault data write-in processing, characterized by comprising:
Step 1: configuration Zookeeper application attribute, property content specifically include that
" zookeeper.connect ": Zookeeper cluster service link address；
" group.id ": Consumer consumption terminal uniquely indicates ID；
" zookeeper.session.timeout.ms ": Zookeeper timeout duration of session；
" zookeeper.sync.time.ms ": the Zookeeper service connection duration；
" auto.commit.interval.ms ": Consumer consumption terminal offset automatically updates the time；
Using the KEY-VALUE key-value pair of properties property file creation attribute, pass through
Java.util.Properties class loads properties property file flow object, calls Properties class
GetProperty (KEY) method obtains VALUE, reuses ConsumerConfig class wrapper KEY-VALUE attribute information,
JavaConsumer consumption terminal is created, is loaded using kafka.consumer.Consumer class method
ConsumerConfig class object creates JavaConsumerConnector connecting object, directly establishes and takes with Kafka cluster
The link of business；
Step 2: JavaConsumer consumption terminal uses the FixedThreadPool static method of Executors object,
The thread pool of given thread number is created, Kafka theme flow data is read: TOPIC theme is specified by Consumer consumption terminal, it will
Theme flow data is stored in LIST set, reads all subregion flow datas that theme includes: obtaining subregion by circulation LIST set
The flow data of Partition is stored in KafkaStream, and there is no limit for size of data；
Step 3: Kafka flow data Stream is switched to object data Object, setting coding by String class method
Format is UTF-8, and is handled using multithreading ThreadPool: fault message is correct, and MySQL database is written；Fault message
Mistake is set as empty, is not written into MySQL database；
It is closed Step 4: removing application, thread mistake and abnormality processing.
The present invention does not limit data package size in entire treatment process, and the processing of all fault datas may be implemented, and
And service needed for reducing data transmission.
Detailed description of the invention
Fig. 1 is the flow chart of the prior art.
Fig. 2 is the flow chart of the method for streaming fault data write-in processing of the present invention.
Specific embodiment
The present invention is further illustrated in the following with reference to the drawings and specific embodiments.
As shown in Fig. 2, it by taking fault data is handled as an example, describes in detail below to the embodiment.
A kind of method of streaming fault data write-in processing, comprising:
Configure Zookeeper application attribute (" zookeeper.connect ": Zookeeper cluster service connection ground
Location IP:PORT；" group.id ": Consumer consumption terminal uniquely indicates ID), create JavaConsumer consumption terminal
(JavaConsumerConnector), Kafka cluster is directly linked；
JavaConsumer consumption terminal create thread pool (FixedThreadPool), read Kafka theme (Topic) and
Subregion (Partition) flow data, there is no limit for size of data；
Kafka flow data (Stream) is switched into object data (Object, UTF-8 coding lattice by String class method
Formula), and use multiple threads (Threads)；
It removes application and closes (Shutdown), thread mistake and abnormality processing (Error&Exception)；
Entire treatment process does not limit data package size, realizes the processing of all fault datas, and reduces data transmission
Required service.
Although reference be made herein to invention has been described for explanatory embodiment of the invention, and above-described embodiment is only this hair
Bright preferable embodiment, embodiment of the present invention are not limited by the above embodiments, it should be appreciated that those skilled in the art
Member can be designed that a lot of other modification and implementations, these modifications and implementations will fall in principle disclosed in the present application
Within scope and spirit.
Claims (1)
1. a kind of method of streaming fault data write-in processing, characterized by comprising:
Step 1: configuration Zookeeper application attribute, property content specifically include that
" zookeeper.connect ": Zookeeper cluster service link address；
" group.id ": Consumer consumption terminal uniquely indicates ID；
" zookeeper.session.timeout.ms ": Zookeeper timeout duration of session；
" zookeeper.sync.time.ms ": the Zookeeper service connection duration；
" auto.commit.interval.ms ": Consumer consumption terminal offset automatically updates the time；
Using the KEY-VALUE key-value pair of properties property file creation attribute, pass through java.util.Properties
Class loads properties property file flow object, and the getProperty method of Properties class is called to obtain VALUE, then
Using ConsumerConfig class wrapper KEY-VALUE attribute information,
JavaConsumer consumption terminal is created, is loaded using kafka.consumer.Consumer class method
ConsumerConfig class object creates JavaConsumerConnector connecting object, directly establishes and takes with Kafka cluster
The link of business；
Step 2: JavaConsumer consumption terminal uses the FixedThreadPool static method of Executors object, creation
The thread pool of given thread number reads Kafka theme flow data: TOPIC theme is specified by Consumer consumption terminal, by theme
Flow data is stored in LIST set, reads all subregion flow datas that theme includes: obtaining subregion by circulation LIST set
The flow data of Partition is stored in KafkaStream, and there is no limit for size of data；
Step 3: Kafka flow data Stream is switched to object data Object by String class method, coded format is set
For UTF-8, and handled using multithreading ThreadPool: fault message is correct, and MySQL database is written；Fault message mistake,
It is set as empty, is not written into MySQL database；
It is closed Step 4: removing application, thread mistake and abnormality processing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811108916.3A CN109344030A (en) | 2018-09-21 | 2018-09-21 | The method of streaming fault data write-in processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811108916.3A CN109344030A (en) | 2018-09-21 | 2018-09-21 | The method of streaming fault data write-in processing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109344030A true CN109344030A (en) | 2019-02-15 |
Family
ID=65306321
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811108916.3A Pending CN109344030A (en) | 2018-09-21 | 2018-09-21 | The method of streaming fault data write-in processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109344030A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110493342A (en) * | 2019-08-21 | 2019-11-22 | 北京明朝万达科技股份有限公司 | Document transmission method, device, electronic equipment and readable storage medium storing program for executing |
| CN111723070A (en) * | 2020-06-18 | 2020-09-29 | 中国银行股份有限公司 | Method and system for detecting data loss |
| CN112395160A (en) * | 2020-11-18 | 2021-02-23 | 上海许继电气有限公司 | Method, device and system for generating relay protection fault report |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150095338A1 (en) * | 2013-09-27 | 2015-04-02 | Linkedln Corporation | Systems and methods for categorizing exceptions and logs |
| CN106533791A (en) * | 2016-12-12 | 2017-03-22 | 武汉烽火信息集成技术有限公司 | End-to-end business quality optimization apparatus and method based on big data platform |
| CN106649000A (en) * | 2017-01-03 | 2017-05-10 | 星环信息科技（上海）有限公司 | Fault recovery method for real-time processing engine, and corresponding server |
| CN106778033A (en) * | 2017-01-10 | 2017-05-31 | 南京邮电大学 | A kind of Spark Streaming abnormal temperature data alarm methods based on Spark platforms |
-
2018
- 2018-09-21 CN CN201811108916.3A patent/CN109344030A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150095338A1 (en) * | 2013-09-27 | 2015-04-02 | Linkedln Corporation | Systems and methods for categorizing exceptions and logs |
| CN106533791A (en) * | 2016-12-12 | 2017-03-22 | 武汉烽火信息集成技术有限公司 | End-to-end business quality optimization apparatus and method based on big data platform |
| CN106649000A (en) * | 2017-01-03 | 2017-05-10 | 星环信息科技（上海）有限公司 | Fault recovery method for real-time processing engine, and corresponding server |
| CN106778033A (en) * | 2017-01-10 | 2017-05-31 | 南京邮电大学 | A kind of Spark Streaming abnormal temperature data alarm methods based on Spark platforms |
Non-Patent Citations (1)
| Title |
|---|
| 佚名: "[Kafka]一Kafka Java Consumer实现(二)", 《CSDN》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110493342A (en) * | 2019-08-21 | 2019-11-22 | 北京明朝万达科技股份有限公司 | Document transmission method, device, electronic equipment and readable storage medium storing program for executing |
| CN110493342B (en) * | 2019-08-21 | 2021-05-14 | 北京明朝万达科技股份有限公司 | File transmission method and device, electronic equipment and readable storage medium |
| CN111723070A (en) * | 2020-06-18 | 2020-09-29 | 中国银行股份有限公司 | Method and system for detecting data loss |
| CN112395160A (en) * | 2020-11-18 | 2021-02-23 | 上海许继电气有限公司 | Method, device and system for generating relay protection fault report |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109344030A (en) | The method of streaming fault data write-in processing | |
| CN109788068B (en) | Heartbeat state information reporting method, device and equipment and computer storage medium | |
| CN109391691A (en) | The restoration methods and relevant apparatus that NAS is serviced under a kind of single node failure | |
| CN110650164B (en) | File uploading method and device, terminal and computer storage medium | |
| CN111949497B (en) | Message queue system and message processing method based on same | |
| CN108322318B (en) | Alarm analysis method and equipment | |
| US10089038B2 (en) | Lock-free first in, first out memory queue architecture | |
| CN110971922A (en) | Live broadcast and microphone connecting state monitoring method, device, storage medium and system | |
| CN106170013B (en) | A kind of Kafka message uniqueness method based on Redis | |
| CN106027379A (en) | Push message receiving processing method | |
| CN106997313B (en) | Signal processing method and system of application program and terminal equipment | |
| CN104580523B (en) | Self-aided terminal supervising data storage method and self-aided terminal monitoring server | |
| CN106375767A (en) | Hybrid video decoder and associated hybrid video decoding method | |
| CN112162899A (en) | Big data cluster service based monitoring method and system | |
| CN110333916A (en) | Request message processing method, device, computer system and readable storage medium storing program for executing | |
| CN108021486B (en) | The method and storage medium that electronic device, collage-credit data are handled | |
| CN109255091A (en) | Method, apparatus, equipment and the computer readable storage medium of shared theme message | |
| CN103309710A (en) | Method and system for loading OUT file through VXWORKS operating system | |
| CN106547609B (en) | Event processing method and device | |
| CN110457133A (en) | A kind of processing method and system of operating-system resources exception | |
| CN108446304A (en) | Data block retrieval system and method | |
| CN115168434A (en) | Data synchronization method and equipment for shared storage cluster database | |
| CN108616461B (en) | Policy switching method and device | |
| CN106953774A (en) | One kind is based on user-defined network packet snapping system | |
| CN112332952A (en) | Underwater acoustic communication method for underwater glider and submerged buoy |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190215 |
|
| RJ01 | Rejection of invention patent application after publication |