CN110647547A - Consumption delay monitoring method and device, electronic equipment and computer readable storage medium - Google Patents
Consumption delay monitoring method and device, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN110647547A CN110647547A CN201910895091.2A CN201910895091A CN110647547A CN 110647547 A CN110647547 A CN 110647547A CN 201910895091 A CN201910895091 A CN 201910895091A CN 110647547 A CN110647547 A CN 110647547A
- Authority
- CN
- China
- Prior art keywords
- data
- consumption
- consumption delay
- time
- kafka system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2477—Temporal data queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the specification discloses a consumption delay monitoring method and device, electronic equipment and a computer-readable storage medium. The method is applied to a stream processing system based on a Spark Streaming framework, and comprises the following steps: reading stream data from a Kafka system according to a preset time interval, wherein the stream data comprises a plurality of pieces of service data carrying production time stamps; determining the consumption delay time of the Kafka system based on a production timestamp of target business data in the streaming data, wherein the target business data is business data with the latest production timestamp in the streaming data; and monitoring the consumption delay of the Kafka system based on the consumption delay time. The method and the device can realize effective monitoring of consumption delay information in the Kafka system.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a consumption delay monitoring method and apparatus, an electronic device, and a computer-readable storage medium.
Background
The existing Spark Streaming framework cannot realize monitoring of consumption delay in the Kafka system, so that the data consumption condition in the Kafka system, such as the residual data processing capacity of the Kafka system, cannot be known during data consumption.
Disclosure of Invention
An object of the embodiments of the present specification is to provide a consumption delay monitoring method, apparatus, electronic device and computer-readable storage medium, so as to implement delay monitoring of data in a Kafka system. The implementation of the embodiments of the present description is as follows.
In a first aspect, a consumption delay monitoring method is provided, which is applied to a stream processing system based on a Spark Streaming framework, and the method includes:
reading stream data from a Kafka system according to a preset time interval, wherein the stream data comprises a plurality of pieces of service data carrying production time stamps;
determining the consumption delay time of the Kafka system based on the production time stamp of target service data in the stream data, wherein the target service data is the service data with the latest production time stamp in the stream data;
based on the consumption delay time, the Kafka system is monitored for consumption delay.
In a second aspect, a consumption delay monitoring apparatus is provided, which is applied to a Streaming processing system based on a Spark Streaming framework, and the apparatus includes:
the data reading module is used for reading stream data from the Kafka system according to a preset time interval, and the stream data comprises a plurality of pieces of service data with production time stamps;
the delay calculation module is used for determining the consumption delay time of the Kafka system based on the production time stamp of target business data in the streaming data, wherein the target business data is the business data with the latest production time stamp in the streaming data;
and the delay monitoring module is used for monitoring the consumption delay of the Kafka system based on the consumption delay time.
In a third aspect, an electronic device is proposed, which comprises a processor, a memory and a computer program stored on the memory and being executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the consumption delay monitoring method described above.
In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and when being executed by a processor, the computer program implements the steps of the consumption delay monitoring method described above.
The embodiment of the specification adopts at least one technical scheme which can achieve the following beneficial effects:
when the Streaming data is read from the Kafka system at preset time intervals by the Streaming processing system based on the Spark Streaming framework, the consumption delay time of the Kafka system can be determined based on the production timestamp of the target service data in the Streaming data read each time, and further, the consumption monitoring of the data in the Kafka system is realized based on the consumption delay time, so that the data consumption condition in the Kafka system can be known in real time.
Drawings
In order to more clearly illustrate the embodiments of the present specification or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only some embodiments described in the present specification, and for those skilled in the art, other drawings can be obtained according to the drawings without any creative effort.
Fig. 1 is an exemplary block diagram of a consumption data monitoring process flow implemented based on a spark streaming framework in the prior art.
Fig. 2 is a schematic step diagram of a consumption delay monitoring method according to an embodiment of the present application.
Fig. 3 is an exemplary block diagram of a consumption data monitoring processing flow implemented based on a spark streaming framework according to an embodiment of the present application.
Fig. 4 is a schematic structural diagram of a consumption delay monitoring apparatus according to an embodiment of the present application.
Fig. 5 is an exemplary block diagram of an electronic device provided by an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be described in detail and completely with reference to the following specific embodiments of the present application and the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.
With the continuous development of network technologies, a large amount of business data is generated when online behaviors such as online shopping, online games, financial transactions, hacking and the like occur. As shown in fig. 1, as a producer end of the service data, the generated service data needs to be stored in the Kafka system as consumption data, and as a consumer end, the sparking streaming processing technology needs to be used to continuously read the consumption data from the Kafka system and perform data analysis and monitoring, and write the processed data into the storage system for being called by the downstream service system.
The streaming processing system that can be implemented based on the streaming processing framework at present can be used to implement monitoring processing of basic information such as start time, end time, processing duration, data volume, and the like of consumption data. In addition, the Kafka system is a high-throughput distributed publish-subscribe messaging system, and is used for storing consumption data generated by a production end.
However, due to the influence of problems such as system performance, when the streaming processing system implemented based on the spark streaming framework pulls consumption data in the Kafka system, the existing streaming processing system cannot monitor consumption delay in the Kafka system, so that the data processing situation in the Kafka system cannot be known, for example, how long data in the Kafka system has not been processed.
In view of this, embodiments of the present disclosure provide a consumption delay monitoring method, an apparatus, an electronic device, and a computer-readable storage medium, and the following detailed description is provided to the technical solutions provided in the present disclosure in combination with the embodiments.
Example one
Referring to fig. 2, a schematic step diagram of a consumption delay monitoring method according to an embodiment of the present disclosure is shown. It should be understood that the consumption delay monitoring method according to an embodiment of the present specification may be executed by, but is not limited to, an electronic device equipped with a streaming system based on spark stream. The consumption delay monitoring method may include the steps of:
and step 11, reading stream data from the Kafka system according to a preset time interval, wherein the stream data comprises a plurality of pieces of service data carrying production time stamps.
In the first embodiment of this specification, the production timestamp carried in the service data may be added to the service data by the production end when the production end produces the service data, or may be added to the service data when the produced service data is written into the Kafka system, and the first embodiment is not limited herein.
Alternatively, the length of the preset time interval may be flexibly designed according to, for example, the system performance of the Kafka system, the data amount of the consumption data, the speed of the production business data of the production end, and the like, and the preset time interval may be 2 seconds, 5 seconds, and the like. In addition, the amount of the service data included in the stream data read each time can also be flexibly set according to the speed of producing and consuming data at the production end, or the data processing capacity of the Kafka system or the stream processing system, and the embodiment is not limited herein.
And step 12, determining the consumption delay time of the Kafka system based on the production time stamp of the target service data in the streaming data.
In the first embodiment of the present specification, the target service data is the service data with the latest production timestamp in the stream data. Alternatively, the consumption delay time may be calculated in various manners, and for example, the consumption delay time of the Kafka system may be determined based on a time difference between a production time stamp of target traffic data in the stream data and a current processing time when the stream data is processed.
As one way of realization, assuming that the production time stamp of the target service data is 2019, month 13, day 11, 25 minutes and 31 seconds, and the time when the stream data containing the target service data is read and processed by the stream processing system is 2019, month 13, day 11, 25 minutes and 50 seconds, then the consumption delay time of the Kafka system, which is determined based on the production time stamp of the target service data 2019, month 13, day 11, 25 minutes and 31 seconds, and the current processing time when the stream data is processed 2019, month 13, day 11, 25 minutes and 50 seconds, is 19 seconds. In addition, it can be determined from the consumption delay time that there is 19 seconds of consumption data waiting processing in the Kafka system.
For another example, in determining the consumption delay time, the consumption delay time of the Kafka system may also be determined based on a time difference between the production time stamp of the target traffic data in the stream data and the current processing time when the target traffic data is processed.
As one way of realization, assuming that the production time stamp of the target service data is 25 minutes 31 seconds at 11 hours at 13 days 11 months in 2019, and the time when the target service data is processed is 25 minutes 50 seconds at 11 hours at 13 days 11 months in 2019, then the consumption delay time of the Kafka system determined from the production time stamp of the target service data at 2019, 13 days 11 hours 25 minutes 31 seconds at 13 months in 2019, and the current processing time at 2019, 13 days 11 minutes 25 minutes 50 seconds at 11 days in 13 months in 2019, is 19 seconds. In addition, it can be determined from the consumption delay time that there is 19 seconds of consumption data waiting processing in the Kafka system.
It should be noted that the stream data being processed or the target service data being processed as described in the foregoing two implementations may refer to, but is not limited to, the stream processing system analyzing basic information such as a processing start time, a processing end time, a processing time length, a data amount, and the like of the stream data read each time based on the stream data or/and the target service data.
It should be understood that, besides the two ways of determining the consumption delay time given in the first embodiment, as an achievable way, the determination of the consumption delay time may also be implemented according to the data amount of the service data in the stream data read each time.
For example, the stream processing system may also determine the consumption delay time in the Kafka system by detecting whether the amount of data in the traffic data in the stream data read from the Kafka system is zero. For example, when the amount of data in the stream data read from the Kafka system is zero, the consumption delay time of the Kafka system is determined to be zero.
For another example, assuming that the data amount of the service data in the stream data read from the Kafka system each time is smaller than the preset data amount, the consumption delay time of the Kafka system may also be determined to be zero.
Further, as an optional implementation manner, when the stream processing system performs consumption data delay monitoring, consumption delay monitoring may be performed on stream data read each time as described in the first embodiment, or consumption delay monitoring may be performed every preset time (for example, 3 times of the preset time interval, etc.) or data reading times (reading 3 times of stream data), and the first embodiment is not limited herein.
It should be noted that, in an implementable scheme, in addition to the function of monitoring the consumption delay time of the read stream data in the above steps 11 to 13, the stream processing system can also analyze and monitor the basic information such as the processing start time, the processing end time, the processing time length, the data amount, etc. of the stream data read each time, and the embodiment is not limited herein.
And step 13, based on the consumption delay time, carrying out consumption delay monitoring on the Kafka system.
In the first embodiment of the present specification, the stream processing system may monitor the processing condition of the consumption data in the Kafka system, such as the amount of data remaining to be processed in the Kafka system, according to the determined consumption delay time.
As an optional implementation manner, in order to implement intuitive and accurate consumption delay monitoring on the Kafka system, the consumption delay monitoring on the Kafka system can be performed by performing visual graphic display on the consumption delay time and based on a visual graphic display result.
For example, when the consumption delay time is visually displayed, the consumption delay time can be read and visually displayed based on the grafana visualization tool. For example, after determining the consumption delay time, the stream processing system may write the consumption delay time into the database, read the consumption delay time of the Kafka system from the database through the grafana visualization graph tool, and perform a visualization graph display on the consumption delay time, which is not limited in this embodiment. Alternatively, in a specific implementation, the aforementioned database may be, but is not limited to, a clickwouse database or the like shown in fig. 3, which is a database that can be used for data analysis.
In combination with the exemplary block diagram of the consumption data monitoring processing flow implemented based on the spark streaming framework provided in fig. 3, it is obvious that the delay processing time of the consumption data in the Kafka system is obtained by adding the delay calculation module to the stream processing system to perform consumption delay monitoring on the consumption data in the Kafka system, so as to know the data consumption situation in the Kafka system in real time.
In addition, when the stream processing system is based on consumption delay time setting, the consumption condition in the Kafka system is monitored in a visual graphic display mode, so that a worker can more intuitively and quickly know the delay processing condition of the consumption data in the Kafka system or the residual consumption data condition in the Kafka system and the like.
Example two
Based on the consumption delay monitoring method provided in the first embodiment, in order to further ensure real-time performance of delay monitoring data and enable workers and the like to quickly know consumption delay time, in the second embodiment, a delay alarm prompt may be performed according to a monitoring result of the consumption delay monitoring method provided in the first embodiment. For example, when the consumption delay time is greater than a preset threshold, a delayed alarm prompt is sent.
Optionally, the preset threshold may be set according to a requirement, for example, the preset threshold may be 10 seconds, 20 seconds, and the like, and the second embodiment is not limited herein.
In addition, as an optional implementation manner, when the stream processing system performs the alarm prompt according to the consumption delay time, the implementation may be implemented in a manner of sound, short message, mail, or the like, for example, the delay alarm prompt may be implemented based on an alarm module included in the grafana visualization tool, or may be implemented by an alarm component that is built in the stream processing system, and the second embodiment is not limited herein.
Further, as an optional implementation manner, after the consumption delay monitoring method provided in the first embodiment of the present invention is used to implement consumption delay monitoring on consumption data in the Kafka system, the consumption delay time or/and stream data that completes the consumption delay monitoring process may be written into the storage system, so as to be called by a downstream service system of the stream processing system.
Wherein, the downstream service system is different according to the different types of the service data in the stream data. For example, when the streaming data is transaction information generated by online shopping behavior, the downstream business system may be a business system that analyzes the user's purchasing taste, attention, and the like based on the transaction information.
Compared with the consumption delay monitoring method provided in the first embodiment, in the consumption delay monitoring method provided in the second embodiment, by adding the delay alarm function and the data writing function (for example, writing the consumption delay time or/and stream data completing the consumption delay monitoring processing into the storage system), the real-time performance of the delay monitoring data can be effectively ensured, and meanwhile, the storage system ensures the accuracy of the data processing result when calling the stream data for processing such as service analysis.
EXAMPLE III
Fig. 4 is a schematic structural diagram of the consumption delay monitoring apparatus 10 according to the third embodiment of the present disclosure. Referring to fig. 4, in a software implementation, the consumption delay monitoring apparatus 10 may include a data reading module 110, a delay calculating module 120, and a delay monitoring module 130. Wherein:
and the data reading module 110 is configured to read stream data from the Kafka system according to a preset time interval, where the stream data includes a plurality of pieces of service data carrying production time stamps.
And the delay calculation module 120 is configured to determine the consumption delay time of the Kafka system based on the production timestamp of the target service data in the stream data, where the target service data is the service data with the latest production timestamp in the stream data.
And a delay monitoring module 130, configured to perform consumption delay monitoring on the Kafka system based on the consumption delay time.
In the embodiment of the present specification, the delay processing time of the consumption data in the Kafka system is obtained by monitoring the consumption delay of the consumption data in the Kafka system, and the data consumption condition in the Kafka system can be known in real time. When the stream processing system monitors the Kafka system based on the consumption delay time, the stream processing system can be realized in a visual graphic display mode, so that a worker can more intuitively and quickly know the delay processing condition of the consumption data in the Kafka system or the residual consumption data condition in the Kafka system and the like.
As an optional implementation manner, the delay calculation module 120 is specifically configured to determine the consumption delay time of the Kafka system based on a time difference between the production timestamp of the target service data in the stream data and the current processing time when the stream data is processed.
Still alternatively, as another optional implementation manner, the delay calculation module 120 may be further configured to determine that the consumption delay time of the Kafka system is zero when the data amount in the stream data read from the Kafka system is zero.
As a further alternative, the delay monitoring module 130 may be specifically configured to perform a visual graphical display on the consumption delay time, and perform consumption delay monitoring on the Kafka system based on the visual graphical display result. For example, the consumption delay time is written into a database, the consumption delay time of the Kafka system is read from the database through a visual graphic tool, and the consumption delay time is visually and graphically displayed.
It can be understood that, since the data reading module 110, the delay calculating module 120, and the delay monitoring module 130 have the same or corresponding technical features as the consumption delay monitoring method provided in the first embodiment, the detailed description of the data reading module 110, the delay calculating module 120, and the delay monitoring module 130 may refer to the detailed description of the consumption delay monitoring method in the first embodiment, and the third embodiment is not repeated herein.
Further, in a specific implementation manner of the embodiment of the present specification, the consumption delay monitoring apparatus 10 may further include an information warning module and an information saving module.
The information alarm module is used for sending a delay alarm prompt when the consumption delay time is larger than a preset threshold value.
The information storage module is used for writing the consumption delay time or/and the stream data completing the consumption delay monitoring processing into the storage system so as to be called by a downstream service system of the stream processing system.
Since the information alarm module and the information storage module have the same or corresponding technical features as those of the consumption delay monitoring method provided in the second embodiment, the detailed description of the information alarm module and the information storage module can refer to the detailed description of the consumption delay monitoring method in the second embodiment, and the third embodiment is not repeated herein.
Example four
Fig. 5 is a schematic structural diagram of an electronic device according to a fourth embodiment of this specification. Referring to fig. 5, at a hardware level, the electronic device includes a processor, and optionally further includes an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory, such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.
The processor, the network interface, and the memory may be connected to each other via an internal bus, which may be an ISA (Industry Standard Architecture) bus, a PCI (peripheral component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 5, but this does not indicate only one bus or one type of bus.
And the memory is used for storing programs. In particular, the program may include program code including computer operating instructions. The memory may include both memory and non-volatile storage and provides instructions and data to the processor.
The processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs the computer program to form the shared resource access control device on the logic level. The processor is used for executing the program stored in the memory and is specifically used for executing the following operations:
reading stream data from a Kafka system according to a preset time interval, wherein the stream data comprises a plurality of pieces of service data carrying production time stamps;
determining the consumption delay time of the Kafka system based on the production time stamp of target service data in the stream data, wherein the target service data is the service data with the latest production time stamp in the stream data;
based on the consumption delay time, the Kafka system is monitored for consumption delay.
The method performed by the consumption delay monitoring apparatus 10 according to the embodiment shown in fig. 4 of the present specification can be applied to or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic blocks disclosed in the embodiments of the present specification may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present specification may be embodied directly in a hardware decoding processor, or in a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor.
The electronic device may further execute the consumption delay monitoring method shown in fig. 2, and implement the functions of the consumption delay monitoring apparatus 10 in the embodiment shown in fig. 4, which are not described herein again in this specification.
Of course, besides the software implementation, the electronic device of the embodiment of the present disclosure does not exclude other implementations, such as a logic device or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, and may also be hardware or a logic device.
EXAMPLE five
An embodiment five of the present specification further proposes a computer-readable storage medium storing one or more programs, where the one or more programs include instructions, which when executed by a portable electronic device including a plurality of application programs, can cause the portable electronic device to execute the method of the embodiment shown in fig. 2, and specifically to execute the following method:
reading stream data from a Kafka system according to a preset time interval, wherein the stream data comprises a plurality of pieces of service data carrying production time stamps;
determining the consumption delay time of the Kafka system based on the production time stamp of target service data in the stream data, wherein the target service data is the service data with the latest production time stamp in the stream data;
based on the consumption delay time, the Kafka system is monitored for consumption delay.
Through the fifth embodiment of the present specification, the consumption delay monitoring of the consumption data in the Kafka system is realized by adding the delay calculation module in the stream processing system to obtain the delay processing time of the consumption data in the Kafka system, so as to ensure the real-time performance of the delay monitoring data.
In addition, when the stream processing system monitors the Kafka system based on the consumption delay time, the stream processing system is realized in a visual graphic display mode, so that a worker can more intuitively and quickly know the delay processing condition of the consumption data in the Kafka system or the residual consumption data condition in the Kafka system and the like.
The above description is only a preferred embodiment of the present disclosure, and is not intended to limit the scope of the present disclosure. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present specification shall be included in the protection scope of the present specification.
The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. One typical implementation device is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
Claims (10)
1. A consumption delay monitoring method applied to a Streaming processing system based on a Spark Streaming framework, the method comprising:
reading stream data from a Kafka system according to a preset time interval, wherein the stream data comprises a plurality of pieces of service data carrying production time stamps;
determining the consumption delay time of the Kafka system based on a production timestamp of target business data in the streaming data, wherein the target business data is business data with the latest production timestamp in the streaming data;
and monitoring the consumption delay of the Kafka system based on the consumption delay time.
2. The consumption delay monitoring method according to claim 1, wherein the step of determining the consumption delay time of the Kafka system based on the production time stamp of the target traffic data in the streaming data comprises:
determining a consumption delay time of the Kafka system based on a time difference between a production time stamp of target service data in the streaming data and a current processing time when the streaming data is processed.
3. The consumption delay monitoring method according to claim 1, wherein the step of performing consumption delay monitoring on the Kafka system based on the consumption delay time comprises:
and performing visual graphic display on the consumption delay time, and performing consumption delay monitoring on the Kafka system based on a visual graphic display result.
4. The consumption delay monitoring method according to claim 3, wherein the step of visually and graphically displaying the consumption delay time comprises:
and writing the consumption delay time into a database, reading the consumption delay time of the Kafka system from the database through a visual graphic tool, and carrying out visual graphic display on the consumption delay time.
5. The consumption delay monitoring method according to any one of claims 1 to 4, wherein the method further comprises:
when the amount of data in the stream data read from the Kafka system is zero, it is determined that the consumption delay time of the Kafka system is zero.
6. The consumption delay monitoring method according to any one of claims 1 to 4, wherein the method further comprises:
and when the consumption delay time is greater than a preset threshold value, sending a delay alarm prompt.
7. The consumption delay monitoring method of claim 1, further comprising:
and writing the consumption delay time or/and the stream data completing the consumption delay monitoring processing into a storage system so as to be called by a downstream business system of the stream processing system.
8. A consumption delay monitoring apparatus applied to a Streaming processing system based on a Spark Streaming framework, the apparatus comprising:
the data reading module is used for reading stream data from the Kafka system according to a preset time interval, wherein the stream data comprises a plurality of pieces of service data with production time stamps;
the delay calculation module is used for determining the consumption delay time of the Kafka system based on the production time stamp of target business data in the streaming data, wherein the target business data is the business data with the latest production time stamp in the streaming data;
and the delay monitoring module is used for monitoring the consumption delay of the Kafka system based on the consumption delay time.
9. An electronic device comprising a processor, a memory and a computer program stored on the memory and executable on the processor, the computer program, when executed by the processor, implementing the steps of the consumption delay monitoring method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the consumption delay monitoring method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910895091.2A CN110647547A (en) | 2019-09-20 | 2019-09-20 | Consumption delay monitoring method and device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910895091.2A CN110647547A (en) | 2019-09-20 | 2019-09-20 | Consumption delay monitoring method and device, electronic equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110647547A true CN110647547A (en) | 2020-01-03 |
Family
ID=69010959
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910895091.2A Pending CN110647547A (en) | 2019-09-20 | 2019-09-20 | Consumption delay monitoring method and device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110647547A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111600774A (en) * | 2020-05-13 | 2020-08-28 | 北京奇艺世纪科技有限公司 | Consumption delay determination method, system, device, equipment and readable storage medium |
| CN113190516A (en) * | 2021-05-26 | 2021-07-30 | 中国工商银行股份有限公司 | Data synchronization monitoring method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170061452A1 (en) * | 2015-08-31 | 2017-03-02 | Wal-Mart Stores, Inc. | System for forecasting using low-rank matrix completion and method therefor |
| CN107423141A (en) * | 2017-06-20 | 2017-12-01 | 北京小度信息科技有限公司 | Information processing method and device |
| CN107704545A (en) * | 2017-11-08 | 2018-02-16 | 华东交通大学 | Railway distribution net magnanimity information method for stream processing based on Storm Yu Kafka message communicatings |
| CN109743137A (en) * | 2019-01-10 | 2019-05-10 | 浙江小泰科技有限公司 | A kind of Distributed Delay message queue processing system for supporting to update |
| CN109861878A (en) * | 2019-01-17 | 2019-06-07 | 平安科技（深圳）有限公司 | The monitoring method and relevant device of the topic data of kafka cluster |
| CN110187980A (en) * | 2019-04-19 | 2019-08-30 | 北京三快在线科技有限公司 | Message queue monitoring method, device, electronic equipment and readable storage medium storing program for executing |
-
2019
- 2019-09-20 CN CN201910895091.2A patent/CN110647547A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170061452A1 (en) * | 2015-08-31 | 2017-03-02 | Wal-Mart Stores, Inc. | System for forecasting using low-rank matrix completion and method therefor |
| CN107423141A (en) * | 2017-06-20 | 2017-12-01 | 北京小度信息科技有限公司 | Information processing method and device |
| CN107704545A (en) * | 2017-11-08 | 2018-02-16 | 华东交通大学 | Railway distribution net magnanimity information method for stream processing based on Storm Yu Kafka message communicatings |
| CN109743137A (en) * | 2019-01-10 | 2019-05-10 | 浙江小泰科技有限公司 | A kind of Distributed Delay message queue processing system for supporting to update |
| CN109861878A (en) * | 2019-01-17 | 2019-06-07 | 平安科技（深圳）有限公司 | The monitoring method and relevant device of the topic data of kafka cluster |
| CN110187980A (en) * | 2019-04-19 | 2019-08-30 | 北京三快在线科技有限公司 | Message queue monitoring method, device, electronic equipment and readable storage medium storing program for executing |
Non-Patent Citations (1)
| Title |
|---|
| 张林: "《矿区下一代网络体系结构和传输层关键技术研究》", 30 September 2016, 中国矿业大学出版社 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111600774A (en) * | 2020-05-13 | 2020-08-28 | 北京奇艺世纪科技有限公司 | Consumption delay determination method, system, device, equipment and readable storage medium |
| CN113190516A (en) * | 2021-05-26 | 2021-07-30 | 中国工商银行股份有限公司 | Data synchronization monitoring method and device |
| CN113190516B (en) * | 2021-05-26 | 2024-02-09 | 中国工商银行股份有限公司 | Data synchronization monitoring method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107526631B (en) | Task monitoring method, device, equipment and medium | |
| CN109327353B (en) | Service flow determination method and device and electronic equipment | |
| US9779406B2 (en) | User feature identification method and apparatus | |
| CN109117595B (en) | Thermal load prediction method and device, readable medium and electronic equipment | |
| CN110334013B (en) | Decision engine testing method and device and electronic equipment | |
| CN110837457B (en) | Task management method and device, electronic equipment and storage medium | |
| CN108243032B (en) | Method, device and equipment for acquiring service level information | |
| CN110647547A (en) | Consumption delay monitoring method and device, electronic equipment and computer readable storage medium | |
| CN112749013A (en) | Thread load detection method and device, electronic equipment and storage medium | |
| CN110743169B (en) | Anti-cheating method and system based on block chain | |
| CN111488289A (en) | Fault positioning method, device and equipment | |
| CN107515811B (en) | Method and device for acquiring power consumption of application program | |
| CN109189677B (en) | Test method and device for updating state of variable value | |
| CN109271453B (en) | Method and device for determining database capacity | |
| CN108920326B (en) | Method and device for determining time-consuming abnormity of system and electronic equipment | |
| CN110672123A (en) | Deviation correcting method and device for mileage | |
| CN110020264B (en) | Method and device for determining invalid hyperlinks | |
| CN111309576A (en) | Pressure testing method and device | |
| CN110795239A (en) | Application memory leakage detection method and device | |
| CN108154377B (en) | Advertisement cheating prediction method and device | |
| CN109615465B (en) | Service order processing method and device and electronic equipment | |
| CN106708605B (en) | Method and device for detecting resource leakage | |
| CN113761082A (en) | Data visualization method, device and system | |
| CN109597743A (en) | Page circle choosing method, click volume statistical method and relevant device | |
| CN113627940A (en) | Transaction amount abnormity warning method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: No.27, 1st floor, building 14, Haidian Road, zhongjiancai District, Beijing 100096 Applicant after: Beijing PERCENT Technology Group Co.,Ltd. Address before: 100096 101 / F, building 14, 27 Jiancai Middle Road, Haidian District, Beijing Applicant before: BEIJING BAIFENDIAN INFORMATION SCIENCE & TECHNOLOGY Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200103 |
|
| RJ01 | Rejection of invention patent application after publication |