CN114428893A - Public opinion real-time monitoring system based on stream-oriented processing - Google Patents

Abstract

The invention discloses a public opinion real-time monitoring system based on streaming processing, which avoids the calculation delay caused by timing scheduling calculation, solves the problem of real-time calculation which cannot be realized by traditional batch processing, and improves the real-time performance and efficiency of public opinion monitoring. The technical scheme is as follows: the system adopts a distributed real-time stream processing technology, particularly, a data bus module and a data real-time calculation module are introduced in the public opinion processing process, the timeliness of the whole public opinion monitoring is improved, and the public opinion monitoring effect is further improved.

Description

Public opinion real-time monitoring system based on stream-oriented processing

Technical Field

The invention relates to a public opinion monitoring system, in particular to a real-time monitoring system for mass public opinions based on a streaming processing technology.

Background

With the rapid development of the internet in China, public opinion monitoring work becomes a part of work content of governments and enterprises. The public opinion monitoring system mainly has the function of monitoring internet public information such as various major news media, mainstream portal websites, forums, blogs, online social network platforms, WeChat public numbers, digital newspapers, industry vertical sites, news clients and the like in real time.

Public opinion monitoring has important significance. Public sentiment can be discovered at the first time, through real-time monitoring of the whole-network public sentiment, the public sentiment can help a user to know various sounds related to the public sentiment on the network at the first time, and the user is helped to smoothly handle various sudden public sentiments. And secondly, public opinion work decision can be assisted, and through automatic identification and early warning of public opinions, the method is helpful for users to take corresponding measures according to negative information in a targeted manner, and provides reference for a decision maker to make a correct public opinion corresponding scheme. And the method can provide scientific decision basis for public opinion solution, is helpful for users to master the public opinion development context and deeply know the network public praise and brand image of the users through comprehensive analysis of the public opinions, thereby making effective prediction and providing scientific decision basis for public opinion solution. And finally, crisis loss can be avoided, and public opinion information can be timely found and development trend of public opinions can be analyzed by monitoring negative news of enterprise images and public monuments in real time, so that intervention guidance of the public opinions is facilitated, and the occurrence of public crisis is prevented.

At present, a traditional public opinion monitoring system in the market mainly adopts a timing batch processing mode, has certain time delay naturally, and cannot effectively meet the requirement of the real-time public opinion monitoring.

The whole data processing flow of the conventional public opinion monitoring system is shown in fig. 1. Firstly, mass original public opinion data are obtained through means of collection, purchase and the like, and the obtained original public opinion data are transmitted to a corresponding database through data transmission to be stored. The computing program is scheduled by a timing scheduler at regular intervals (usually on the order of hours) to read the public sentiment data to be computed from the database, and the public sentiment data is computed and counted. And then writing the calculation and statistical results back to the database for other programs to read. The front-end query program performs data query operation regularly (usually in the order of minutes), acquires the latest calculation and statistical result and displays the result to the operator.

As can be seen from the above flow, besides the inherent data acquisition and data transmission delays, two additional delays are introduced in the whole flow due to the timed batch operation, one is the delay introduced by the timing calculation (hour level) and the other is the delay introduced by the timing inquiry (minute level). The data is finally presented to the user from the acquisition, if a batch process is adopted, there is at least an hour-level delay.

Disclosure of Invention

The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

The invention aims to solve the problems and provides a public opinion real-time monitoring system based on streaming processing, which avoids the calculation delay caused by timing scheduling calculation, solves the problem of real-time calculation which cannot be realized by traditional batch processing, greatly improves the real-time performance and efficiency of public opinion monitoring, and can help business personnel to better realize the public opinion monitoring.

The technical scheme of the invention is as follows: the invention discloses a public opinion real-time monitoring system based on streaming processing, which comprises a data acquisition module, a data storage module, a data real-time pushing module, a data real-time calculating module and a data bus module, wherein:

the data bus module is used for providing a transmission channel for interaction among modules in the system, and realizing data transmission, data persistence and data distribution;

the data acquisition module is used for acquiring public opinion data in real time;

the data real-time calculation module is used for distributed real-time calculation of public opinion data;

the data storage module is used for persistent storage and query of data;

and the data real-time pushing module is used for receiving and pushing data in real time.

According to an embodiment of the streaming processing-based public opinion real-time monitoring system of the present invention, the data obtaining module is further configured to: the method comprises the steps of automatically collecting internet text information by using a distributed crawler technology, analyzing the captured internet text information by using an XML (extensive Makeup language) path language method, obtaining concerned information from the captured internet text information, and forming structured public opinion data.

According to an embodiment of the public opinion real-time monitoring system based on streaming processing, the data acquisition module adopts a task scheduling framework to trigger a crawler program at regular time to capture internet text information.

According to an embodiment of the streaming-based public opinion real-time monitoring system, the data bus module uses a distributed message system for data transmission, distribution and persistence.

According to an embodiment of the streaming-based public opinion real-time monitoring system of the present invention, the data bus module is further configured to: the data are received and stored persistently, and the forwarding and storage of the data are realized by utilizing a distributed high-availability technology, wherein the data bus module supports the publishing and subscribing functions of messages, a producer of the data publishes the data to the data bus module, and a consumer of the data subscribes consumption data from the data bus module.

According to an embodiment of the public opinion real-time monitoring system based on streaming processing of the present invention, the data real-time computing module employs a distributed streaming processing framework, and the data real-time computing module is further configured to: the data real-time calculation module subscribes data of the data bus module, stores calculation results in a data driving mode, and issues the calculation results to the data bus module after set logic conditions are met.

According to an embodiment of the public opinion real-time monitoring system based on streaming processing of the present invention, the data real-time computation module further includes a cluster management unit, a distributed storage unit and a data computation path, wherein the cluster management unit is configured to perform unified scheduling and management, the cluster management unit has functions including task submission, task scheduling, error recovery, resource allocation and authority control, the distributed storage unit is configured to perform persistent storage on an intermediate result generated in the data real-time computation module, and when a computation node fails, the computation result is recovered by the distributed storage unit, and the data computation path has a standardized processing architecture, including: the data processing system comprises a data source unit, a data preprocessing unit, a plurality of data computing units and a data collecting unit, wherein the data source unit is used for acquiring data from a data source including a data bus module, the data preprocessing unit is used for preprocessing, preprocessing and data splicing of the data, the data computing unit is used for computing, aggregating, sequencing and the like of the data, and the data collecting unit is used for collecting computing results from all the data computing units and carrying out global processing.

According to an embodiment of the public opinion real-time monitoring system based on streaming processing, the data storage module adopts a full-text search engine with distributed multi-user capability, and the data storage module is further configured to: the data storage module subscribes original data in the consumption data bus module and data calculated by the data real-time calculation module, and stores the subscribed data on a disk according to a predefined mode, wherein the predefined storage mode on the disk has a structure and is used for reversely indexing the stored public opinion data so as to facilitate subsequent word segmentation retrieval and single point query of the public opinion data.

According to an embodiment of the streaming processing-based public opinion real-time monitoring system of the present invention, the data real-time pushing module employs a layered one-stop lightweight open-source framework, and the data real-time pushing module is further configured to:

the data real-time pushing module subscribes original data of the data bus module and data calculated by the data real-time calculating module, and pushes the data to the front end for service display by using a websocket protocol after service logic processing.

Compared with the prior art, the invention has the following beneficial effects: the system of the invention introduces a distributed real-time stream processing technology into the system, and particularly introduces a data bus module and a data real-time calculation module in the public opinion processing process, thereby improving the timeliness of the whole public opinion monitoring and further improving the public opinion monitoring effect.

Drawings

The above features and advantages of the present disclosure will be better understood upon reading the detailed description of embodiments of the disclosure in conjunction with the following drawings. In the drawings, components are not necessarily drawn to scale, and components having similar relative characteristics or features may have the same or similar reference numerals.

Fig. 1 is a schematic diagram illustrating a data processing flow of a conventional public opinion monitoring system.

Fig. 2 is a schematic diagram illustrating a system architecture of an embodiment of a streaming-based public opinion real-time monitoring system according to the present invention.

Fig. 3 shows a schematic diagram of the internal framework of a data bus module in the embodiment of the system shown in fig. 1.

Fig. 4 shows a schematic diagram of the internal architecture of the data real-time computation module in the system embodiment shown in fig. 1.

Detailed Description

The invention is described in detail below with reference to the figures and specific embodiments. It is noted that the aspects described below in connection with the figures and the specific embodiments are only exemplary and should not be construed as imposing any limitation on the scope of the present invention.

Fig. 2 shows a system architecture of an embodiment of a streaming-based public opinion real-time monitoring system of the present invention. Referring to fig. 2, the public opinion real-time monitoring system of the present embodiment includes: the device comprises a data acquisition module, a data storage module, a data real-time pushing module, a data real-time calculating module and a data bus module.

The data bus module is located at the center of all modules in the system, and the modules are interacted with each other mainly through the data bus, namely the data bus module provides a transmission channel for the interaction among the modules in the system. Public sentiment data are mainly transmitted among the modules, other types of data can be transmitted, and the whole framework has no special requirements on the data types. The modules interact with each other in a data-driven manner.

The data acquisition module is used for the real-time collection of public sentiment data, and the inside concrete processing logic of module is: the distributed crawler technology crawler4j (an interface provided by an open source Java class library for grabbing Web pages) is used for automatically collecting massive Internet text information, and the information comprises but is not limited to: news, electronic newspapers, macroscopic data, self-media data, business data, administrative penalty data, market data, and the like. And analyzing the captured internet text information by using an XPath (XML path language) method, and acquiring concerned information from the captured internet text information to form structured public opinion data. A task scheduling framework of Quartz (an open source job scheduling framework written by Java) is adopted to trigger a crawler program at regular time to perform webpage crawling work, the scheduling cycle is short, and the crawling work of the whole public opinion has higher timeliness.

The data bus module realizes the functions of data transmission, data persistence and data distribution. The module uses Kafka (high throughput distributed publish-subscribe message system) distributed message system (the implementation of Kafka is only an example of the present invention, and is not limited to this, and may also be other types of data bus modules), and the specific processing logic inside the module is: and receiving data sent to the data bus module, performing persistent storage on the received data, and realizing efficient forwarding and safe storage of the data by using distributed high-availability technologies and the like. The module supports the publishing and subscribing functions of messages, a producer of data can publish data to the data bus module, and a consumer of data can subscribe to consume data from the data bus module. The data bus module is a central module for data transmission of the whole system, the internal architecture of the data bus module is shown in fig. 3, the data bus shields the difference of production and consumption speeds between a producer and a consumer, and the effect of data caching is achieved. Meanwhile, the client side of various programming languages is provided, interaction among all modules is facilitated through a data bus, and the decoupling effect is achieved.

The data real-time computing module realizes the distributed real-time computing function of public opinion data. The module introduces a Flink (distributed stream data stream engine written in Java and Scala) distributed stream processing framework (the implementation of the Flink is only an example of the present invention, and is not limited to this, and may be replaced by other types of stream processing frameworks), and the specific processing logic inside the module is as follows: the data real-time computing module subscribes data of the data bus module, and the data real-time computing module adopts a data driving mode, namely, each piece of subscribed data triggers one-time computing, and stores a computing result in the data real-time computing module, and the computing result is issued to the data bus module after a set logic condition is reached. The data real-time computing module is a system core module, adopts a distributed streaming processing mode, and greatly improves the data computing efficiency compared with a batch processing mode.

The internal architecture of the data real-time computation module is shown in fig. 4, the whole data real-time computation module is uniformly scheduled and managed by a cluster management unit, and the cluster management unit has functions including task submission, task scheduling, error recovery, resource allocation and authority control. The distributed storage unit is mainly used for persistently storing intermediate results generated in real-time data calculation, and when a calculation node fails, the calculation results can be recovered through the distributed storage unit, so that data loss caused by server failure is avoided. The whole architecture in the middle of the data real-time computation module is a data computation path, and the data computation path has a standardized processing architecture and comprises the following components: the system comprises a data source unit, a data preprocessing unit, a plurality of data calculating units and a data collecting unit, wherein the data source unit is used for acquiring data from a data bus or other data sources, the data preprocessing unit is used for preprocessing, data splicing and the like of the data, the data calculating unit is used for calculating, aggregating, sequencing and the like of the data, and the data collecting unit is used for collecting calculation results from all the data calculating units and carrying out global processing. The single units can be subjected to distributed expansion, and the effect of improving the computing efficiency is achieved by utilizing more computing resources. Each data computation unit may store its own computation state, including intermediate results of computations, underlying data, etc. The calculation state is stored in the server where each module is located, and is synchronized to the distributed storage unit at set time intervals, so that the purpose of persistent storage is achieved.

The data storage module realizes the functions of persistent storage and query of data. The module adopts an elastic search (distributed multi-user full-text search engine) storage engine, and the specific processing logic inside the module is as follows: the data storage module subscribes original data in the consumption data bus module and data calculated by the data real-time calculation module, and stores the subscribed data on a disk according to a predefined mode, the predefined storage mode on the disk has a good structure, the stored public opinion data can be reversely indexed, the follow-up participle retrieval of the public opinion data is facilitated, and the single-point query of the data is controlled to be in millisecond-level response.

The data real-time pushing module realizes the real-time receiving and pushing functions of data. The module adopts a Spring (layered one-stop lightweight open source framework) framework, and the specific processing logic inside the module is as follows: the data real-time pushing module subscribes original data of the data bus module and data calculated by the data real-time calculating module, and pushes the data to the front end for service display by utilizing a websocket (protocol for performing full duplex communication on a single TCP connection) technology after the data is processed by service logic. The data real-time pushing module adopts a websocket pushing mode to push data in real time, so that the foreground timed polling query is avoided, and the data display efficiency is greatly improved.

In addition, the application scene of public opinion monitoring aimed by the invention can be expanded to a certain extent, such as business scenes of transaction wind control, settlement wind control and the like in a financial transaction system.

While, for purposes of simplicity of explanation, the methodologies are shown and described as a series of acts, it is to be understood and appreciated that the methodologies are not limited by the order of acts, as some acts may, in accordance with one or more embodiments, occur in different orders and/or concurrently with other acts from that shown and described herein or not shown and described herein, as would be understood by one skilled in the art.

Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a web site, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk (disk) and disc (disc), as used herein, includes Compact Disc (CD), laser disc, optical disc, Digital Versatile Disc (DVD), floppy disk and blu-ray disc where disks (disks) usually reproduce data magnetically, while discs (discs) reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (9)

1. The utility model provides a public opinion real-time monitoring system based on STREAMING is handled, its characterized in that, the system includes data acquisition module, data storage module, the real-time propelling movement module of data, the real-time calculation module of data and data bus module, wherein:

the data bus module is used for providing a transmission channel for interaction among modules in the system, and realizing data transmission, data persistence and data distribution;

the data acquisition module is used for acquiring public opinion data in real time;

the data real-time calculation module is used for distributed real-time calculation of public opinion data;

the data storage module is used for persistent storage and query of data;

and the data real-time pushing module is used for receiving and pushing data in real time.

2. The streaming-based public opinion real-time monitoring system according to claim 1, wherein the data obtaining module is further configured to: the method comprises the steps of automatically collecting internet text information by using a distributed crawler technology, analyzing the captured internet text information by using an XML (extensive Makeup language) path language method, obtaining concerned information from the captured internet text information, and forming structured public opinion data.

3. The streaming-based public opinion real-time monitoring system as claimed in claim 2, wherein the data acquisition module uses a task scheduling framework to trigger a crawler program periodically to capture internet text information.

4. The streaming-based public opinion real-time monitoring system according to claim 1, wherein the data bus module uses a distributed message system for data transmission, distribution and persistence.

5. The streaming-based public opinion real-time monitoring system according to claim 4, wherein the data bus module is further configured to: the data are received and stored persistently, and the forwarding and storage of the data are realized by utilizing a distributed high-availability technology, wherein the data bus module supports the publishing and subscribing functions of messages, a producer of the data publishes the data to the data bus module, and a consumer of the data subscribes consumption data from the data bus module.

6. The streaming-based public opinion real-time monitoring system according to claim 1, wherein the data real-time computing module employs a distributed streaming framework, and the data real-time computing module is further configured to: the data real-time calculation module subscribes data of the data bus module, stores calculation results in a data driving mode, and issues the calculation results to the data bus module after set logic conditions are met.

7. The streaming processing-based public opinion real-time monitoring system according to claim 6, wherein the data real-time computing module further includes a cluster management unit, a distributed storage unit and a data computing path, wherein the cluster management unit is configured to perform unified scheduling and management, the cluster management unit has functions including task submission, task scheduling, error recovery, resource allocation and authority control, the distributed storage unit is configured to persistently store intermediate results generated in the data real-time computing module, and recover the computing results through the distributed storage unit when a computing node fails, and the data computing path has a standardized processing architecture, including: the data processing system comprises a data source unit, a data preprocessing unit, a plurality of data computing units and a data collecting unit, wherein the data source unit is used for acquiring data from a data source including a data bus module, the data preprocessing unit is used for preprocessing, preprocessing and data splicing of the data, the data computing unit is used for computing, aggregating, sequencing and the like of the data, and the data collecting unit is used for collecting computing results from all the data computing units and carrying out global processing.

8. The streaming-based public opinion real-time monitoring system according to claim 1, wherein the data storage module employs a distributed multi-user-capability full-text search engine, the data storage module further configured to: the data storage module subscribes original data in the consumption data bus module and data calculated by the data real-time calculation module, and stores the subscribed data on a disk according to a predefined mode, wherein the predefined storage mode on the disk has a structure and is used for reversely indexing the stored public opinion data so as to facilitate subsequent word segmentation retrieval and single point query of the public opinion data.

9. The streaming-based public opinion real-time monitoring system according to claim 1, wherein the data real-time pushing module adopts a layered one-stop lightweight open-source framework, and the data real-time pushing module is further configured to:

the data real-time pushing module subscribes original data of the data bus module and data calculated by the data real-time calculating module, and pushes the data to the front end for service display by using a websocket protocol after service logic processing.

Images