CN112506978A - Big data real-time processing method, device and equipment - Google Patents
Big data real-time processing method, device and equipment Download PDFInfo
- Publication number
- CN112506978A CN112506978A CN202011476441.0A CN202011476441A CN112506978A CN 112506978 A CN112506978 A CN 112506978A CN 202011476441 A CN202011476441 A CN 202011476441A CN 112506978 A CN112506978 A CN 112506978A
- Authority
- CN
- China
- Prior art keywords
- data
- message queue
- real
- consumption
- target data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims description 21
- 238000012545 processing Methods 0.000 claims abstract description 138
- 238000000034 method Methods 0.000 claims abstract description 42
- 230000002159 abnormal effect Effects 0.000 claims description 20
- 238000012544 monitoring process Methods 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 14
- 238000007405 data analysis Methods 0.000 abstract description 11
- 238000004364 calculation method Methods 0.000 description 19
- 238000010586 diagram Methods 0.000 description 8
- 238000004458 analytical method Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 7
- 230000000694 effects Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 230000005856 abnormality Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000010223 real-time analysis Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 238000013501 data transformation Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
Abstract
The application provides a method, a device and equipment for processing big data in real time. The method comprises the following steps: the server monitors the service platform using DTS. When the service data is generated in the service platform, the server acquires the service data through the DTS. And the server adds the service data acquired through the DTS into a consumption message queue. The server can use a Flink framework to realize consumption of the target data in the consumption message queue, and a processing result is obtained. The method improves the data analysis speed and improves the real-time processing efficiency of the data.
Description
Technical Field
The present application relates to computer technologies, and in particular, to a method, an apparatus, and a device for processing big data in real time.
Background
With the development of the internet, the proportion of the e-commerce in daily consumption is gradually increased. And with the development of e-commerce, the service data and log information generated on the service platform will increase continuously.
In the prior art, data in a database is generally acquired through a button for real-time analysis of data access information and log information, so that data analysis is realized.
However, as the business information and the access data increase, the existing data analysis method has the problem of high database access pressure, and further the analysis speed is easy to slow.
Disclosure of Invention
The application provides a big data real-time processing method, a big data real-time processing device and big data real-time processing equipment, which are used for solving the problem that the existing data analysis mode is high in database access pressure and low in analysis speed.
In a first aspect, the present application provides a big data real-time processing method, including:
acquiring target data, wherein the target data is service data generated on a service platform;
adding the target data into a consumption message queue, wherein the consumption message queue is used for storing the target data;
and processing the target data in the consumption queue to obtain a processing result.
Optionally, the acquiring target data includes:
monitoring platform service data in a database;
and when data in the platform service in the database changes, acquiring the changed service data.
Optionally, the adding the target data to a consumption message queue, where the consumption message queue is used to store the target data, includes:
determining the service type of the target according to the target data;
and adding the target data into a consumption message queue corresponding to the service type according to the service type.
Optionally, the processing target data in the consumption queue includes:
acquiring data to be processed, wherein the data to be processed is acquired from a consumption message queue according to a first-in first-out rule in the consumption message queue;
processing the data to be processed to obtain a processing result;
and adding the processing result into a second message queue, wherein the second message queue is used for storing the processing result of the data in the consumption queue.
Optionally, the method further comprises:
and displaying the processing result in the second message queue.
Optionally, the method further comprises:
judging whether the processing result is abnormal or not;
and when the processing result is abnormal, sending an abnormal early warning.
In a second aspect, the present application provides a big data real-time processing apparatus, including:
the acquisition module is used for acquiring target data, and the target data is service data generated on a service platform;
the queue module is used for adding the target data into a consumption message queue, and the consumption message queue is used for storing the target data;
and the processing module is used for processing the target data in the consumption queue to obtain a processing result.
Optionally, the acquiring target data includes:
the monitoring submodule is used for monitoring platform service data in the database;
and the first acquisition submodule is used for acquiring the changed service data when the data in the platform service in the database is changed.
Optionally, the adding the target data to a consumption message queue, where the consumption message queue is used to store the target data, includes:
the determining submodule is used for determining the service type of the target according to the target data;
and the distribution submodule is used for adding the target data into a consumption message queue corresponding to the service type according to the service type.
Optionally, the processing target data in the consumption queue includes:
the second acquisition submodule is used for acquiring data to be processed, and the data to be processed is acquired from the consumption message queue according to a first-in first-out rule in the consumption message queue;
the processing submodule is used for processing the data to be processed to obtain a processing result;
and the storage submodule is used for adding the processing result into a second message queue, and the second message queue is used for storing the processing result of the data in the consumption queue.
Optionally, the apparatus further comprises: and the display module is specifically used for displaying the processing result in the second message queue.
Optionally, the apparatus further comprises: the alarm module is specifically used for judging whether the processing result is abnormal or not; and when the processing result is abnormal, sending an abnormal early warning.
In a third aspect, the present application provides a server, comprising: a server and a display screen;
the server is used for monitoring a database of the service platform and realizing real-time processing of big data by executing any one of the possible design methods of the first aspect and the first aspect to obtain a real-time processing result;
and the display screen is used for displaying the real-time processing result.
In a fourth aspect, the present application provides a readable storage medium, in which a computer program is stored, and when the program is executed by at least one processor of a server, the server executes the real-time big data processing method in any one of the possible designs of the first aspect and the first aspect.
In a fifth aspect, the present application provides a computer program product comprising a computer program that when executed by a processor is configured to implement the method for real-time processing of big data in the first aspect and any one of the possible designs of the first aspect.
According to the big data real-time processing method, device and equipment, the business platform is monitored by using the DTS; when business data are generated in a business platform, the business data are obtained through a DTS; adding the service data acquired through the DTS into a consumption message queue; and the method for consuming the target data in the consumption message queue by using the Flink framework to obtain a processing result realizes the effects of improving the data analysis speed and improving the real-time processing efficiency of the data.
Drawings
In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 is a schematic system architecture diagram of a big data real-time processing system according to an embodiment of the present application;
fig. 2 is a flowchart of a big data real-time processing method according to an embodiment of the present application;
fig. 3 is a flowchart of another big data real-time processing method according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a big data real-time processing apparatus according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of another big data real-time processing apparatus according to an embodiment of the present application;
fig. 6 is a schematic hardware structure diagram of a big data real-time processing system according to an embodiment of the present application.
Detailed Description
To make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be clearly and completely described below with reference to the drawings in the present application, and it is obvious that the described embodiments are some, but not all embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
With the development of the internet, the proportion of the e-commerce in daily consumption is gradually increased. And with the development of e-commerce, the service data and log information generated on the service platform will increase continuously. Currently, a user usually implements analysis on service data on a service platform through data analysis. For example, the double 11 pin count screen in ali. The way of presenting the service data analysis result through a real-time dashboard is being adopted by more and more enterprises.
At present, a server usually adopts an offline analysis mode to analyze service data on a service platform. That is, the data index displayed on the large screen is generally the analysis result of the business data of the previous day. Obviously, the analysis mode does not meet the requirement of the user on real-time performance. In the prior art, data acquisition is usually performed from a database by using a button. However, as the amount of traffic increases, the amount of access to a database at the time of data writing is already large in itself. At this time, if the keytle is used to search the database in real time, it is obvious that the access pressure of the database is increased, and the performance of the database is affected. The real-time analysis method not only easily causes database abnormity, but also has the problems of low analysis speed and the like.
In order to solve the problems, the application provides a big data real-time processing method. According to the method and the device, an asynchronous message storage and real-time data synchronization are realized by using a framework system combining DTS, a message queue, Flink and kvStore, so that the database access amount is reduced, the database performance is ensured, and the data analysis speed is increased.
In this application, after the Service platform generates the Service Data, the server may use a Data Transformation Service (DTS) to synchronize the Data into the consumption message queue while writing the Data into the database. DTS is a set of tools that are used for data transfer and synchronization. The DTS may extract, transform, and merge data from different sources into one or more target locations. Moreover, the DTS also has the performance advantages of high performance, high reliability and high expansion. Wherein the consuming message queue is a distributed queue.
Specifically, the server may obtain service data generated by the service platform through the DTS. And the server sends the service data to a consumption message queue of a corresponding theme according to the service type of the service data. At this time, the server does not need to care whether the service data in the consumption message queue is consumed or not. Therefore, even if the service platform generates a large amount of service data, the consumption message queue can not be blocked, and the processing efficiency of the data in the service platform is ensured.
And the monitoring process of the consumption message queue monitors the consumption message queue, and when the service data is stored in the consumption message queue, the server acquires the service log message of the corresponding theme and inserts the service data into the database.
In the application, the server consumes the data in the consumption message queue in real time by using a Flink framework. The Flink is a framework and a distributed processing engine, is used for carrying out stateful computation on unbounded and bounded data streams, and has the characteristics of low delay and high throughput. The server utilizes the function of Flink flow type calculation/batch processing calculation to realize real-time consumption of data in the consumption message queue by controlling the time window of data analysis calculation, and a processing result is obtained.
In the present application, the server also uses a distributed memory database (kvStore) to update the processing result in real time. The server notifies the front-end web program of the update message. The large screen realizes the effect of displaying index data or performing abnormal early warning in real time through the page of the web program.
The technical solution of the present application will be described in detail below with specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.
Fig. 1 shows a system architecture diagram of a big data real-time processing system according to an embodiment of the present application. As shown in the figure, the DTS monitors the change of the service data in the service platform in real time. When the service data of the service platform changes, the DTS sends the changed service data to the consumption message queue. The Flink consumes the data in the consumption message queue and implements real-time processing of the data through stream processing. Flink sends the processing results to the second message queue. The second message queue is used for storing the processing result of the service data. And displaying the processing result in the second consumption queue on a large screen.
In the present application, a server is used as an execution subject to execute the big data real-time processing method of the following embodiments. Specifically, the execution subject may be a hardware device of the server, or a software application implementing the following embodiments in the server, or a computer-readable storage medium installed with a software application implementing the following embodiments.
Fig. 2 shows a flowchart of a big data real-time processing method according to an embodiment of the present application. On the basis of the embodiment shown in fig. 1, as shown in fig. 2, with a server as an execution subject, the method of this embodiment may include the following steps:
s101, target data is obtained, and the target data is service data generated on a service platform.
In this embodiment, the server monitors the service platform using the DTS. When the service data is generated in the service platform, the server acquires the service data through the DTS. The service platform can be an e-commerce platform. The business data can be business data such as transaction data, access data and comment data in the business platform.
S102, adding the target data into a consumption message queue, wherein the consumption message queue is used for storing the target data.
In this embodiment, the server adds the service data acquired through the DTS to the consumption message queue. In order to facilitate the processing of the target data S103, a plurality of consumption message queues may be included in the server. Wherein each consumption message queue may have a corresponding traffic type. Alternatively, the data in each consumption queue is used to calculate a data index.
And S103, processing the target data in the consumption queue to obtain a processing result.
In this embodiment, the server further includes a process for monitoring the queue of consumed messages. When the monitoring process monitors that target data which are not consumed exist in a certain consumption message queue, the monitoring process reminds a server to process the target data. The server may use the Flink framework to enable consumption of the target data in the consumption message queue. The data in the consumption message queue is read in a first-in first-out mode. The read data is consumed, i.e. processed, by a preset data index calculation method. And the server calculates the data in the consumption queue by a preset data index calculation method to obtain a processing result.
For example, when the data index is calculated in the statistical order total number, after the server obtains a target data, i.e., a business order, the server may accumulate the order in the current order total number in an accumulation manner.
After the processing of the target data is completed, the server also updates the processing result in real time by using the kvStore. The server notifies the front-end web program of the update message. The large screen realizes the effect of displaying index data or performing abnormal early warning in real time through the page of the web program.
According to the big data real-time processing method, the server uses the DTS to monitor the service platform. When the service data is generated in the service platform, the server acquires the service data through the DTS. And the server adds the service data acquired through the DTS into a consumption message queue. The server can use a Flink framework to realize consumption of the target data in the consumption message queue, and a processing result is obtained. In the application, the asynchronous operation of the database is realized by using the mode of matching the DTS and the consumption message queue, the data analysis speed is increased on the basis of ensuring the data processing in the service platform, and the real-time processing of the data is ensured.
Fig. 3 is a flowchart illustrating another method for processing big data in real time according to an embodiment of the present application. On the basis of the embodiments shown in fig. 1 to fig. 2, as shown in fig. 3, with a server as an execution subject, the method of the embodiment may include the following steps:
s201, platform service data in a database are monitored.
In this embodiment, the server monitors the service platform using the DTS.
S202, when data in the platform service in the database changes, the changed service data is obtained.
In this embodiment, when the server monitors that the data of the service platform changes, the server obtains the changed service data. The data change of the service receipt may include the service platform generating new data or the data in the service platform changing. The changed business data may include business data such as transaction data, access data, comment data, and the like.
S203, determining the service type of the target according to the target data.
In this embodiment, after acquiring the target data, the server determines the service type of the target data according to the content of the target data. The service type may be determined according to a product type, or the service type may be determined according to a consumption amount, or the service type may be determined according to a user, or the service type may be determined according to a data index. When the server generates the consumption message queue, a plurality of consumption message queues can be generated according to the service type, so that the target data can be calculated subsequently.
And S204, adding the target data into a consumption message queue corresponding to the service type according to the service type.
In this embodiment, after determining the service type of the target data, the server adds the target data to the corresponding consumption message queue.
S205, obtaining data to be processed, wherein the data to be processed is obtained from the consumption message queue according to a first-in first-out rule in the consumption message queue.
In this embodiment, when unprocessed data exists in the consumption message queue, the server uses the Flink framework to acquire the data to be processed from the consumption message queue. The data to be processed is target data added into the consumption message queue.
And S206, processing the data to be processed to obtain a processing result.
In this embodiment, after the server obtains the data to be processed, the server calculates the data index by using the data to be processed according to a preset data index calculation method. The data index may be updated based on the original data index. Alternatively, the data index may be a new data index calculated according to the data index calculation method.
After the server obtains one piece of data to be processed, the server can simultaneously obtain a plurality of data index calculation methods related to the data to be processed. The server adds the data to be processed into each data index calculation method one by one, and a plurality of new data indexes are obtained through calculation.
For example, one piece of data to be processed is a business order, and the business order is an order for the user a to purchase the commodity B. Wherein, the user A purchases 5 commodities B, and the total amount is 10 yuan. At this time, the data index calculation method may include a calculation method of order statistics, a calculation method of total consumption of the user a, a calculation method of total sales of the article B, a calculation method of average price of the article B, and the like.
And S207, adding the processing result into a second message queue, wherein the second message queue is used for storing the processing result of the data in the consumption queue.
In this embodiment, after the server obtains the processing result through calculation, the server adds the processing result to the second consumption queue. The second consumption queue is used for storing the processing result and waiting for the front-end web program to read the processing result.
Wherein the second consumption queue may be stored under the kvStore framework.
And S208, displaying the processing result in the second message queue.
In this embodiment, the server further includes a monitoring process, where the monitoring process is configured to monitor whether data in the second message queue is updated. When the data in the second message queue is updated, the monitoring process sends a notification to the web program. The front-end web program of the server reads the processing result from the second message queue according to the notification. And the front-end web program stores the processing result to a corresponding position according to the processing result, so that the processing result is displayed in the web page.
S209, judging whether the processing result has abnormity.
In this embodiment, after obtaining the processing result according to the calculation in S205, the server may further determine whether the processing result is abnormal according to a preset index threshold. Wherein the server has a different metric threshold for each data metric. After the server obtains the processing result, the index threshold corresponding to the processing result may be determined according to the processing result.
And S210, sending an abnormity early warning when the processing result is abnormal.
In this embodiment, when there is an exception in the processing result, the exception may include that the processing result is abnormally large, the processing result is abnormally small, the value of the processing result exceeds the index threshold, the value of the processing result is smaller than the index threshold, and the like. And the server sends an abnormity alarm to the user according to the abnormity. The user is a user who uses the server to analyze the service data. The information of the abnormal alarm may include a data index where an abnormality occurs, an abnormal value, target data calculated to obtain the processing result, and the like.
According to the big data real-time processing method, the server uses the DTS to monitor the service platform. When the server monitors that the data of the service platform changes, the server acquires the changed service data. And after acquiring the target data, the server judges the service type of the target data according to the content of the target data. And after determining the service type of the target data, the server adds the target data into a corresponding consumption message queue. When unprocessed data exists in the consumption message queue, the server acquires the data to be processed from the consumption message queue by using a Flink framework. And after the server acquires the data to be processed, calculating a data index by using the data to be processed according to a preset data index calculation method. And after the server acquires the data to be processed, calculating a data index by using the data to be processed according to a preset data index calculation method. When the data in the second message queue is updated, the monitoring process sends a notification to the web program to realize the presentation of the processing result in the web page. The server can also judge whether the processing result is abnormal according to a preset index threshold value. And when the processing result is abnormal, the server sends an abnormal early warning. In the application, the asynchronous operation of the database is realized by using the mode of matching the DTS and the consumption message queue, the data analysis speed is increased on the basis of ensuring the data processing in the service platform, and the real-time processing of the data is ensured. Meanwhile, the method and the device also send alarm information when the processing result is abnormal in an abnormal early warning mode, and help a user to improve data monitoring efficiency.
Fig. 4 shows a schematic structural diagram of a big data real-time processing device according to an embodiment of the present application, and as shown in fig. 4, the big data real-time processing device 10 according to this embodiment is used to implement operations corresponding to a server in any of the above method embodiments, where the big data real-time processing device 10 according to this embodiment includes:
the obtaining module 11 is configured to obtain target data, where the target data is service data generated on a service platform.
And the queue module 12 is used for adding the target data into a consumption message queue, and the consumption message queue is used for storing the target data.
And the processing module 13 is configured to process the target data in the consumption queue to obtain a processing result.
In one example, the big data real-time processing device 10 further includes: the alarm module 14 is specifically configured to determine whether an abnormality exists in the processing result. And when the processing result has abnormality, sending an abnormality early warning.
In one example, the big data real-time processing device 10 further includes: the display module 15 is specifically configured to display a processing result in the second message queue.
The real-time big data processing apparatus 10 provided in the embodiment of the present application may implement the method embodiment, and for concrete implementation principles and technical effects, reference may be made to the method embodiment, which is not described herein again.
Fig. 5 shows a schematic structural diagram of another big data real-time processing device according to an embodiment of the present application, and based on the embodiment shown in fig. 4, as shown in fig. 5, the big data real-time processing device 10 of the present embodiment is used for implementing operations corresponding to a server in any one of the method embodiments, and each module in the big data real-time processing device 10 of the present embodiment includes:
and the monitoring submodule 111 is used for monitoring platform service data in the database.
The first obtaining sub-module 112 is configured to obtain changed service data when data in the platform service in the database changes.
The determining submodule 121 is configured to determine a service type of the target according to the target data.
And the distributing submodule 122 is configured to add the target data to a consumption message queue corresponding to the service type according to the service type.
The second obtaining sub-module 131 is configured to obtain data to be processed, where the data to be processed is obtained from the consumption message queue according to a first-in first-out rule in the consumption message queue.
And the processing submodule 132 is configured to process the data to be processed to obtain a processing result.
The storage submodule 133 is configured to add the processing result to a second message queue, where the second message queue is used to store the processing result of the data in the consumption queue.
The real-time big data processing apparatus 10 provided in the embodiment of the present application may implement the method embodiment, and for concrete implementation principles and technical effects, reference may be made to the method embodiment, which is not described herein again.
Fig. 6 shows a hardware structure diagram of a big data real-time processing system according to an embodiment of the present application. As shown in fig. 6, the big data real-time processing system 20 is configured to implement the big data real-time processing method in any of the above method embodiments, and the big data real-time processing system 20 of this embodiment may include: server and display screen.
The server 21 is configured to monitor a database of the service platform, and implement real-time processing of big data by using the big data real-time processing method in the foregoing embodiment, so as to obtain a real-time processing result.
And the display screen 22 is used for displaying the real-time processing result. The display screen may be a real-time large screen.
The server may include, among other things, a memory, a processor, and a communication interface.
Wherein the memory is for storing a computer program. The Memory may include a Random Access Memory (RAM), and may further include a Non-Volatile Memory (NVM), such as at least one magnetic disk Memory, and may also be a usb disk, a removable hard disk, a read-only Memory, a magnetic disk or an optical disk.
The processor is used for executing the computer program stored in the memory to realize the real-time processing method of the big data in the above embodiment. Reference may be made in particular to the description relating to the method embodiments described above.
Alternatively, the memory may be separate or integrated with the processor.
The server 21 may also include a bus when the memory is a separate device from the processor. The bus is used to connect the memory and the processor. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, the buses in the figures of the present application are not limited to only one bus or one type of bus.
The communication interface can be connected with the processor through a bus and used for sending a real-time processing result of the processor to the display screen.
The server provided in this embodiment may be used to execute the above-mentioned big data real-time processing method, and the implementation manner and the technical effect are similar, which are not described herein again.
The present application also provides a computer-readable storage medium, in which a computer program is stored, and the computer program is used for implementing the methods provided by the above-mentioned various embodiments when being executed by a processor.
The computer-readable storage medium may be a computer storage medium or a communication medium. Communication media includes any medium that facilitates transfer of a computer program from one place to another. Computer storage media may be any available media that can be accessed by a general purpose or special purpose computer. For example, a computer readable storage medium is coupled to the processor such that the processor can read information from, and write information to, the computer readable storage medium. Of course, the computer readable storage medium may also be integral to the processor. The processor and the computer-readable storage medium may reside in an Application Specific Integrated Circuit (ASIC). Additionally, the ASIC may reside in user equipment. Of course, the processor and the computer-readable storage medium may also reside as discrete components in a communication device.
The computer-readable storage medium may be implemented by any type of volatile or nonvolatile Memory device or combination thereof, such as Static Random-Access Memory (SRAM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk or optical disk. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.
The present application also provides a program product comprising execution instructions stored in a computer-readable storage medium. The at least one processor of the device may read the execution instructions from the computer-readable storage medium, and the execution of the execution instructions by the at least one processor causes the device to implement the methods provided by the various embodiments described above.
Embodiments of the present application further provide a chip, which includes a memory and a processor, where the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that a device in which the chip is installed executes the method in the above various possible embodiments.
It should be understood that the Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present invention may be embodied directly in a hardware processor, or in a combination of the hardware and software modules within the processor.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of modules is merely a division of logical functions, and an actual implementation may have another division, for example, a plurality of modules may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.
Modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may exist alone physically, or two or more modules are integrated into one unit. The unit formed by the modules can be realized in a hardware form, and can also be realized in a form of hardware and a software functional unit.
The integrated module implemented in the form of a software functional module may be stored in a computer-readable storage medium. The software functional module is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device) or a processor to execute some steps of the methods according to the embodiments of the present application.
Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. Which when executed performs steps comprising the method embodiments described above. And the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same. Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: it is also possible to modify the solutions described in the previous embodiments or to substitute some or all of them with equivalents. And the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.
Claims (10)
1. A big data real-time processing method is characterized by comprising the following steps:
acquiring target data, wherein the target data is service data generated on a service platform;
adding the target data into a consumption message queue, wherein the consumption message queue is used for storing the target data;
and processing the target data in the consumption queue to obtain a processing result.
2. The method of claim 1, wherein the obtaining target data comprises:
monitoring platform service data in a database;
and when data in the platform service in the database changes, acquiring the changed service data.
3. The method of claim 1, wherein the adding the target data to a consumption message queue, the consumption message queue for storing the target data, comprises:
determining the service type of the target according to the target data;
and adding the target data into a consumption message queue corresponding to the service type according to the service type.
4. The method of claim 1, wherein the processing target data in the consumption queue comprises:
acquiring data to be processed, wherein the data to be processed is acquired from a consumption message queue according to a first-in first-out rule in the consumption message queue;
processing the data to be processed to obtain a processing result;
and adding the processing result into a second message queue, wherein the second message queue is used for storing the processing result of the data in the consumption queue.
5. The method according to any one of claims 1-4, further comprising:
and displaying the processing result in the second message queue.
6. The method according to any one of claims 1-4, further comprising:
judging whether the processing result is abnormal or not;
and when the processing result is abnormal, sending an abnormal early warning.
7. A big data real-time processing device, which is characterized in that the device comprises:
the acquisition module is used for acquiring target data, wherein the target data is platform service data which changes in a database;
the queue module is used for adding the target data into a message queue, and the message queue is used for storing the target data to be processed;
and the processing module is used for processing the target data in the consumption queue to obtain a processing result.
8. A big data real-time processing system, characterized in that, the system includes: a server and a display screen;
the server is used for monitoring a database of a service platform and realizing real-time processing of big data by the big data real-time processing method of any one of claims 1 to 6 to obtain a real-time processing result;
and the display screen is used for displaying the real-time processing result.
9. A computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, is configured to implement the big data real-time processing method according to any one of claims 1 to 6.
10. A computer program product, characterized in that the computer program product comprises a computer program which, when being executed by a processor, is adapted to carry out the big data real-time processing method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011476441.0A CN112506978A (en) | 2020-12-15 | 2020-12-15 | Big data real-time processing method, device and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011476441.0A CN112506978A (en) | 2020-12-15 | 2020-12-15 | Big data real-time processing method, device and equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112506978A true CN112506978A (en) | 2021-03-16 |
Family
ID=74973498
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011476441.0A Pending CN112506978A (en) | 2020-12-15 | 2020-12-15 | Big data real-time processing method, device and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112506978A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113626163A (en) * | 2021-07-20 | 2021-11-09 | 上海泛宥信息科技有限公司 | Lightweight distributed increment self-scheduling method, system, equipment and medium |
| CN113656517A (en) * | 2021-09-03 | 2021-11-16 | 中国联合网络通信集团有限公司 | Account checking file processing method and device, electronic equipment and storage medium |
| CN116132540A (en) * | 2023-04-13 | 2023-05-16 | 北京东大正保科技有限公司 | Multi-service system data processing method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109672627A (en) * | 2018-09-26 | 2019-04-23 | 深圳壹账通智能科技有限公司 | Method for processing business, platform, equipment and storage medium based on cluster server |
| CN111061804A (en) * | 2019-10-30 | 2020-04-24 | 平安科技（深圳）有限公司 | Asynchronous data processing method, device, equipment and storage medium based on big data |
| CN111143415A (en) * | 2019-12-26 | 2020-05-12 | 政采云有限公司 | Data processing method and device and computer readable storage medium |
| CN111209310A (en) * | 2020-04-23 | 2020-05-29 | 太平金融科技服务（上海）有限公司 | Service data processing method and device based on stream computing and computer equipment |
| CN111221793A (en) * | 2019-12-31 | 2020-06-02 | 新华网股份有限公司 | Data mining method, platform, computer equipment and storage medium |
-
2020
- 2020-12-15 CN CN202011476441.0A patent/CN112506978A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109672627A (en) * | 2018-09-26 | 2019-04-23 | 深圳壹账通智能科技有限公司 | Method for processing business, platform, equipment and storage medium based on cluster server |
| CN111061804A (en) * | 2019-10-30 | 2020-04-24 | 平安科技（深圳）有限公司 | Asynchronous data processing method, device, equipment and storage medium based on big data |
| CN111143415A (en) * | 2019-12-26 | 2020-05-12 | 政采云有限公司 | Data processing method and device and computer readable storage medium |
| CN111221793A (en) * | 2019-12-31 | 2020-06-02 | 新华网股份有限公司 | Data mining method, platform, computer equipment and storage medium |
| CN111209310A (en) * | 2020-04-23 | 2020-05-29 | 太平金融科技服务（上海）有限公司 | Service data processing method and device based on stream computing and computer equipment |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113626163A (en) * | 2021-07-20 | 2021-11-09 | 上海泛宥信息科技有限公司 | Lightweight distributed increment self-scheduling method, system, equipment and medium |
| CN113656517A (en) * | 2021-09-03 | 2021-11-16 | 中国联合网络通信集团有限公司 | Account checking file processing method and device, electronic equipment and storage medium |
| CN113656517B (en) * | 2021-09-03 | 2023-06-06 | 中国联合网络通信集团有限公司 | Account checking file processing method and device, electronic equipment and storage medium |
| CN116132540A (en) * | 2023-04-13 | 2023-05-16 | 北京东大正保科技有限公司 | Multi-service system data processing method and device |
| CN116132540B (en) * | 2023-04-13 | 2023-08-01 | 北京东大正保科技有限公司 | Multi-service system data processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112506978A (en) | Big data real-time processing method, device and equipment | |
| CN106407190B (en) | Event record query method and device | |
| WO2022267735A1 (en) | Service data processing method and apparatus, computer device, and storage medium | |
| CN107679081B (en) | Information modification method and device, computer equipment and computer readable storage medium | |
| US9292336B1 (en) | Systems and methods providing optimization data | |
| CN106202081B (en) | Real-time data processing method and device | |
| JP2013545209A (en) | Method and apparatus for managing orders in a financial market | |
| CN108492150B (en) | Method and system for determining entity heat degree | |
| CN111339073A (en) | Real-time data processing method and device, electronic equipment and readable storage medium | |
| CN111177563A (en) | Information recommendation method and device, electronic equipment and storage medium | |
| RU2622850C2 (en) | Method and server for processing product identifiers and machine-readable storage medium | |
| CN108875048B (en) | Report generation method and device, electronic equipment and readable storage medium | |
| CN108932241B (en) | Log data statistical method, device and node | |
| CN108428138B (en) | Customer survival rate analysis device and method based on customer clustering | |
| WO2019019596A1 (en) | Breakpoint list processing method, device, server and medium | |
| CN112102040A (en) | Global inventory control method and system in distributed environment | |
| CN114218303B (en) | Transaction data processing system, processing method, medium and equipment | |
| CN115578105A (en) | Method and device for processing transaction data | |
| CN116361571A (en) | Artificial intelligence-based merchant portrait generation method, device, equipment and medium | |
| CN113204467B (en) | Method, device, equipment and storage medium for monitoring online service system | |
| CN114358673A (en) | Abnormal logistics early warning method and device, computer equipment and storage medium | |
| CN114281807A (en) | Data quality auditing method, device, equipment and storage medium | |
| CN110148044B (en) | Method and device for setting buffering threshold for accounting | |
| CN112131276A (en) | Data statistics method, electronic equipment and readable storage medium | |
| CN109214874B (en) | IP product operation data processing method, device, equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20210316 |
|
| WD01 | Invention patent application deemed withdrawn after publication |