CN111984433A

CN111984433A - Business data processing method, display method, device, electronic equipment and medium

Info

Publication number: CN111984433A
Application number: CN202010762023.1A
Authority: CN
Inventors: 李文学; 史忠伟
Original assignee: Wuba Co Ltd
Current assignee: Wuba Co Ltd
Priority date: 2020-07-31
Filing date: 2020-07-31
Publication date: 2020-11-24

Abstract

The embodiment of the invention provides a business data processing method, a business data display device, electronic equipment and a storage medium, wherein the processing method comprises the following steps: acquiring service data in real time; storing the service data into a corresponding message queue; performing data processing on the service data in each message queue according to the time dimension to obtain a data characteristic value in corresponding time; and sending the data characteristic value in the corresponding time to a first intermediate storage container for storage, so that when the first intermediate storage container receives a request for accessing the data value by the online service system or the third-party service system, the first intermediate storage container calculates the data value in the corresponding time window according to the data characteristic value and feeds the data value back to the online service system or the third-party service system. Therefore, efficient access support is provided for the online service system or the third-party service system, and the real-time calculation accuracy of the online service system or the third-party service system is improved.

Description

Business data processing method, display method, device, electronic equipment and medium

Technical Field

The present invention relates to the field of network technologies, and in particular, to a service data processing method, a service data display method, an apparatus, an electronic device, and a storage medium.

Background

In the prior art, open-source real-time computation framework components such as Spark Streaming, Flink, Storm and the like are generally used for performing real-time index computation. The general idea of these open source frameworks is to monitor the message queue data in real time and calculate the data in the unit time window according to a fixed sliding step. Particularly, a user who acquires a large amount of information of a website by using a crawler needs to be intercepted, user behavior data are generally required to be collected in real time aiming at the current website service scene, the number of pages browsed by the user is calculated in real time, and when the number of pages visited by the same user in a certain time period is found to be larger than a certain threshold (such as 500 pages and the like), the user needs to be intercepted in time.

The following description will be given by taking the real-time calculation using Spark Streaming as an example. If the time window is 1 hour, the maximum threshold for the number of web pages visited by the user is 500 pages, and the sliding step is 15 minutes, then Saprk Streaming will calculate the value of 1 last 1 hour every 15 minutes, if the current number of pages viewed by the user at 10:00 is 499 pages, so 10: the time point 00 shows no exception, when the number of browsed web pages reaches 600 pages when the time point reaches 10:10, the current system only stores the value of 10:00, and 10:15 is the next calculation time point, that is, at this time, although the number of browsed web pages of the user reaches the maximum threshold value, the calculation point of the next time window is not reached, and all the systems do not process the response, that is, the user is not intercepted.

Therefore, in the related art, for the number of web pages browsed by the user, errors occur in the calculation of the data amount due to the mismatching of the calculation time and the event time, so that the accuracy of the real-time data calculation amount is reduced.

Disclosure of Invention

The technical problem to be solved by the embodiments of the present invention is to provide a service data processing method and a service data display method, so as to solve the technical problem in the prior art that an error occurs in a data calculation amount due to mismatching of calculation time and event time, which causes inaccuracy in a real-time data calculation amount.

Correspondingly, the embodiment of the invention also provides a business data processing device and a business data display device, which are used for ensuring the realization and application of the method.

In order to solve the problems, the invention is realized by the following technical scheme:

a first aspect provides a service data processing method, including:

acquiring service data in real time;

storing the service data into a corresponding message queue;

performing data processing on the service data in each message queue according to the time dimension to obtain a data characteristic value in corresponding time;

and sending the data characteristic value in the corresponding time to a first intermediate storage container for storage, so that when the first intermediate storage container receives a request for accessing the data value by an online service system or a third-party service system, the first intermediate storage container calculates the data value in the corresponding time window according to the data characteristic value and feeds the data value back to the online service system or the third-party service system.

Optionally, after performing data processing on the service data in each message queue according to the time dimension to obtain a data characteristic value in a corresponding time, the method further includes:

performing accumulation calculation and/or deduplication calculation on the data characteristic values in the corresponding time;

and sending the result of the accumulation calculation and/or the deduplication calculation to a second intermediate storage container for storage, so that when the second intermediate storage container receives a request of an online service system for accessing a data value, the second intermediate storage container calculates the data value in a corresponding time window according to the result of the accumulation calculation and/or the deduplication calculation, and feeds back the data value to the online service system or a third-party service system.

Optionally, the performing accumulation calculation and/or deduplication calculation on the data characteristic values in the corresponding time includes:

respectively sending the data characteristic values in corresponding time to a first delay queue consisting of a Kaffka structure, wherein the first delay queue stores the data characteristic values according to corresponding timestamps and then informs a first computing center to carry out accumulation calculation on the data characteristic values, the received data characteristic values are subjected to expiration management through the first delay queue, the expired data characteristic values are sent to the first computing center, and the first computing center subtracts the expired data characteristic values from the accumulation calculation results according to the expiration management; and/or

Respectively sending the data characteristic values in the corresponding time to a sequence queue formed by an intermediate storage container set and a second delay queue formed by a Kafka of a Kaffka, wherein the sequence queue and the second delay queue respectively store the data characteristic values according to corresponding time stamps, the sequence queue cuts the stored data characteristic values according to the size of a window period, and after the cutting is finished, a second computing center is informed to compute the number of the residual data characteristic values in the current sequence queue; and when the sequence queue receives an expired message sent by the second delay queue, the sequence queue cuts the expired message according to the window period, and after the cutting is finished, the sequence queue informs the second computing center to compute the number of the residual data characteristic values in the current queue.

A second aspect provides a service data display method, including:

receiving an inquiry request for inquiring the online service operation of a user;

according to the query request, requesting a data value of the current online service operation of the user from an intermediate storage container; when the intermediate storage container receives the query request, calculating a data value of the user online service operation in the current time window according to the stored data characteristic value in each time, and feeding back the data value;

receiving a data value of the user online service operation sent in the current time window in the intermediate storage container;

and displaying the data value of the online service operation of the user so as to determine whether to process the online service operation of the user according to the data value.

A third aspect provides a service data processing apparatus, including:

the acquisition module is used for acquiring service data in real time;

the first sending module is used for sending the service data to a corresponding message queue;

the processing module is used for carrying out data processing on the service data in each message queue according to the time dimension to obtain a data characteristic value in corresponding time;

and the second sending module is used for sending the data characteristic value obtained by the processing module within the corresponding time to a first intermediate storage container for storage, so that when the first intermediate storage container receives a request for accessing the data value by an online service system or a third-party service system, the first intermediate storage container calculates the data value within the corresponding time window according to the data characteristic value and feeds the data value back to the online service system or the third-party service system.

Optionally, the apparatus further comprises:

the computing module is used for performing accumulation computation and/or duplicate removal computation on the data characteristic values in the corresponding time after the processing module obtains the data characteristic values in the corresponding time;

and the third sending module is used for sending the result of the accumulation calculation and/or the deduplication calculation calculated by the calculating module to a second intermediate storage container for storage, so that when the second intermediate storage container receives a request of an online service system for accessing a data value, the second intermediate storage container calculates the data value in a corresponding time window according to the result of the accumulation calculation and/or the deduplication calculation, and feeds the data value back to the online service system or a third-party service system.

Optionally, the calculation module includes: a fourth sending module and an accumulation calculating module; and/or, a fifth sending module and a deduplication calculation module, wherein,

the fourth sending module is configured to send the data feature values in the corresponding time to a first delay queue composed of kafka, and the first delay queue stores the data feature values according to corresponding time stamps and then notifies the first computing center;

the accumulation calculation module is configured to perform accumulation calculation on the data characteristic value through the first calculation center, perform expiration management on the received data characteristic value through the first delay queue, send the expired data characteristic value to the first calculation center, and subtract the expired data characteristic value from an accumulation calculation result by the first calculation center;

the fifth sending module is configured to send the data eigenvalues within the corresponding time to a sequential queue composed of a set of intermediate storage containers and a second delay queue composed of kafka of the kaffka, respectively; storing the data characteristic values by the sequence queue and the second delay queue according to corresponding time stamps respectively;

the duplicate removal calculation module is used for cutting the stored data characteristic values according to the window period size through the sequence queue and informing the second calculation center to calculate the number of the residual data characteristic values in the current sequence queue after cutting; and when the sequence queue receives an expired message sent by the second delay queue, the sequence queue cuts the expired message according to the window period, and after the cutting is finished, the sequence queue informs the second computing center to compute the number of the residual data characteristic values in the current queue.

A fourth aspect provides a service data display apparatus, including:

the first receiving module is used for receiving an inquiry request for inquiring the online service operation of a user;

the request module is used for requesting the data value of the current online service operation of the user from the intermediate storage container according to the query request; when the intermediate storage container receives the query request, calculating a data value of the user online service operation in the current time window according to the stored data characteristic value in each time, and feeding back the data value;

a second receiving module, configured to receive a data value of the user online service operation sent in the current time window in the intermediate storage container;

and the display module is used for displaying the data value of the online business operation of the user so as to determine whether to process the online business operation of the user according to the data value.

A fifth aspect provides an electronic device comprising: the system comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the computer program realizes the business data processing method or the business data presentation method when being executed by the processor.

A sixth aspect provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a business data processing method as described above or implements a business data presentation method as described above.

A seventh aspect provides a computer program product, wherein instructions of the computer program product, when executed by a processor of an electronic device, cause the electronic device to execute a method for processing business data or a method for presenting business data as described above.

Compared with the prior art, the embodiment of the invention at least comprises the following advantages:

in the embodiment of the invention, in order to quickly and accurately respond to the requirement of the on-line service system or the third-party service system on the real-time index, the client or the terminal collects the service data in real time and stores the collected service data into the corresponding message queue, processing and calculating the data in the message queue in real time, storing the calculated data characteristic value in the corresponding time in an intermediate storage container which can support high-efficiency access, such that upon receiving a request to access a data value sent by the online service system or a third-party service system, calculating the final data value according to the data characteristic value corresponding to the time window, synchronously returning the calculated data value to the corresponding online service system or the third-party service system, therefore, efficient access support is provided for the online service system or the third-party service system, and the real-time calculation accuracy of the online service system or the third-party service system is improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

Fig. 1 is a flowchart of a service data processing method according to an embodiment of the present invention;

fig. 2 is a schematic diagram of an application example of a service data processing method according to an embodiment of the present invention;

fig. 3 is another flowchart of a method for processing service data according to an embodiment of the present invention;

fig. 4 is a schematic diagram of another application example of a service data processing method according to an embodiment of the present invention;

fig. 5 is a flowchart of a service data presentation method according to an embodiment of the present invention;

fig. 6 is a schematic structural diagram of a service data processing apparatus according to an embodiment of the present invention;

fig. 7 is another schematic structural diagram of a service data processing apparatus according to an embodiment of the present invention;

fig. 8 is a schematic structural diagram of a service data presentation apparatus according to an embodiment of the present invention.

Detailed Description

In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.

Before understanding the present invention, technical terms related to the present invention are introduced, specifically as follows:

and (3) calculating in real time: the present invention refers to real-time streaming computing in the field of big data, wherein the real-time streaming computing may also be referred to as real-time computing or streaming computing, and the concepts in the field of big data are similar.

Time window: the method refers to a method for measuring the average value of the average values: 10 to 10, etc., the length of which can be set as desired.

Sliding step length: the length of the sliding window, the time interval during which the window operation is performed. A window operation is performed every batch processing interval.

Message queue: which may also be referred to as message middleware, is a container that holds messages during their transmission. Common message queues may be some open source components such as Kafka, RabbitMQ, MetaQ, and the like.

Referring to fig. 1, a flowchart of a service data processing method according to an embodiment of the present invention is shown in the prior art for understanding the technical terms, and specifically includes the following steps:

step 101: acquiring service data in real time;

step 102: storing the service data into a corresponding message queue;

step 103: performing data processing on the service data in each message queue according to the time dimension to obtain a data characteristic value in corresponding time;

step 104: and sending the data characteristic value in the corresponding time to a first intermediate storage container for storage, so that when the first intermediate storage container receives a request for accessing the data value by an online service system or a third-party service system, the first intermediate storage container calculates the data value in the corresponding time window according to the data characteristic value and feeds the data value back to the online service system or the third-party service system.

The service data processing method provided by the embodiment of the invention can be applied to a mobile terminal, a server, a client, a back-end or online service system or a third-party service system and the like, and is not limited herein.

The following describes in detail specific implementation steps of the service data processing method provided in the embodiment of the present invention with reference to fig. 1.

First, step 101 is executed to obtain service data in real time.

The service data in this step may be user behavior data collected in real time by the background, for example, the number of pages browsed by the user is collected in real time. For example, the behavior data of the user can be monitored in real time by using an open source real-time framework component such as Spark Streaming, Flink, Storm, etc.

Next, step 102 is executed to store the service data in the corresponding message queue.

In this step, the service data collected in real time is stored in a corresponding message queue, for example, subscription messages are stored in a corresponding subscription message queue, for example, consumption messages are stored in a corresponding consumption message queue, and the like. Each message queue, among others, may consist of a Kafka (Kafka), a high-throughput distributed publish-subscribe messaging system that handles all the action flow data of consumers in a web site. Such as web browsing, searching and other user activity data, which is typically addressed by processing logs and log aggregations due to throughput requirements. The data stored in kafka only occur sequentially, and the deletion strategy of the data is to be accumulated to a certain degree or deleted after a certain time.

The purpose of Kafka message queuing is to unify online and offline message processing through the Hadoop parallel load mechanism, and also to provide real-time messages through clustering. The Kafka message queue is used for storing the business data information of the consumers acquired in real time.

And step 103 is executed, and data processing is performed on the service data in each message queue according to the time dimension to obtain a data characteristic value in the corresponding time.

In this step, the time dimension is a measurement scale which takes time as a description and expresses variables. That is, the unit of time represents the user's behavior data over time (year, month, day, hour, minute, second, etc.). It should be noted that the time dimension itself has two dimensions, one is a hierarchy (along the vertical direction) and the other is time (along the horizontal direction).

In this step, it is necessary to analyze the service data in each message queue, extract the dimension and index to be calculated from the service data according to the time dimension, and perform calculation to obtain the data feature value in the current time, for example, analyze the data in the Topic (Topic) or consumption queue of the subscription message queue in real time, extract the dimension (i.e., an identifier used to represent the identity of the user, etc.) to be calculated from the original data and calculate the data corresponding to the index (e.g., the IP of the user, etc.) to obtain the data feature value in the corresponding time, and then execute step 104, and send the data feature value of the extracted dimension and index to the first intermediate storage container.

And finally, executing step 104, sending the data characteristic value in the corresponding time to a first intermediate storage container for storage, so that when the first intermediate storage container receives a request for accessing the data value by the online service system or the third-party service system, the first intermediate storage container calculates the data value in the corresponding time window according to the data characteristic value, and feeds the data value back to the online service system or the third-party service system.

In this step, the first intermediate storage container may be composed of Redis, or may be simply referred to as a first Redis. Redis is an open-source, network-supported, memory-based, optional-persistence-based, high-performance key-value pair storage database written using ANSI C; alternatively, Redis is an open source, in-memory stored data structure server that can be used as a database, cache, and message queue agent. It supports data types such as strings, hash tables, lists, collections, ordered collections, bitmaps, Hyperlogs, etc. The Redis can support an intermediate storage container with efficient access, and provides efficient access support for an online business system or a third-party business system.

In this embodiment, the first Redis includes a Zset (referred to as a Redis Zset set or a Redis Zset structure), and the received data feature values in the corresponding time are sequentially stored according to a time sequence by the Zset structure, for example, the data feature values stored in the corresponding time by the Zset structure are sequentially: the time is time-1, the corresponding data characteristic value is k-1, the time is time-2, the corresponding data characteristic value is k-2, the time is time-3, the corresponding data characteristic value is k-3 and the like.

When the first Redis receives a request for accessing the data value by the online service system or the third-party service system, the first Redis calculates the data value in the corresponding time window according to the data characteristic value, and feeds the data value back to the online service system or the third-party service system, so that efficient access support is provided for the online service system or the third-party service system.

For convenience of understanding, please refer to fig. 2 together, which is a schematic diagram of an application example of the service data processing method provided in the embodiment of the present invention, as shown in fig. 2, if three data, k-3, k-4, and k-5, are stored in the message queue (taking the three data as an example, in an actual application, the three data are not limited to this), the three data in the message queue are sequentially analyzed according to a time dimension to obtain data header characteristic values, such as k-1, k-2, and k-3, in a corresponding time, and then, the data characteristic value corresponding to time-1 is k-1 (time-1 to k-1 may be used to represent a corresponding relationship between the two, or may be referred to as a key value pair); the data characteristic value corresponding to the time-2 is k-2 (the time-2-k-2 can be used for representing the corresponding relation between the two); the data characteristic value corresponding to the time-3 is k-3 (the corresponding relation between the time-3 and the k-3 can be represented by the time-3 and the k-3) and is sequentially stored in a Redis Zset structure according to time, the Zset is an upgrade version of the Set, a sequence attribute is added on the basis of the Set, the Set is a Set, and the concept of the Set is the combination of a stack of non-repeated values. Some collective data may be stored using the Set data structure provided by Redis.

When the Redis receives a request for accessing a data value sent by other online service systems or third-party service systems, the Redis calculates a final data value according to the data characteristic value corresponding to the time window, and synchronously returns the calculated data value to the corresponding online service system or third-party service system, so that efficient access support is provided for the online service system or third-party service system, and the real-time calculation accuracy of the online service system or third-party service system is improved.

It should be noted that the service data processing method provided by the foregoing embodiment is suitable for a case where the online access amount is not very large, and if the online service access amount is not high, the number is large, and the time window period is long, the Redis load is too high, and the response efficiency is high.

Referring to fig. 3, another flowchart of a service data processing method according to an embodiment of the present invention is shown, where the method includes:

step 301: acquiring service data in real time;

step 302: storing the service data into a corresponding message queue;

step 303: performing data processing on the service data in each message queue according to the time dimension to obtain a data characteristic value in corresponding time;

it should be noted that steps 301 to 303 are the same as steps 101 to 103, and the specific implementation process thereof is described in detail in the corresponding steps in the above embodiments, which is not described herein again.

Step 304: performing accumulation calculation and/or deduplication calculation on the data characteristic values in the corresponding time;

in this step, in one case, according to a user requirement, performing an accumulation calculation, that is, respectively sending the data feature values in the corresponding time to a first delay queue composed of kafka structures, the first delay queue storing the data feature values according to corresponding timestamps, then notifying the first calculation center to perform the accumulation calculation on the data feature values, performing expiration management on the received data feature values through the first delay queue, sending the expired data feature values to the first calculation center, and subtracting the expired data feature values from the result of the accumulation calculation by the first calculation center. It should be noted that the first delay queue is used to immediately transmit the received data (i.e. the data characteristic value) to the first computing center, and the first computing center transmits the data again according to the time delay of the computing time window.

For example, the number of downloads of website data is counted, and this type of calculation does not need to store process data to reduce resource overhead, and only needs to accumulate the counts according to a time window, so a delay queue method may be adopted, that is, data in each time period is delayed for a specified time and then sent back to a calculation center, when new data is received, an accumulator is used to add the data to a count value, and at the same time, the data is delayed for one window period in the delay queue. When the computing center receives the data returned by the delay queue, the accumulator is used to add or accumulate the corresponding negative value.

In another case, according to the user requirement, performing deduplication calculation, that is, sending the data feature values in the corresponding time to a sequence queue composed of intermediate storage container sets (such as Redis Zset combinations or structures) and a second delay queue composed of kafka structures, respectively, where the sequence queue and the second delay queue store the data feature values according to corresponding timestamps, and the sequence queue clips the stored data feature values according to the window period size, and notifies a second calculation center to calculate the number of the remaining data feature values in the current sequence queue after the clipping is completed; and when the sequence queue receives an expired message sent by the second delay queue, the sequence queue cuts the expired message according to the window period size, and informs the second computing center to compute the number of the residual K values in the current queue after cutting.

For example, calculations of duplicate values, such as statistical UV, maximum, minimum, etc., need to be removed. This method uses a combination of delay queues (i.e. a second delay queue, for distinguishing from the first delay queue) and a sequential queue, and the functions of the second delay queue are the same as those of the first delay queue mentioned above, and the difference in this embodiment is to reduce the resource overhead of the delay queues, and it is not necessary to store detailed data, but only one signal is stored. The sequential queue is realized by a Redis Zset structure, a part of details of received data is maintained in the sequential queue, and the principle of the queue is first-in first-out and queue insertion is not allowed. When new data is received and needs to be put into the queue, the current new data needs to be put at the tail of the sequence queue, a timestamp is used as a subscript of the current data, or when a signal is sent by the delay queue, expired data at the head of the queue is checked and dequeued, and meanwhile, a second computing center is informed to recalculate a data value.

In another case, according to different user requirements, the accumulation calculation and the deduplication calculation are performed simultaneously, that is, the data eigenvalue within the time corresponding to the user is sent to the first delay queue and the first calculation center, and the data eigenvalue within the time corresponding to the other user is sent to the sequence queue and the second delay queue, and the subsequent processing processes thereof are described in detail in the above corresponding manner, and are not described again here.

Step 305: and sending the result of the accumulation calculation and/or the deduplication calculation to a second intermediate storage container for storage, so that when the second intermediate storage container receives a request of an online service system for accessing a data value, the second intermediate storage container calculates the data value in a corresponding time window according to the result of the accumulation calculation and/or the deduplication calculation, and feeds back the data value to the online service system or a third-party service system.

In this step, the result of the accumulation calculation and/or the deduplication calculation is sent to a second intermediate storage container for storage, where the second intermediate storage container may also be composed of Redis. That is, by performing an accumulation calculation and/or a deduplication calculation, the result is an accumulated data value or a deduplicated data value, such as K: 2, and the second intermediate storage container stores a data value K: 2, the intermediate storage container in this case can support a large amount of service access on the line or browsing access with a long time window period, that is, Redis used as an intermediate storage container of the online service system or the third-party service system, and provides efficient access support for the online service system or the third-party service system.

In the embodiment of the invention, in order to quickly and accurately respond to the requirement of the online service system or the third-party service system on the real-time index, the client or the terminal collects the service data in real time, stores the collected service data into the corresponding message queue, processes and calculates the data in the message queue in real time, performs accumulative calculation and/or deduplication calculation on the calculated data characteristic value in the corresponding time, stores the result of the accumulative calculation and/or deduplication calculation in an intermediate storage container which can support efficient access, so that when a request for accessing the data value sent by the online service system or the third-party service system is received, the final data value is calculated according to the data characteristic value corresponding to a time window, and synchronously returns the calculated data value to the corresponding online service system or the third-party service system, therefore, efficient access support is provided for the online service system or the third-party service system, and the real-time calculation accuracy of the online service system or the third-party service system is improved.

For convenience of understanding, please refer to fig. 4 together, which is a schematic diagram of another application example of the service data processing method provided in the embodiment of the present invention, as shown in fig. 4, if three data, k-3, k-4, and k-5, are stored in the message queue (taking the three data as an example, in an actual application, the three data are not limited to this), the three data in the message queue are sequentially subjected to data processing (such as parsing processing, etc.) according to a time dimension, so as to obtain characteristic values of a data header in a corresponding time, such as k-1, k-2, and k-3, and then k-1, k-2, and k-3 are respectively sent to a corresponding delay queue and a corresponding sequence queue for processing.

In this embodiment, for the case of the accumulation calculation, the data characteristic value of the corresponding time obtained after the data processing is sent to the corresponding first delay queue (i.e., delay queue 1, which is composed of a kafka structure), and the first delay queue stores the data and the corresponding timestamp to the tail of the queue. Then, the first delay queue informs the first computing center (i.e., the computing center of the accumulation computation) to perform the accumulation computation on the data characteristic value, i.e., the first computing center adds 1 to the accumulated data value. When the first delay queue detects that the data characteristic value stored in the first delay queue is over-expired or over-expired, for example, offset-1 is over-expired, the first delay queue sends the over-expired or over-expired data characteristic value (for example, offset-1) to the first computing center, and the first computing center dequeues the over-expired or over-expired data characteristic value, that is, the accumulated data value is reduced by 1, that is, the purpose of clearing the over-expired or over-expired data is achieved.

It should be noted that, for the accumulation calculation provided by the embodiment of the present invention, through the delay queue, the expired data may be notified to the first calculation center, and the first calculation center performs a corresponding subtraction, so that the calculation window period slides forward according to the event without storing detailed data in the window period, and the accuracy of calculating the data value in the time window is effectively improved.

For the case of duplicate removal calculation, the data characteristic values obtained at the corresponding time after data processing are respectively sent to a second deferred queue (i.e. delay queue 2, the delay queue 2 is composed of a kafka structure) and an ordered queue (the ordered queue is composed of a Redis Zset set or structure).

In this embodiment, the principle of implementing the sequential queue is based on a Redis Zset (value, score) structure, and the timestamp is taken as score. Zset has a deduplication function, and is stored only once for the same K value, score is the timestamp stored last time, and the data feature values in the sequence queue are arranged from beginning to end according to the time sequence.

In this embodiment, the clipping method adopted by the sequence queue is as follows: the data characteristic values in the sequence queue are arranged from head to tail according to the time sequence, so that the head data characteristic value of the sequence queue is an earlier data characteristic value, for example, the window period is 1 hour, the current time is taken as the standard, the sequence queue performs backward pushing for 1 hour, if data beyond 1 hour exists in the sequence queue, the sequence queue clips the head of the queue from the time point of backward pushing for 1 hour, and therefore, after clipping, the sequence queue only stores the data characteristic value after the latest 1 hour is removed from repetition.

For example, in this embodiment, when the sequence queue receives the new data K-3, (data K-3, timestamp time-3) is placed at the tail of the sequence queue, and then the sequence queue is cut from the head according to the window period size. When the clipping action is completed, the computation center of the deduplication computation (i.e. the second computation center) is notified to compute the number of remaining K values in the current order queue.

When the sequence queue receives the message sent by the delay queue, the head of the queue is directly cut according to the size of the window period, and after the cutting action is completed, the second computing center is informed to compute the number of the residual K values in the current queue.

It should be noted that, in the deduplication calculation provided by the embodiment of the present invention, by sequential queue clipping implemented by Redis, a value in a sliding window period based on an event can be accurately calculated, and compared with a common approximate deduplication algorithm (HyperLogLog, which can save storage space but cannot solve the precision problem of data calculation and the problem of obsolete data when a window period slides forward based on an event), the embodiment of the present invention not only saves storage space, but also improves the accuracy of data amount calculation.

For the deduplication calculation provided by the embodiment of the present invention, the API provided by the Flink technology can support the implementation of event-triggered sliding window period calculation, but the calculation based on the event forward sliding still needs to adopt the embodiment provided by the present invention when a larger window period occurs.

And finally, the result of the accumulation calculation and/or the deduplication calculation is sent to a second intermediate storage container Redis for storage, so that when the second Redis receives a request of an online service system for accessing a data value, the second Redis calculates the data value in a corresponding time window according to the result of the accumulation calculation and/or the deduplication calculation, and feeds the data value back to the online service system or a third-party service system. Therefore, efficient access support is provided for the online service system or the third-party service system, and the real-time calculation accuracy of the online service system or the third-party service system is improved.

In the embodiment of the invention, the service data acquired in real time is processed and calculated, the calculated data characteristic values in the corresponding time are stored in the delay queue and the sequence queue, the accumulation calculation and/or the deduplication calculation are respectively carried out through the corresponding calculation centers, and the results of the accumulation calculation and/or the deduplication calculation are stored in the intermediate storage container Redis, so that when a request for accessing the data values sent by the online service system or the third-party service system is received, the final data values are calculated according to the data characteristic values corresponding to the time windows, and the calculated data values are synchronously returned to the corresponding online service system or the third-party service system, thereby providing efficient access support for the online service system or the third-party service system, and improving the real-time calculation accuracy of the online service system or the third-party service system.

Referring to fig. 5, a flowchart of a service data displaying method according to an embodiment of the present invention is shown, where the method includes:

step 501: receiving an inquiry request for inquiring the online service operation of a user;

in this step, the online service system or the third-party service system receives an inquiry request for inquiring the online service operation of the user, wherein the inquiry request may include an identifier of the user to be inquired.

The online service system or the third-party service system can be a searching system, a recommending system, a wind control system, an artificial intelligence system and the like, and the flow of various internet products and the like can be inquired and browsed by adopting the computing mode of the embodiment of the invention.

For example, when an online service system finds that a certain user maliciously utilizes a crawler to acquire a large amount of information of a website, the malicious crawler needs to be identified and intercepted, and for the service scene, user behavior data needs to be collected in real time and the number of pages browsed by the user needs to be calculated in real time. At this time, the administrator sends a query request for the user through the online service system.

Step 502: according to the query request, requesting a data value of the current online service operation of the user from an intermediate storage container; when the intermediate storage container receives the query request, calculating a data value of the user online service operation in the current time window according to the stored data characteristic value in each time, and feeding back the data value;

in this step, the intermediate storage container may be a first intermediate storage container or a second intermediate storage container, where functions and functions of the first intermediate storage container and the second intermediate storage container are described above and are not described herein again.

In this step, the online service system or the third party service system requests a data value of the current online service operation of the user from an intermediate storage container (such as Redis) according to the query request, and the Redis calculates the data value of the online service operation of the user in the current time window according to the stored data characteristic value in each time in the received query request, and feeds back the data value.

The process of calculating, by the Redis, the data value of the online service operation of the user in the current time window according to the stored data feature value in each time in the received query request is described in detail in the above corresponding embodiment, and is not described herein again.

Step 503: receiving the data value of the user online service operation sent by the intermediate storage container in the current time window;

in the step, the online service system or the third-party service system receives the data value of the online service operation of the user in the intermediate storage container and the current time window.

Step 504: and displaying the data value of the online service operation of the user so as to determine whether to process the online service operation of the user according to the data value.

In this step, the online service system or the third-party service system displays the data value of the user to the administrator, so that the administrator can determine whether to process the online service operation of the user according to the data value, for example, if the administrator determines that the number of pages visited by the same user is greater than 500 pages within 1 hour according to the data value, the administrator needs to intercept the amount of visited used in time, and the like.

In the embodiment of the invention, when an online service system or a third-party service system receives a query request for querying the online service operation of a user, according to the query request, a data value of the online service operation of the current user is requested to an intermediate storage container Redis, and the data value of the online service operation of the user in the current time window is sent after the Redis is received; and displaying the data value of the online business operation of the user. That is to say, in the embodiment of the present invention, when the online service system or the third-party service system receives the query request, the online service system or the third-party service system may quickly obtain the data value of the online service operation of the user in the current time window from the Redis, and display the data value of the online service operation of the user to the administrator through the online service system or the third-party service system, so as to facilitate the management to process the online service operation of the user according to the data value. The technical scheme provided by the embodiment of the invention can solve the problem of error in calculation caused by mismatching of calculation time and event time when the conventional calculation frame component is used for performing some real-time calculation. Not only the time for inquiring the user online business operation is saved, but also the accuracy rate for calculating the user online business operation in the current time window is improved.

It should be noted that in the embodiment of the present invention, since various internet products such as search, recommendation, wind control, artificial intelligence, etc. all require a large amount of real-time indexes to calculate the flow rate in the time window to support the upper-layer application, the accuracy, the real-time performance, and the speed of concurrent response of the calculation determine the quality of the internet products.

Particularly, in internet products, such as a search and recommendation service system, in order to find potential needs of a user, shorten a distance from the user to goods or information, and improve user experience, a large number of data feature values are required to characterize behavior data of the user. Similarly, in the field of information security, the policy engine established on the artificial intelligence technology has been advanced to the aspect of the functions of the wind control product, and accordingly, each policy system does not have to perform a large amount of feature calculations to support accurate response of a model algorithm or an artificial rule to a request, so that the calculation speed, response time and accuracy of the data feature values provided by the embodiment of the invention directly influence the judgment of the online systems on the user requirements.

It should be noted that, for simplicity of description, the method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the illustrated order of acts, as some steps may occur in other orders or concurrently in accordance with the embodiments of the present invention. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred and that no particular act is required to implement the invention.

Referring to fig. 6, a schematic structural diagram of a service data processing apparatus according to an embodiment of the present invention may specifically include the following modules: an obtaining module 601, a first sending module 602, a processing module 603 and a second sending module 604, wherein,

the obtaining module 601 is configured to obtain service data in real time;

the first sending module 602 is configured to send the service data to a corresponding message queue;

the processing module 603 is configured to perform data processing on the service data in each message queue according to a time dimension, so as to obtain a data characteristic value in a corresponding time;

the second sending module 604 is configured to send the data characteristic value obtained by the processing module within the corresponding time to a first intermediate storage container for storage, so that when the first intermediate storage container receives a request for accessing the data value from an online service system or a third-party service system, the first intermediate storage container calculates the data value within the corresponding time window according to the data characteristic value, and feeds back the data value to the online service system or the third-party service system.

Optionally, the apparatus further comprises: a calculation module 701 and a third sending module 702, which are schematically shown in fig. 7, wherein,

the calculating module 701 is configured to perform accumulation calculation and/or deduplication calculation on the data characteristic values in the corresponding time after the processing module 603 obtains the data characteristic values in the corresponding time;

the third sending module 702 is configured to send the result of the accumulation calculation and/or the deduplication calculation, which is calculated by the calculating module 701, to a second intermediate storage container for storage, so that when the second intermediate storage container receives a request for accessing a data value by an online service system, the second intermediate storage container calculates the data value in a corresponding time window according to the result of the accumulation calculation and/or the deduplication calculation, and feeds back the data value to the online service system or a third-party service system.

Optionally, the calculation module may include: a fourth sending module and an accumulation calculating module; or comprises the following steps: a fifth sending module and a duplicate removal calculation module; or comprises the following steps: a fourth sending module, an accumulation calculating module, a fifth sending module and a duplication eliminating calculating module, wherein,

the fourth sending module is configured to send the data feature values in the corresponding time to a first delay queue composed of a kafka structure of the kaffa, respectively, and the first delay queue stores the data feature values according to corresponding timestamps and then notifies the first computing center;

the accumulation calculation module is used for performing accumulation calculation on the data characteristic value through the first calculation center, performing expiration management on the received data characteristic value through the first delay queue, sending the expired data characteristic value to the first calculation center, and subtracting the expired data characteristic value from the result of the accumulation calculation by the first calculation center;

the fifth sending module is used for sending the data characteristic values in the corresponding time to a sequence queue consisting of a set of intermediate storage containers and a second delay queue consisting of kafka of the kaffka respectively; storing the data characteristic values by the sequence queue and the second delay queue according to corresponding time stamps respectively;

the duplicate removal calculation module is used for cutting the stored data characteristic values according to the window period size through the sequence queue and informing a second calculation center to calculate the number of the residual data characteristic values in the current sequence queue after cutting; and when the sequence queue receives an expired message sent by the second delay queue, the sequence queue cuts the expired message according to the window period, and after the cutting is finished, the sequence queue informs the second computing center to compute the number of the residual data characteristic values in the current queue.

Referring to fig. 8, a schematic structural diagram of a service data display device according to an embodiment of the present invention is shown, where the device includes: a first receiving module 801, a requesting module 802, a second receiving module 803, and a presenting module 804, wherein,

the first receiving module 801 is configured to receive an inquiry request for inquiring about an online service operation of a user;

the request module 802 is configured to request a data value of a current online service operation of a user from an intermediate storage container according to the query request; when the intermediate storage container receives the query request, calculating a data value of the user online service operation in the current time window according to the stored data characteristic value in each time, and feeding back the data value;

the second receiving module 803 is configured to receive a data value of the user online service operation sent in the current time window in the intermediate storage container;

the display module 804 is configured to display a data value of the user online service operation, so as to determine whether to process the user online service operation according to the data value.

For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.

Optionally, an embodiment of the present invention further provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when executed by the processor, the computer program implements the steps of the service data processing method described above or implements the processes of the service data display method embodiment described above, and can achieve the same technical effect, and in order to avoid repetition, the details are not described here again.

An embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when executed by a processor, the computer program implements the steps of the service data processing method or implements the processes of the service data presentation method embodiment, and can achieve the same technical effects, and in order to avoid repetition, the description is omitted here. The computer-readable storage medium may be a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.

In an exemplary embodiment, a computer program product is further provided, and when an instruction in the computer program product is executed by a processor of an electronic device, the electronic device is enabled to execute the steps of the service data processing method or implement the processes of the service data presentation method embodiment, and the same technical effect can be achieved, and in order to avoid repetition, details are not repeated here.

The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, terminals (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing terminal to cause a series of operational steps to be performed on the computer or other programmable terminal to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While preferred embodiments of the present invention have been described, additional variations and modifications of these embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the embodiments of the invention.

Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or terminal that comprises the element.

The service data processing method, the service data display method, the device, the electronic device and the storage medium provided by the invention are introduced in detail, and a specific example is applied in the text to explain the principle and the implementation mode of the invention, and the description of the above embodiment is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims

1. A method for processing service data is characterized by comprising the following steps:

acquiring service data in real time;

storing the service data into a corresponding message queue;

2. The method of claim 1, wherein after the data processing is performed on the service data in each message queue according to the time dimension to obtain the data characteristic value in the corresponding time, the method further comprises:

3. The method of claim 2, wherein the performing accumulation and/or de-duplication calculations on the data characteristic values in the corresponding time comprises:

respectively sending the data characteristic values in corresponding time to a first delay queue consisting of a Kaffka structure, wherein the first delay queue stores the data characteristic values according to corresponding timestamps and then informs a first computing center to carry out accumulation calculation on the data characteristic values, the received data characteristic values are subjected to expiration management through the first delay queue, the expired data characteristic values are sent to the first computing center, and the first computing center subtracts the expired data characteristic values from the accumulation calculation results;

and/or

4. A business data display method is characterized by comprising the following steps:

5. A service data processing apparatus, comprising:

the acquisition module is used for acquiring service data in real time;

6. The apparatus of claim 5, further comprising:

7. The apparatus of claim 6, wherein the computing module comprises: a fourth sending module and an accumulation calculating module; and/or, a fifth sending module and a deduplication calculation module, wherein,

the fifth sending module is configured to send the data eigenvalues within the corresponding time to a sequential queue composed of a set of intermediate memories and a second delay queue composed of kafka, respectively; storing the data characteristic values by the sequence queue and the second delay queue according to corresponding time stamps respectively;

8. A business data presentation apparatus, comprising:

9. An electronic device, comprising: memory, processor and computer program stored on the memory and executable on the processor, which when executed by the processor implements the steps of the business data processing method of any one of claims 1 to 3 or implements the steps of the business data presentation method of claim 4.

10. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the steps of the business data processing method according to any one of claims 1 to 3 or the steps of the business data presentation method according to claim 4.