CN116823023A

CN116823023A - Offline computing method, device, equipment and storage medium for data

Info

Publication number: CN116823023A
Application number: CN202310612083.9A
Authority: CN
Inventors: 贾光楠; 刘彦江; 高坡; 尉乃升
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2023-05-26
Filing date: 2023-05-26
Publication date: 2023-09-29

Abstract

The disclosure provides an offline computing method, device, equipment and storage medium for data, and relates to the technical field of data processing, in particular to the technical fields of big data, cloud computing, data analysis, offline computing and the like. The specific implementation scheme is as follows: generating a second pre-calculation index of at least one time dimension based on the first pre-calculation index according to the time demand information of the target service; determining a first data source according to the first data dimension information of the second pre-calculation index; and performing off-line calculation based on the first data source according to the first calculation logic information and the first time dimension information of the second pre-calculation index to obtain a first pre-calculation result of the second pre-calculation index. According to the technology disclosed by the invention, the offline calculation is performed by utilizing the second pre-calculation index, so that the calculation result of the index required by the service can be obtained in advance, the time-consuming problem of calculating the index required by the service in real time is solved, and the overall calculation efficiency of the service is improved.

Description

Offline computing method, device, equipment and storage medium for data

Technical Field

The disclosure relates to the technical field of data processing, in particular to the technical fields of big data, cloud computing, data analysis, off-line computing and the like.

Background

An offline calculation is a calculation that is performed with all input data known before the start of the calculation and with the result being obtained immediately after a problem is solved. The large data belongs to the calculation part of the data, and real-time calculation is performed in the part corresponding to offline calculation.

Disclosure of Invention

The disclosure provides an offline computing method, device and equipment for data and a storage medium.

According to an aspect of the present disclosure, there is provided an offline computing method of data, including:

generating a second pre-calculation index of at least one time dimension based on the first pre-calculation index according to the time demand information of the target service;

determining a first data source according to the first data dimension information of the second pre-calculation index; and

and performing off-line calculation based on the first data source according to the first calculation logic information and the first time dimension information of the second pre-calculation index to obtain a first pre-calculation result of the second pre-calculation index.

According to another aspect of the present disclosure, there is provided an offline computing device of data, comprising:

the generation module is used for generating a second pre-calculation index of at least one time dimension based on the first pre-calculation index according to the time demand information of the target service;

The first determining module is used for determining a first data source according to the first data dimension information of the second pre-calculation index; and

and the first calculation module is used for performing off-line calculation based on the first data source according to the first calculation logic information and the first time dimension information of the second pre-calculation index so as to obtain a first pre-calculation result of the second pre-calculation index.

According to another aspect of the present disclosure, there is provided an electronic device including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein, the liquid crystal display device comprises a liquid crystal display device,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of the embodiments of the present disclosure.

According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform a method according to any one of the embodiments of the present disclosure.

According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements a method according to any of the embodiments of the present disclosure.

According to the technology disclosed by the invention, the offline calculation is performed by utilizing the second pre-calculation index, so that the calculation result of the index required by the service can be obtained in advance, the time-consuming problem of calculating the index required by the service in real time is solved, and the overall calculation efficiency of the service is improved.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.

Drawings

The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a flow diagram of a method of offline computing of data according to an embodiment of the present disclosure;

FIG. 2 is a schematic application diagram of an offline computing method of data according to an embodiment of the present disclosure;

FIG. 3 is a flow diagram of a method of offline computing of data according to an embodiment of the present disclosure;

FIG. 4 is a schematic application diagram of an offline computing method of data according to an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of an offline computing device of data according to an embodiment of the present disclosure;

fig. 6 is a block diagram of an electronic device for implementing an offline computing method of data of an embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

As shown in fig. 1, an embodiment of the present disclosure provides an offline computing method of data, including:

step S101: and generating a second pre-calculation index of at least one time dimension based on the first pre-calculation index according to the time demand information of the target service.

Step S102: the first data source is determined based on the first data dimension information of the second pre-computed index. And

step S103: and performing off-line calculation based on the first data source according to the first calculation logic information and the first time dimension information of the second pre-calculation index to obtain a first pre-calculation result of the second pre-calculation index.

According to the embodiment of the disclosure, it is to be noted that:

the target service can be understood as any service requiring computational analysis by using data. That is, the offline computing method of data according to the embodiments of the present disclosure may be applicable to the data analysis and calculation requirements based on the index in any business field. For example, the target business may be a marketing service business, a manufacturing business, a telephone business, a multimedia business, and the like.

The first pre-calculation index can be understood as a judgment index for performing data analysis on a certain service, and the index needs to be calculated by using data of at least one dimension. The first pre-calculation index may be any dimension of a judgment index used by the target service in data analysis, which is not particularly limited herein, and may be selected and adjusted according to needs. For example, the first pre-calculated indicator may be an indicator of total number of people, total amount, cumulative value, average, conversion, etc. The determination manner of the first pre-calculation index may be selected and adjusted according to the requirement of the target service, which is not specifically limited herein. For example, the first pre-calculation index may be a judgment index that is often used when the target service performs data analysis. And/or the first pre-computed index may be a criterion index that is computationally intensive. For another example, the first pre-calculation index may be a judgment index playing a decisive role in the analysis and calculation of the current data of the target service.

Time requirement information can be understood as requirement information for the time dimension of the criterion index (first pre-calculated index). For example, from the time demand information, it can be known that the target business wishes to evaluate and analyze hourly data, daily data, monthly data, or yearly data using an evaluation index. That is, the second pre-calculated index that needs to generate several time dimensions can be known from the time demand information. Specifically, when the first pre-calculation index is the total sales amount and the time dimension of the time demand information is monthly and weekly, the second pre-calculation index obtained based on the first pre-calculation index includes at least two time dimension indexes of monthly and weekly, that is, the total sales amount of monthly and the total sales amount of weekly.

The first pre-calculation index includes at least second calculation logic information, second data dimension information, and time dimension information. From the second data dimension information, it is known on which dimensions the first pre-calculation index needs to be calculated based on. From the second calculation logic information, it can be known with which calculation logic (addition, subtraction, multiplication, division, etc.) the first pre-calculation index needs to calculate the data of these dimensions. From the time dimension information, the time dimension of the data to be used by the first pre-calculation index can be known.

The second pre-calculation index includes at least first data dimension information, first calculation logic, and first time dimension information. From the first data dimension information, it may be known on which dimensions of data the second pre-calculation index needs to be calculated, and it may be further confirmed which first data source the dimensions of data are stored in. From the first calculation logic information, it can be known with which calculation logic (addition, subtraction, multiplication, division, etc.) the second pre-calculation index needs to calculate the data of these dimensions. From the first time dimension information, the time dimension of the data to be used by the second pre-calculation index can be known. For example, if the second pre-calculation index is to calculate sales per week, it is known from the first time dimension information that the second pre-calculation index needs to use sales data per day within a week. The first calculation frequency of the second pre-calculation index can also be known according to the first time dimension information. For example, if the second pre-calculation index is to calculate sales per week, the first calculation frequency is weekly, that is, each week needs to obtain the latest first pre-calculation result (sales of the current week) by offline calculation based on the data (sales data) in the corresponding first data source (the data source storing sales data) according to the second pre-calculation index.

The first data source may be understood as a data source for storing first data used when offline computing the second pre-computed metrics. Wherein the first data may be determined from the first data dimension information. When determining that the second pre-calculation index requires using the first data of multiple dimensions in offline calculation according to the first data dimension information, a first data source storing the first data of each dimension needs to be determined.

According to the first calculation logic information and the first time dimension information of the second pre-calculation index, offline calculation is performed based on the first data source, which can be understood as follows: first data required by offline computation is acquired from a first data source according to first time dimension information, and then the first data is subjected to offline computation by utilizing operation logic determined by first computation logic information, so that a first pre-computation result of a second pre-computation index is obtained through computation.

According to the technology of the embodiment of the disclosure, the calculation result of the index required by the target service can be obtained in advance by performing offline calculation by using the second pre-calculation index. When the target service index calculation task is consistent with the second pre-calculation index, the target service can directly call the first pre-calculation result which is pre-calculated offline based on the second pre-calculation index, and real-time calculation is not needed by the target service index calculation task. The method effectively solves the time-consuming problem of calculating the indexes required by the target service in real time and the problem that the results cannot be calculated and displayed in time, and improves the overall calculation efficiency and calculation performance of the target service.

In one example, the process of index analysis and calculation often requires a large amount of machine resources and is very time-consuming, and the focus of index analysis and calculation is how to quickly obtain corresponding data analysis results according to a certain index logic according to numerous dimension attributes, and meanwhile, the attribute data of various events need to be combined, the relevant dimensions are optimized or selected for effective analysis and calculation, and finally, a corresponding data analysis report or a data signboard and the like are obtained. In recent years, due to rapid development of digital economic technology, large data development is rapid, and in the traditional real-time data analysis process, a data user does not have a dimension index storage device or a corresponding dimension index platform, and most of cases cannot perform good dimension selection and rapid index analysis on a company business entity, so that the traditional data analysis mode is very passive. Meanwhile, many companies often contain numerous event entity attribute dimensions, so that data of event attributes are often huge, and the calculation of analysis and calculation of corresponding indexes is complex and slow. Therefore, the traditional data real-time calculation method is time-consuming and cannot timely acquire the calculation result of the dimension index. Aiming at the problem of slow real-time calculation of traditional data, the method disclosed by the embodiment of the invention is based on extracting the dimension index (second pre-calculation index), and improves the speed of dimension index calculation under a large number of data scenes by storing and calculating the dimension index in batches in advance. The problems that the real-time computing platform consumes more time and cannot calculate and display results quickly in a multi-dimensional index analysis scene are solved by quickly and accurately determining the second pre-calculation index based on the first pre-calculation index and storing the calculation logic of the second pre-calculation index and performing off-line calculation. By performing off-line calculation based on the second pre-calculation index, the calculation performance of the index can be improved, the calculation result of the dimension index (the second pre-calculation index) is displayed on a data analysis platform or a data billboard in real time, and the use experience and convenience of the first pre-calculation result of the final second pre-calculation index are ensured.

In one example, the second pre-calculation index determined in step S101, and/or the first pre-calculation result obtained in step S103 through offline calculation may be stored in the preset storage space, so as to subsequently retrieve at least one first pre-calculation result corresponding to the second pre-calculation index from the preset storage space. The preset storage space may be understood as any data container in the prior art, and is not specifically limited herein. The preset storage space may be understood as a preset data container for storing the second pre-calculation index and the first pre-calculation result thereof. The preset storage space may store a plurality of first pre-calculation results of the second pre-calculation index, where the first pre-calculation results are results obtained by calculating the second pre-calculation index based on different first data at different time nodes.

The manner in which the second pre-calculation index and the first pre-calculation result are stored and maintained in the storage space is not particularly limited herein, and any manner of data storage and maintenance in the prior art may be employed as needed. For example, a corresponding memory table is generated for each second pre-calculation index for the storage of the first pre-calculation result of each second pre-calculation index.

In one example, the offline computing method of data of the embodiments of the present disclosure may be applied to a product marketing scenario. The product marketing business needs to evaluate the market acceptance of the A product. Wherein the time demand information is the total number of consumers that the product marketing business needs to know monthly and weekly A products. The first pre-calculated index is the total number of consumer products a in the product marketing business. Based on the first pre-calculated index, a second pre-calculated index of the total number of consumption of the A product per month and a second pre-calculated index of the total number of consumption of the A product per month can be generated according to the time demand information of the target business. The first data dimension information is data of the number of consumers of the product A. Based on the first data dimension information of the second pre-calculated metrics, a first data source storing data of the number of consumer products of A can be determined. The first time dimension information is calculated time units of weekly. The first calculation logic information is to add up the total number of consumers for seven days of the week. According to the first calculation logic information and the first time dimension information of the second pre-calculation index, offline calculation is performed based on the first data source, and a specific value of the total consumption number of the A product in each week (a first pre-calculation result) and a specific value of the total consumption number of the A product in each month (a first pre-calculation result) can be obtained.

In one example, the offline computing method of the data of the embodiments of the present disclosure may be applied to a large-scale offline processing framework, i.e., an offline computing framework that may be decoupled from a real-time computing framework. The large-scale offline processing framework can meet the calculation requirement of index data (second pre-calculation index) of a data user of the target service, and the result of the second pre-calculation index is calculated in advance in an offline calculation mode, so that the data user can directly and quickly acquire the calculation result when the calculation result of the index data is required to be used, the calculation result is not required to be acquired in a real-time calculation mode, and the calculation time and the occupation of calculation resources are saved.

In one implementation, the offline computing method of the data of the embodiment of the present disclosure includes steps S101 to S103, where step S101: generating a second pre-computed index of the at least one time dimension based on the first pre-computed index according to the time demand information of the target service, comprising:

step S1011: and determining first time dimension information of at least one time dimension according to the time demand information of the target service.

Step S1012: and determining the first calculation logic information according to the second calculation logic information of the first pre-calculation index.

Step S1013: and determining the first data dimension information according to the second data dimension information of the first pre-calculation index.

Step S1014: and generating a second pre-calculation index of at least one time dimension according to the first time dimension information, the first calculation logic information and the first data dimension information.

According to the embodiment of the disclosure, it is to be noted that:

time demand information used for representing time dimensions for which target business needs to establish indexes so as to realize data analysis of the business. For example, when the target business is found to be frequently focused on data analysis of sales of products through analysis, the total sales amount of the A product is determined as a first pre-calculated index. And according to the data analysis frequency of the target service, determining that the target service is often concerned about the weekly sales of the product, and determining the time dimension of the time demand information to be weekly. Based on this, it can be determined that the first time dimension information includes at least the calculation frequency of the second pre-calculation index as offline calculation once per week, and that the first data used when the second pre-calculation index is offline calculated is sales amount data generated on seven days of the week.

From the second data dimension information, it is known on which dimensions the first pre-calculation index needs to be calculated based on. Based on this, the first data dimension information of the second pre-calculation index may be determined, i.e. it may be determined on which dimensions of data the second pre-calculation index needs to be calculated.

From the second calculation logic information, it can be known with which calculation logic (addition, subtraction, multiplication, division, etc.) the first pre-calculation index needs to calculate the data of these dimensions. Based on this, it is possible to determine the first calculation logic information of the second pre-calculation index, i.e. with which calculation logic the second pre-calculation index needs to be calculated.

According to the technology of the embodiment of the disclosure, the second pre-calculation index required by the target service can be generated rapidly and accurately based on the first pre-calculation index according to the time demand information of the target service. The offline calculation is performed by utilizing the second pre-calculation index which is more attached to the analysis requirement of the target service, and the obtained first pre-calculation result can hit the target service index calculation task in a larger probability, so that the possibility that the target service directly obtains the calculation result of the target service required index is improved, the time-consuming problem of real-time calculation of the target service required index is solved, and the overall calculation efficiency of the service is improved.

In one implementation, the offline computing method of the data of the embodiments of the present disclosure includes steps S101 to S103, and steps S1011 to S1014, where step S1014: generating a second pre-calculation index for at least one time dimension from the first time dimension information, the first calculation logic information, and the first data dimension information, comprising:

In the case of determining first time dimension information including a plurality of time dimensions, a second pre-calculation index for each time dimension is generated from the first calculation logic information, the first data dimension information, and the first time dimension information corresponding to each time dimension.

According to the embodiment of the disclosure, it is to be noted that:

the first time dimension information including a plurality of time dimensions may be understood as first time dimension information including a plurality of dimensions of year, month, week, day, hour, minute, and the like.

When the first time dimension information of a plurality of time dimensions is included, a corresponding second pre-calculation index of the time dimension of year, a second pre-calculation index of the time dimension of month, a second pre-calculation index of the time dimension of week and the like are generated. The first calculation logic information of the second pre-calculation indexes of the different time dimensions may be the same, the first data dimension information may also be the same, and the difference is that the second pre-calculation indexes of the different time dimensions, the data amounts of the first data used in the off-line calculation are different. For example, a second pre-calculation index in the time dimension of a year requires an offline calculation using the first data generated during the year. The second pre-calculation index, which takes months as the time dimension, requires only offline calculations using the first data generated during one month.

According to the technology of the embodiment of the disclosure, the requirement of the index calculation task of different time dimensions of the target service can be met by generating the second pre-calculation indexes of the plurality of time dimensions. The offline calculation is performed by utilizing the second pre-calculation indexes with a plurality of time dimensions, and the obtained first pre-calculation result can hit the index calculation task of the target service in a larger probability, so that the possibility that the target service directly obtains the calculation result of the index required by the service is improved, the time-consuming problem of calculating the index required by the target service in real time is solved, and the overall calculation efficiency of the service is improved.

In one implementation, the offline computing method of the data of the embodiment of the present disclosure includes steps S101 to S103, where step S102: determining a first data source from the first data dimension information of the second pre-computed index, comprising:

and determining first data required to be used for offline calculation according to the first data dimension information of the second pre-calculation index.

And determining the data source storing the first data as the first data source to be accessed.

According to the embodiment of the disclosure, it is to be noted that:

when determining that the second pre-calculation index requires using the first data of multiple dimensions in offline calculation according to the first data dimension information, a first data source storing the first data of each dimension needs to be determined.

The specific manner of data access and the data transfer means for accessing data from the first data source may be selected and adjusted as desired and are not specifically limited herein.

According to the technology of the embodiment of the disclosure, by determining the first data source to be accessed, the second pre-calculation index can be more convenient and quicker when offline calculation is performed, the offline calculation efficiency and the offline calculation performance are improved, and therefore the first pre-calculation result corresponding to the second pre-calculation index is calculated more quickly.

In one implementation, the offline computing method of the data in the embodiment of the disclosure includes steps S101 to S103, and further includes:

and determining a first calculation frequency according to the first time dimension information of the second pre-calculation index.

And under the condition that the calculation time of the second pre-calculation index meets the first calculation frequency, performing off-line calculation based on the first data source to obtain a first pre-calculation result of the second pre-calculation index.

According to the embodiment of the disclosure, it is to be noted that:

the first calculation frequency may be understood as a frequency of offline calculation using the second pre-calculation index. For example, the second pre-calculation index is to calculate the sales of the product weekly, and the first calculation frequency is once a week.

The calculation time of the second pre-calculation index satisfies the first calculation frequency, which can be understood as the time that the time interval elapsed after the time node of the last offline calculation of the second pre-calculation index reaches the first calculation frequency. For example, when the time of the last offline calculation of the second pre-calculation index is monday, when the calculation time of the second pre-calculation index reaches the next week, the first calculation frequency of one-week calculation is satisfied, and offline calculation based on the first data by using the second pre-calculation index is needed.

The data of the first data source may be updated over time. That is, when the time passes by one week, the new data generated during the week is updated and stored in the first data source, so as to ensure that the second pre-calculation index can perform a new round of calculation based on the new data stored in the update, thereby obtaining a new first pre-calculation result.

According to the technology of the embodiment of the disclosure, the second pre-calculation index is continuously subjected to offline calculation through the first calculation frequency, so that the corresponding first pre-calculation result is continuously obtained based on the second pre-calculation index, the requirement of an index calculation task of a target service is met, when the index calculation task is received, the corresponding first pre-calculation result is stored in a pre-calculation mode, the corresponding first pre-calculation result is directly called according to the index calculation task, and timely feedback display of the calculation result of the index calculation task is realized.

In one example, in the case that the second pre-calculation index is plural, it is necessary to determine the first calculation frequency of each second pre-calculation index, and sequentially make each second pre-calculation index perform a new round of offline calculation according to the first calculation frequency of each second pre-calculation index. And realizing time-sharing offline calculation of a plurality of second pre-calculation indexes and updating of corresponding first pre-calculation results.

According to the technology of the embodiment of the disclosure, the resource occupation condition and the calculation pressure of offline calculation can be effectively slowed down. And the corresponding first pre-calculation result is stored in a pre-calculation mode when the index calculation task is received, so that the corresponding first pre-calculation result is directly called according to the index calculation task, and timely feedback display of the calculation result of the index calculation task is realized.

In one example, as shown in fig. 2, the first calculation frequency of the second pre-calculated index is offline calculated once per month. Then offline calculation is required to be performed on 1 month 1 day, 2 months 1 day, 3 months 1 day, and 4 months 1 day based on the second pre-calculation index and the data in the first data source, and the first pre-calculation results obtained by the four months calculation are stored in the preset storage space of the second pre-calculation index.

In one implementation, the offline computing method of the data of the embodiment of the present disclosure includes steps S101 to S103, where in step S101: before generating the second pre-calculation index of the at least one time dimension based on the first pre-calculation index according to the time demand information of the target service, the method further comprises:

and determining the candidate index as a first pre-calculation index in the case that the index calculation duration of the candidate index meets the duration threshold and/or in the case that the index use frequency of the candidate index meets the frequency threshold.

According to the embodiment of the disclosure, it is to be noted that:

candidate metrics may be understood as metrics used by the historical metrics calculation task for the target business.

The index calculation time length can be understood as the time length spent by the candidate index to perform real-time calculation once.

The frequency of index use can be understood as the number of times the candidate index is used.

According to the technology of the embodiment of the disclosure, more indexes used by the index calculation task of the target service can be determined, and the indexes are determined to be the first pre-calculation indexes, so that the second pre-calculation indexes generated based on the first pre-calculation indexes are more likely to hit a new index calculation task. Therefore, the possibility that the target service directly obtains the calculation result of the service required index is improved, the time-consuming problem of real-time calculation of the target service required index is solved, and the overall calculation efficiency of the service is improved.

and determining a preset storage space corresponding to the second pre-calculation index.

And storing the first pre-calculation result into a preset storage space and associating the first pre-calculation result with the second pre-calculation index.

According to the embodiment of the disclosure, it is to be noted that:

the preset storage space may be understood as a preset data container for storing the second pre-calculation index and the first pre-calculation result thereof. The preset storage space may store a plurality of first pre-calculation results of the second pre-calculation index, where the first pre-calculation results are results obtained by calculating the second pre-calculation index based on different first data at different time nodes.

According to the technology of the embodiment of the disclosure, the first pre-calculation result corresponding to the second pre-calculation index is stored in the preset storage space, so that when the second pre-calculation index hits a new index calculation task, the corresponding first pre-calculation result can be quickly called from the preset storage space, and the display result is fed back to a user of the target service.

In one implementation, as shown in fig. 3, the offline computing method of the data in the embodiment of the disclosure includes steps S101 to S103, and further includes:

Step S301: in the case of receiving the index calculation task, a target calculation index of the index calculation task is determined.

Step S302: and outputting a first pre-calculation result of the second pre-calculation index as a calculation result of the index calculation task under the condition that the second pre-calculation index matched with the target calculation index is stored in the preset storage space.

According to the embodiment of the disclosure, it is to be noted that:

the second pre-calculation index matched with the target calculation index can be understood as the same calculation logic information, time dimension information and data dimension information of the target calculation index and the target calculation index. For example, the index calculation task calculates the total amount of sales of the A product from 1 month 1 to 1 month 2 years 2023, and the second pre-calculation index calculates the total amount of sales of the A product for each month, and then the two match. At this time, the first pre-calculation result of the second pre-calculation index from 1 month 1 day to 2 months 1 day is called in the preset storage space, and can be used as the calculation result of the index calculation task.

According to the technology of the embodiment of the disclosure, the quick response of the index calculation task can be realized. When the target service index calculation task is consistent with the second pre-calculation index, the target service can directly call the first pre-calculation result which is pre-calculated offline based on the second pre-calculation index, and real-time calculation is not needed by the target service index calculation task. The method effectively solves the time-consuming problem of calculating the indexes required by the target service in real time and the problem that the results cannot be calculated and displayed in time, and improves the overall calculation efficiency and calculation performance of the target service.

Step S303: and under the condition that the second pre-calculation index matched with the target calculation index is not stored in the preset storage space, determining the second data source according to the third data dimension information of the target calculation index.

Step S304: and carrying out real-time calculation based on the second data source according to the third calculation logic information of the target calculation index so as to obtain a calculation result of the target calculation index.

Step S305: and associating the calculation result with the target calculation index and storing the calculation result into the target storage space.

According to the embodiment of the disclosure, it is to be noted that:

according to the third data dimension information, it can be known which dimensions of data the target calculation index needs to be calculated based on.

From the third calculation logic information, it is known with which calculation logic (addition, subtraction, multiplication, division, etc.) the target calculation index needs to calculate the data of these dimensions.

The second data source may be understood as a data source for storing data used in calculation of the target calculation index.

According to the technology of the embodiment of the disclosure, the condition that the offline calculation does not hit the target calculation index can be made up in a real-time calculation mode, so that the calculation result can be output in time, and the requirement of an index calculation task can be met.

In one implementation, the offline computing method of the data in the embodiment of the disclosure includes steps S101 to S103, S301, S303 to S305, and further includes:

and determining a second calculation frequency according to the second time dimension information of the target calculation index.

And under the condition that the calculation time of the target calculation index meets the second calculation frequency, acquiring corresponding second data from a second data source.

And performing off-line calculation based on the third calculation logic information of the target calculation index and the second data to obtain a second pre-calculation result of the target calculation index.

According to the embodiment of the disclosure, it is to be noted that:

the second calculation frequency can be understood as a frequency of offline calculation using the target calculation index. For example, the target calculation index is to calculate the sales of the product weekly, and the second calculation frequency is once a week.

The calculation time of the target calculation index satisfies the second calculation frequency, which can be understood as the time when the elapsed time interval after the time node of the last offline calculation of the target calculation index reaches the second calculation frequency. For example, when the time of the last offline calculation of the target calculation index is monday, when the calculation time of the target calculation index reaches the next week, the second calculation frequency of one-week calculation is satisfied, and offline calculation based on the first data by using the target calculation index is needed.

The data of the second data source may be updated over time. That is, when the time passes by one week, the new data generated during the week is updated and stored in the second data source, so as to ensure that the target calculation index can perform a new round of calculation based on the new data stored in the update, thereby obtaining a new second pre-calculation result.

According to the technology of the embodiment of the disclosure, the target calculation index is continuously subjected to offline calculation through the second calculation frequency, so that the corresponding second pre-calculation result is continuously obtained based on the target calculation index, the requirement of the target service index calculation task is met, when the target calculation task is received, the corresponding second pre-calculation result is stored in a pre-calculation mode, the corresponding second pre-calculation result is directly called according to the target calculation task, and timely feedback display of the calculation result of the target calculation task is realized.

In one example, the offline computing method of the data of the embodiment of the disclosure further includes:

and generating sub-target calculation indexes of at least one time dimension based on the target calculation indexes according to the time demand information of the target service.

And determining a data source according to the data dimension information of the sub-target calculation index. And

and performing off-line calculation based on the data source according to the calculation logic information and the time dimension information of the sub-target calculation index to obtain a pre-calculation result of the sub-target calculation index.

According to the technology of the embodiment of the disclosure, the calculation result of the target service required index can be obtained in advance by performing offline calculation by using the sub-target calculation index. When the target service index calculation task is consistent with the sub-target calculation index, the target service can directly call a pre-calculation result which is pre-calculated offline based on the sub-target calculation index, and real-time calculation is not needed by the target service index calculation task. The method effectively solves the time-consuming problem of calculating the indexes required by the target service in real time and the problem that the results cannot be calculated and displayed in time, and improves the overall calculation efficiency and calculation performance of the target service.

In one example, as shown in fig. 4, the second pre-computed index and the target pre-computed index are stored in the same pre-set memory space. The first calculation frequency of the second pre-calculation index is offline calculation once per month. Then offline calculation is required to be performed on 1 month 1 day, 2 months 1 day, 3 months 1 day, and 4 months 1 day based on the second pre-calculation index and the data in the first data source, and the first pre-calculation results obtained by the four months calculation are stored in the preset storage space of the second pre-calculation index. The second calculation frequency of the target calculation index is that offline calculation is performed once a day. Then offline calculation is required to be performed on 1 month 1 day, 1 month 2 days, 1 month 3 days, and 1 month 4 days based on the target calculation index and the data in the second data source, and the second pre-calculation results obtained by four days of calculation are stored in the preset storage space of the target calculation index.

In one example, the offline computing method of the data of the embodiments of the present disclosure is performed by a dimension index access module, a dimension index processing module, and a dimension index query module. The dimension index access module is used for accessing the first data source. The dimension index processing module is used for executing steps S101 to S103. The dimension index query module performs steps S301 to S305. Specific:

and the dimension index access module is used for accessing data according to event data sources (first data sources) with different index dimensions (second pre-calculated indexes).

And the dimension index processing module is used for preprocessing in advance according to the system dimension index processing requirement. The method comprises the following steps:

and selecting and storing dimension index processing logic. The calculation logic of the common dimension index (first pre-calculation index) is uniformly managed, namely, the dimension indexes which have long calculation time and need to be frequently searched are selected in advance, such as a plurality of headcount numbers, total amount, accumulated value, average value, conversion rate and the like. And analyzing and combining each common dimension index to obtain a second pre-calculated index.

Dimension index processing logic converts the code. The second pre-calculated index is converted into a code for storage. And generating a corresponding storage table for the calculation result of each second pre-calculation index, wherein the storage table is used for storing the dimension index processing result (first pre-settlement result), namely, classifying, aggregating and calculating the media platform data of the same hierarchy dimension according to different time dimensions of different hierarchies, including years, months, days, hours, minutes and the like, and storing the media platform data in a data storage medium (preset storage space).

And D, dimension index offline batch calculation. The method comprises the following steps:

off-line batch calculation of the second pre-calculated index provides a batch processing task. That is, a unified or non-unified time-sharing computing device is added to each processing logic (second pre-computing index) of the plurality of time dimensions, data preprocessing (obtaining a first pre-computing result) of the dimension index results is performed in the large-scale offline processing architecture, and the data results are prepared for the dimension index query module. For example, after the encoding process is performed on the calculation logic (second pre-calculation index). Corresponding offline computing tasks need to be added to the device, offline computing tasks similar to daily, weekly, monthly and the like can be added according to specific computing logic, and computing results are stored in corresponding containers (preset storage spaces) to provide preprocessing data for the dimension index query module.

And the dimension index query module is used for judging and displaying according to the dimension index query selection. The method comprises the following steps:

and selecting a required dimension index. Specific: and determining a target calculation index of the index calculation task by combining the data analysis business requirement.

And D, judging a dimension index selection result. The method comprises the following steps: and (3) performing corresponding calculation logic comparison at the back end, and outputting a first pre-calculation result of the corresponding second pre-calculation index immediately if the comparison finds that the calculation logic (target calculation index) exists in the container. If the comparison finds that the calculation logic does not exist in the container, the calculation logic is added into the container after the new calculation logic (target calculation index) is coded, and meanwhile, the calculation logic is added into a corresponding offline batch processing task, and the back end is preprocessed for the next query. For example, if the target calculation index is the saved dimension index calculation logic (second pre-calculation index), the result (first pre-calculation result) is quickly obtained from the container in real time, if the result is not matched, after the real-time calculation is performed, the target calculation index and the corresponding calculation result are stored in the container, so that the next query efficiency is improved.

And displaying the dimension index query result. The method comprises the following steps: and according to the requirements of the user and the output capacity of the front-end page, timely displaying the query data (the first pre-calculation result or the calculation result corresponding to the target calculation index) in the form of a report or a chart.

As shown in fig. 5, an embodiment of the present disclosure provides an offline computing device of data, including:

a generating module 510, configured to generate a second pre-calculation indicator of the at least one time dimension based on the first pre-calculation indicator according to the time requirement information of the target service.

The first determining module 520 is configured to determine the first data source according to the first data dimension information of the second pre-calculation index. And

the first calculating module 530 is configured to perform offline calculation based on the first data source according to the first calculation logic information and the first time dimension information of the second pre-calculation index, so as to obtain a first pre-calculation result of the second pre-calculation index.

In one embodiment, the generating module 510 is configured to:

the first determining sub-module is used for determining first time dimension information of at least one time dimension according to time demand information of the target service.

And the second determining submodule is used for determining the first calculating logic information according to the second calculating logic information of the first pre-calculating index.

And the third determining submodule is used for determining the first data dimension information according to the second data dimension information of the first pre-calculation index.

And the generation sub-module is used for generating a second pre-calculation index of at least one time dimension according to the first time dimension information, the first calculation logic information and the first data dimension information.

In one embodiment, the generating submodule is to:

In one embodiment, the first determining module 520 is configured to:

In one embodiment, the offline computing device of data further comprises:

the first frequency determining module is used for determining a first calculation frequency according to the first time dimension information of the second pre-calculation index.

The first acquisition module is used for acquiring corresponding first data from the first data source under the condition that the calculation time of the second pre-calculation index meets the first calculation frequency.

And the second calculation module is used for performing off-line calculation based on the first calculation logic information and the first data of the second pre-calculation index so as to obtain a first pre-calculation result of the second pre-calculation index.

In one embodiment, the offline computing device of data further comprises:

and the second determining module is used for determining the candidate index as a first pre-calculated index in the case that the index calculation duration of the candidate index meets the duration threshold value and/or in the case that the index use frequency of the candidate index meets the frequency threshold value.

In one embodiment, the offline computing device of data further comprises:

and the third determining module is used for determining a preset storage space corresponding to the second pre-calculation index.

The first storage module is used for storing the first pre-calculation result into a preset storage space and associating the first pre-calculation result with the second pre-calculation index.

In one embodiment, the offline computing device of data further comprises:

and the fourth determining module is used for determining a target calculation index of the index calculation task under the condition that the index calculation task is received.

The first matching module is used for outputting a first pre-calculation result of the second pre-calculation index as a calculation result of the index calculation task under the condition that the second pre-calculation index matched with the target calculation index is stored in the preset storage space.

In one embodiment, the offline computing device of data further comprises:

And the second matching module is used for determining a second data source according to the third data dimension information of the target calculation index under the condition that the second pre-calculation index matched with the target calculation index is not stored in the preset storage space.

And the third calculation module is used for carrying out real-time calculation based on the second data source according to the third calculation logic information of the target calculation index so as to obtain a calculation result of the target calculation index.

And the second storage module is used for associating the calculation result with the target calculation index and storing the calculation result into the target storage space.

In one embodiment, the offline computing device of data further comprises:

and the second frequency determining module is used for determining a second calculation frequency according to the second time dimension information of the target calculation index.

The second obtaining module is used for obtaining corresponding second data from the second data source under the condition that the calculation time of the target calculation index meets the second calculation frequency.

And the fourth calculation module is used for performing off-line calculation based on the third calculation logic information and the second data of the target calculation index so as to obtain a calculation result of the target calculation index.

For descriptions of specific functions and examples of each module and sub-module of the apparatus in the embodiments of the present disclosure, reference may be made to the related descriptions of corresponding steps in the foregoing method embodiments, which are not repeated herein.

In the technical scheme of the disclosure, the acquisition, storage, application and the like of the related user personal information all conform to the regulations of related laws and regulations, and the public sequence is not violated.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

Fig. 6 illustrates a schematic block diagram of an example electronic device 600 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile apparatuses, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing apparatuses. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 6, the apparatus 600 includes a computing unit 601 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 may also be stored. The computing unit 601, ROM 602, and RAM 603 are connected to each other by a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

Various components in the device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, mouse, etc.; an output unit 607 such as various types of displays, speakers, and the like; a storage unit 608, such as a magnetic disk, optical disk, or the like; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.

The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as an offline computing method of data. For example, in some embodiments, the offline computing method of data may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 600 via the ROM 602 and/or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the offline computing method of data described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the offline computing method of data in any other suitable manner (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

It should be appreciated that the various forms of flow shown above may be used, and that the reordering, adding, or adding of rows may be performed in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.

The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions, improvements, etc. that are within the principles of the present disclosure are intended to be included within the scope of the present disclosure.

Claims

1. An offline computing method of data, comprising:

and performing offline calculation based on the first data source according to the first calculation logic information and the first time dimension information of the second pre-calculation index to obtain a first pre-calculation result of the second pre-calculation index.

2. The method of claim 1, wherein generating a second pre-computed indicator of the at least one time dimension based on the first pre-computed indicator according to the time demand information of the target service comprises:

determining first time dimension information of at least one time dimension according to time demand information of a target service;

determining first calculation logic information according to second calculation logic information of the first pre-calculation index;

determining first data dimension information according to second data dimension information of the first pre-calculation index;

generating a second pre-calculation index of at least one time dimension according to the first time dimension information, the first calculation logic information and the first data dimension information.

3. The method of claim 2, wherein generating a second pre-computed indicator of at least one time dimension from the first time dimension information, the first computation logic information, and the first data dimension information comprises:

and under the condition that first time dimension information comprising a plurality of time dimensions is determined, generating a second pre-calculation index of each time dimension according to the first calculation logic information, the first data dimension information and the first time dimension information corresponding to each time dimension.

4. The method of claim 1, wherein determining a first data source from first data dimension information of the second pre-computed index comprises:

determining first data required to be used for offline calculation according to the first data dimension information of the second pre-calculation index;

5. The method of claim 1, further comprising:

determining a first calculation frequency according to the first time dimension information of the second pre-calculation index;

6. The method according to any one of claims 1 to 5, wherein before generating the second pre-calculated indicator of the at least one time dimension based on the first pre-calculated indicator according to the time demand information of the target service, further comprises:

and determining the candidate index as a first pre-calculation index in the case that the index calculation duration of the candidate index meets a duration threshold value and/or in the case that the index use frequency of the candidate index meets a frequency threshold value.

7. The method of any one of claims 1 to 5, further comprising:

determining a preset storage space corresponding to the second pre-calculation index;

and storing the first pre-calculation result into the preset storage space and associating the first pre-calculation result with the second pre-calculation index.

8. The method of claim 7, further comprising:

under the condition that an index calculation task is received, determining a target calculation index of the index calculation task;

and outputting a first pre-calculation result of the second pre-calculation index as a calculation result of the index calculation task under the condition that the second pre-calculation index matched with the target calculation index is stored in the preset storage space.

9. The method of claim 7, further comprising:

determining a second data source according to third data dimension information of the target calculation index under the condition that the second pre-calculation index matched with the target calculation index is not stored in the preset storage space;

according to the third calculation logic information of the target calculation index, carrying out real-time calculation based on the second data source to obtain a calculation result of the target calculation index;

And associating the calculation result with the target calculation index and storing the calculation result into a target storage space.

10. The method of claim 9, further comprising:

determining a second calculation frequency according to the second time dimension information of the target calculation index;

acquiring corresponding second data from the second data source under the condition that the calculation time of the target calculation index meets the second calculation frequency;

11. An offline computing device for data, comprising:

12. The apparatus of claim 11, wherein the generating module is configured to:

the first determining submodule is used for determining first time dimension information of at least one time dimension according to time demand information of the target service;

the second determining submodule is used for determining the first calculating logic information according to the second calculating logic information of the first pre-calculating index;

a third determining sub-module, configured to determine first data dimension information according to second data dimension information of the first pre-calculation index;

13. The apparatus of claim 12, wherein the generation sub-module is to:

14. The apparatus of claim 11, wherein the first determination module is configured to:

15. The apparatus of claim 11, further comprising:

the first frequency determining module is used for determining a first calculation frequency according to the first time dimension information of the second pre-calculation index;

the first acquisition module is used for acquiring corresponding first data from the first data source under the condition that the calculation time of the second pre-calculation index meets the first calculation frequency;

and the second calculation module is used for performing off-line calculation based on the first calculation logic information of the second pre-calculation index and the first data so as to obtain a first pre-calculation result of the second pre-calculation index.

16. The apparatus of any of claims 11 to 15, further comprising:

and the second determining module is used for determining the candidate index as a first pre-calculation index in the case that the index calculation duration of the candidate index meets a duration threshold value and/or in the case that the index use frequency of the candidate index meets a frequency threshold value.

17. The apparatus of any of claims 11 to 15, further comprising:

a third determining module, configured to determine a preset storage space corresponding to the second pre-calculation index;

and the first storage module is used for storing the first pre-calculation result into the preset storage space and associating the first pre-calculation result with the second pre-calculation index.

18. The apparatus of claim 17, further comprising:

a fourth determining module, configured to determine a target calculation index of an index calculation task when the index calculation task is received;

and the first matching module is used for outputting a first pre-calculation result of the second pre-calculation index as a calculation result of the index calculation task under the condition that the second pre-calculation index matched with the target calculation index is stored in the preset storage space.

19. The apparatus of claim 17, further comprising:

the second matching module is used for determining a second data source according to third data dimension information of the target calculation index under the condition that the second pre-calculation index matched with the target calculation index is not stored in the preset storage space;

The third calculation module is used for carrying out real-time calculation based on the second data source according to the third calculation logic information of the target calculation index so as to obtain a calculation result of the target calculation index;

and the second storage module is used for associating the calculation result with the target calculation index and storing the calculation result into a target storage space.

20. The apparatus of claim 19, further comprising:

the second frequency determining module is used for determining a second calculation frequency according to the second time dimension information of the target calculation index;

the second acquisition module is used for acquiring corresponding second data from the second data source under the condition that the calculation time of the target calculation index meets the second calculation frequency;

and the fourth calculation module is used for performing off-line calculation based on the third calculation logic information of the target calculation index and the second data so as to obtain a calculation result of the target calculation index.

21. An electronic device, comprising:

at least one processor; and

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10.

22. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1 to 10.

23. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 10.