CN112579590A - Data processing method, device, equipment and storage medium - Google Patents
Data processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN112579590A CN112579590A CN201910933441.XA CN201910933441A CN112579590A CN 112579590 A CN112579590 A CN 112579590A CN 201910933441 A CN201910933441 A CN 201910933441A CN 112579590 A CN112579590 A CN 112579590A
- Authority
- CN
- China
- Prior art keywords
- granularity
- data
- task
- interface
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a data processing method, a device, equipment and a storage medium, which are characterized in that business data to be processed in a data processing period are obtained, data granularity processing is carried out on the business data to be processed based on a pre-established data granularity model, the business data after data granularity is obtained, granularity information of a corresponding data granularity interface in the data granularity model is obtained based on a pre-determined data granularity interface, the data processing period is used as time granularity, a task to be dispatched is generated according to the time granularity and the granularity information, and the task to be dispatched is sent to a task queue. The purpose of efficiently generating the tasks to be dispatched is achieved, and the tasks to be dispatched are sent to the task queue, so that the efficiency of processing the service data is improved.
Description
Technical Field
The invention belongs to the technical field of data processing, and particularly relates to a data processing method, a data processing device, data processing equipment and a storage medium.
Background
With the advent of the information age, more and more business data are received by users through equipment, and when the data volume of the business data needing to be processed periodically is huge, the business data are divided, tasks are generated, and the tasks are dispatched to an input queue by a distributed method.
However, each time the service data is processed, the service data needs to be split, a task is generated, and the task is dispatched to the input queue, that is, how to split the service data needs to be considered in addition to the service data processing itself, so that the processing efficiency of the service data is reduced.
Disclosure of Invention
In view of the above, the present invention provides a data processing method, apparatus, device and storage medium, which are used to improve the processing efficiency of service data. The technical scheme is as follows:
a first aspect of an embodiment of the present invention provides a data processing method, where the method includes:
acquiring service data to be processed in a data processing period;
performing data granularity processing on the to-be-processed service data based on a pre-established data granularity model to obtain service data after data granularity processing;
acquiring granularity information corresponding to the data granularity interface in the data granularity model based on a predetermined data granularity interface;
and taking the data processing period as time granularity, generating a task to be dispatched according to the time granularity and the granularity information, and sending the task to a task queue.
Optionally, the process of pre-establishing the data granularity model includes:
defining granularity information for describing service data after data granularity, wherein the granularity information at least comprises a granularity identifier, a granularity name and a granularity type, the granularity identifier is used for indicating a unique identifier of the service data after the data granularity, the granularity name is used for indicating the service significance of the service data after the data granularity, and the granularity type is used for indicating the data type of the service data after the data granularity;
establishing the data granularity model based on the defined granularity information.
Optionally, the method further includes:
establishing a corresponding relation between the data granularity model and a scheduler;
setting a data processing period of the scheduler for scheduling the data in the data granularity model;
and creating a corresponding data granularity interface based on the granularity information, wherein different service types in the granularity name correspond to different types of data granularity interfaces.
Optionally, obtaining granularity information corresponding to the data granularity interface in the data granularity model based on a predetermined data granularity interface includes:
determining the type of granularity information corresponding to a data granularity interface used by the current execution calling granularity information, wherein the data granularity interface is predetermined;
if the data granularity interface is used for calling granularity identification, acquiring the granularity identification in the data processing period in the data granularity model based on the data granularity interface;
if the data granularity interface is used for calling the granularity name, acquiring the granularity name in the data processing period in the data granularity model based on the data granularity interface;
and if the data granularity interface is used for calling the granularity type, acquiring the granularity type in the data processing period in the data granularity model based on the data granularity interface.
Optionally, taking the data processing cycle as a time granularity, generating a task to be served according to the time granularity and the granularity information, and sending the task to a task queue, where the method includes:
if the granularity information is a granularity identifier, taking the data processing period as time granularity, generating a task to be dispatched according to the time granularity and the granularity identifier, and sending the task to a task queue;
if the granularity information is a granularity name, taking the data processing period as time granularity, generating a task to be dispatched according to the time granularity and the granularity name, and sending the task to a task queue;
and if the granularity information is of a granularity type, taking the data processing period as time granularity, generating a task to be dispatched according to the time granularity and the granularity type, and sending the task to a task queue.
A second aspect of the embodiments of the present invention provides a data processing apparatus, including:
the first obtaining module is used for obtaining service data to be processed in a data processing period;
the second obtaining module is used for carrying out data granularity processing on the to-be-processed service data based on a pre-established data granularity model to obtain the service data after data granularity processing;
the acquisition module is used for acquiring granularity information corresponding to the data granularity interface in the data granularity model based on a predetermined data granularity interface;
and the sending module is used for taking the data processing period as time granularity, generating a task to be dispatched according to the time granularity and the granularity information, and sending the task to a task queue.
Optionally, the second obtaining module includes:
the defining unit is used for defining granularity information used for describing data-granulized service data, wherein the granularity information at least comprises a granularity identifier, a granularity name and a granularity type, the granularity identifier is used for indicating a unique identifier of the data-granulized service data, the granularity name is used for indicating the service significance of the data-granulized service data, and the granularity type is used for indicating the data type of the data-granulized service data;
and the establishing unit is used for establishing the data granularity model based on the defined granularity information.
Optionally, the apparatus further comprises:
the establishing module is used for establishing the corresponding relation between the data granularity model and the scheduler;
the setting module is used for setting a data processing period of the data in the data granularity model scheduled by the scheduler;
and the creating module is used for creating corresponding data granularity interfaces based on the granularity information, wherein different service types in the granularity names correspond to different types of data granularity interfaces.
A third aspect of the embodiments of the present invention provides a storage medium, where the storage medium includes a stored program, and when the program runs, a device in which the storage medium is located is controlled to execute the data processing method according to the first aspect of the embodiments of the present invention.
A fourth aspect of the embodiments of the present invention provides a data processing apparatus, including a processor and a memory, where the memory stores a program, and the processor is configured to execute the program, where when the program runs, the data processing method according to the first aspect of the embodiments of the present invention is performed.
Compared with the prior art, the technical scheme provided by the invention has the following advantages:
the method comprises the steps of obtaining service data to be processed in a data processing period, conducting data granularity processing on the service data to be processed based on a pre-established data granularity model to obtain service data after data granularity, obtaining granularity information of a corresponding data granularity interface in the data granularity model based on a pre-determined data granularity interface, taking the data processing period as time granularity, generating a task to be dispatched according to the time granularity and the granularity information, and sending the task to be dispatched to a task queue. The purpose of efficiently generating the tasks to be dispatched is achieved, and the tasks to be dispatched are sent to the task queue, so that the efficiency of processing the service data is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a flow chart illustrating a data processing method according to an embodiment of the present invention;
FIG. 2 is a flow chart illustrating a process of pre-establishing a data granularity model according to an embodiment of the present invention;
FIG. 3 depicts a flowchart that illustrates registering a data granularity model in a data processing system, as provided by an embodiment of the invention;
fig. 4 is a flowchart illustrating obtaining granularity information of a corresponding data granularity interface in a data granularity model according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating a process of generating a task to be served according to time granularity and granularity information and sending the task to a task queue according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram illustrating a data processing apparatus according to an embodiment of the present invention.
Detailed Description
The invention provides a data processing method, a data processing device, data processing equipment and a storage medium, which are used for improving the processing efficiency of service data.
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As shown in fig. 1, a data processing method provided by an embodiment of the present invention is shown, and the method includes the following steps:
s101, obtaining service data to be processed in a data processing period.
In S101, the data processing cycle is a cycle in which a preset scheduler performs service data scheduling. That is, the service data is processed based on the data processing cycle. Preferably, the data processing period is set to be one hour, and the scheduler needs to process the traffic data in the last hour every other hour.
In the process of executing S101, the business data includes, but is not limited to, advertisement data, order data, and effectiveness data of the advertisement.
It should be noted that the type of the specific acquired service data is determined by the specific service.
And S102, performing data granularity processing on the service data to be processed based on the pre-established data granularity model to obtain the service data after data granularity.
In S102, the data granularity refers to a level of refinement or integration of the data in the data warehouse. According to the data granularity refinement standard: the higher the refinement degree is, the smaller the granularity is; the lower the degree of refinement, the larger the particle size.
In the process of executing S102, the service data to be processed in the obtained data processing cycle is subjected to data granularity processing through the data granularity model. It should be noted that the data granularity model is pre-established, and the data granularity model describes the service data based on the granularity information, so as to obtain the service data after data granularity. The granularity information includes at least a granularity identification, a granularity name, and a granularity type. The granularity information is not limited to this, and the granularity information may be set by the service according to the needs.
Therefore, after the service data is processed by the data granularity model, the service data divided according to the granularity can be obtained.
For example: according to the method, a certain e-commerce platform needs to summarize all order data of the last hour according to commodities every other hour, the order data are sent to a data granularity model, and the obtained order data after data granularity comprises data granularity with the granularity identification of Time of hour, the granularity name of Time and the granularity type of Time, data granularity with the granularity identification of Shop ID and the granularity type of numerical value.
It should be noted that, the specific implementation process of pre-establishing the data granularity model, as shown in fig. 2, mainly includes:
and S201, defining granularity information for describing the service data after data granularity.
In S201, the granularity information includes at least a granularity identification, a granularity name, and a granularity type.
The granularity identification is used for indicating the unique identification of the service data after the data granularity.
The granularity name is used for indicating the business significance of the business data after the data granularity.
The granularity type is used for indicating the data type of the service data after the data granularity.
In the process of executing S201, by defining the granularity identifier, the granularity identifier can be used to indicate a unique identifier of the service data after data granularity; the granularity name is defined, so that the granularity name can be used for indicating the service significance of the service data after data granularity; the granularity type is defined, so that the granularity type can be used for indicating the data type of the service data after the data granularity.
It should be noted that the data granularity model is established based on the defined granularity identification, the granularity name and the granularity type, but not limited to.
And S202, establishing a data granularity model based on the defined granularity information.
In the process of executing S202, a corresponding data storage table is established with the granularity identifier, the granularity name, and the granularity type as a header, and a data granularity model is created based on the data storage table. That is, the created data granularity model may embody the service data subjected to data granularity in the form of a data storage table.
For example: as shown in table 1, in order to summarize all order data in the last hour for a certain e-commerce platform after data granularity processing, a set of granularities (time granularity, store identification granularity) may be defined.
Table 1:
| granularity identification | Name of particle size | Type of particle size |
| TimeOfHour | Time | Time type |
| ShopID | Shop mark | Numerical type |
For example: as shown in table 2, in order to calculate the advertisement effectiveness data of the previous day for all the clients after the data graining process, a set of granularities (date granularity, client identification granularity, item identification granularity) may be defined.
Table 2:
| granularity identification | Name of particle size | Type of particle size |
| Date | Date | Date type |
| ProfileKey | Customer identification | Numerical type |
| CampaignKey | Item identification | Numerical type |
S103, acquiring granularity information of the corresponding data granularity interface in the data granularity model based on the predetermined data granularity interface.
In the process of executing S103, after the service data is subjected to data granularity processing by the data granularity model according to the granularity information defined in the data granularity model, the service data after data granularity is obtained, and the service data after data granularity is stored in the data granularity model. When the data processor cycle comes, the scheduler schedules the granularity information corresponding to the data granularity interface in the data granularity model based on the predetermined data granularity interface.
It should be noted that the data granularity interface is predetermined, and is specifically created when a pre-created data granularity model is registered in the data processing system.
Fig. 3 is a flowchart of registering a data granularity model in a data processing system according to an embodiment of the present invention, which mainly includes the following steps.
S301, establishing a corresponding relation between the data granularity model and the scheduler.
In S301, the data granularity model is a data granularity model created based on the method disclosed in fig. 2.
The scheduler is used for calling the data granularity interface so as to obtain the granularity information in the data granularity model.
In the process of executing S301, in order to obtain the granularity information in the data granularity model through the data granularity interface, it is necessary to establish a correspondence relationship between the data granularity model and the scheduler.
It should be noted that the scheduler can invoke any data granularity interface.
And S302, setting a data processing period of the data in the data granularity model scheduled by the scheduler.
In the process of executing S302, by setting the data processing period of the service data, if the current time reaches the data processing period of the service data, the scheduler starts to work, calls the corresponding data granularity interface, and calls the granularity information in the data granularity model.
For example: the data processing period of the service data is one day, and when the current time reaches the data processing period of the service data, the scheduler calls the corresponding data granularity interface to acquire the granularity information in the data granularity model.
It should be noted that the data processing period of the scheduler for scheduling data in the data granularity model may be set according to actual situations, and is not described herein again.
And S303, creating a corresponding data granularity interface based on the granularity information.
In the process of executing S303, a data granularity interface corresponding to the granularity identifier is created according to the granularity identifier, that is, the data granularity interface can only obtain granularity information under the granularity identifier from the data granularity model; according to the granularity name, a data granularity interface corresponding to the granularity name is created, namely the data granularity interface can only obtain granularity information under the granularity name from a data granularity model; and according to the granularity type, creating a data granularity interface corresponding to the granularity type, namely the data granularity interface can only obtain the granularity information under the granularity type from the data granularity model.
For example: as shown in table 1, the granularity identifier includes a store identifier, a data granularity interface corresponding to the store identifier is established, and when the granularity information of the store identifier needs to be acquired, the scheduler acquires the granularity information of the store identifier from the data granularity model through the data granularity interface corresponding to the store identifier.
It should be noted that the predetermined data granularity interface includes, but is not limited to, a data granularity interface corresponding to the store identification.
And S104, taking the data processing period as time granularity, generating a task to be dispatched according to the time granularity and the granularity information, and sending the task to a task queue.
In the process of executing S104, a data processing cycle is used as a time granularity, a task to be dispatched is generated according to the time granularity and the granularity information, and the task to be dispatched is sent to the task queue, so as to wait for processing of the granularity information.
For example: the data processing period of order data of a certain e-commerce platform is one hour, and if the current time is just one hour away from the last time of order data processing, the data processing period is one hour and serves as the time granularity. And generating a task to be dispatched according to the time granularity and the acquired granularity information, then sending the task to be dispatched to a task queue, and waiting for processing the granularity information.
It should be noted that, the data processing periods of the service data are different, and the corresponding time granularities are also different.
According to the data processing method disclosed by the embodiment of the invention, the service data to be processed in the data processing period is obtained, the data granularity processing is performed on the service data to be processed based on the pre-established data granularity model, the service data after data granularity is obtained, the service data after data granularity comprises the granularity identification, the granularity name and the granularity type, the granularity information of the corresponding data granularity interface in the data granularity model is obtained based on the pre-determined data granularity interface, the data processing period is used as the time granularity, the task to be dispatched is generated according to the time granularity and the granularity information, and the task to be dispatched is sent to the task queue. The purpose of efficiently generating the tasks to be dispatched is achieved, and the tasks to be dispatched are sent to the task queue, so that the efficiency of processing the service data is improved.
Based on the data processing method disclosed in fig. 1 in the embodiment of the present invention, in S103 shown in fig. 1, a specific implementation process for obtaining granularity information of a corresponding data granularity interface in a data granularity model based on a predetermined data granularity interface is mainly included, as shown in fig. 4:
s401, determining the type of the granularity information corresponding to the data granularity interface used for executing the current calling granularity information, if the data granularity interface is used for calling the granularity identification, executing S402, if the data granularity interface is used for calling the granularity name, executing S403, and if the data granularity interface is used for calling the granularity type, executing S404.
And S402, acquiring granularity identification in a data processing period in the data granularity model based on the data granularity interface.
And S403, acquiring the granularity name in the data processing period in the data granularity model based on the data granularity interface.
And S404, acquiring the granularity type in the data processing period in the data granularity model based on the data granularity interface.
According to the data processing method disclosed by the embodiment of the invention, the service data to be processed in the data processing period is obtained, the data granularity processing is performed on the service data to be processed based on the pre-established data granularity model, the service data after data granularity is obtained, the service data after data granularity comprises the granularity identification, the granularity name and the granularity type, the granularity information of the corresponding data granularity interface in the data granularity model is obtained based on the pre-determined data granularity interface, the data processing period is used as the time granularity, the task to be dispatched is generated according to the time granularity and the granularity information, and the task to be dispatched is sent to the task queue. The purpose of efficiently generating the tasks to be dispatched is achieved, and the tasks to be dispatched are sent to the task queue, so that the efficiency of processing the service data is improved.
Based on the data processing method disclosed in fig. 1 in the embodiment of the present invention, in S104 shown in fig. 1, a data processing cycle is taken as a time granularity, a task to be dispatched is generated according to the time granularity and granularity information, and is sent to a specific implementation process in a task queue, as shown in fig. 5, the specific implementation process mainly includes:
s501, determining whether the granularity information is granularity identification, granularity name or granularity type, if the granularity information is the granularity identification, executing S502, if the granularity information is the granularity name, executing S503, and if the granularity information is the granularity type, executing S504.
And S502, taking the data processing period as time granularity, generating a task to be dispatched according to the time granularity and the granularity identification, and sending the task to a task queue.
And S503, taking the data processing period as time granularity, generating a task to be dispatched according to the time granularity and the granularity name, and sending the task to a task queue.
S504, the data processing cycle is used as time granularity, the task to be dispatched is generated according to the time granularity and the granularity type, and the task to be dispatched is sent to the task queue.
According to the data processing method disclosed by the embodiment of the invention, the service data to be processed in the data processing period is obtained, the data granularity processing is performed on the service data to be processed based on the pre-established data granularity model, the service data after data granularity is obtained, the service data after data granularity comprises the granularity identification, the granularity name and the granularity type, the granularity information of the corresponding data granularity interface in the data granularity model is obtained based on the pre-determined data granularity interface, the data processing period is used as the time granularity, the task to be dispatched is generated according to the time granularity and the granularity information, and the task to be dispatched is sent to the task queue. The purpose of efficiently generating the tasks to be dispatched is achieved, and the tasks to be dispatched are sent to the task queue, so that the efficiency of processing the service data is improved.
The data processing method disclosed in the above embodiment of the present invention is exemplified. For example, the period is one hour, the service data is order data classified by certain e-commerce platform according to commodities, and the granularity information defined in the data granularity model includes granularity identification, granularity name and granularity type, and the granularity type includes time granularity and shop identification granularity. Accordingly, the predetermined data granularity interfaces are a granularity identification interface, a granularity name interface and a granularity type interface.
The specific process for processing the order data comprises the following steps:
s601: and acquiring all order data in the last hour summarized by the E-commerce platform according to the commodity types.
S602: and carrying out data granularity processing on the order data based on a pre-established data granularity model to obtain the order data after data granularity.
In S602, the order data divided into the granularity identifier of TimeOfHour, the granularity name of time, and the granularity type of time granularity, and the order data divided into the granularity identifier of shop id, the granularity name of shop identifier, and the granularity type of shop identifier granularity may be obtained.
S603: and determining that the data granularity interface used by the current execution scheduling granularity information is a granularity name interface, wherein the type of the corresponding granularity information is a shop identifier.
In S603, the type of the data granularity interface used specifically may be determined according to the current requirement.
S604: and obtaining order data which is described by taking the granularity name as a shop mark in the data granularity model based on the granularity name interface and the type of the granularity information.
S605: and taking one hour as time granularity, generating a task to be served according to the time granularity and order data which is described by taking the granularity name as the shop identifier, and sending the task to a task queue.
In S605, if there are 100 store identities, 100 tasks are generated according to the one-hour filling time granularity, and the tasks are sent to the task queue to wait for consumption.
According to the data processing method disclosed by the embodiment of the invention, based on the predetermined data granularity interface, the granularity information of the corresponding data granularity interface in the data granularity model is obtained, the data processing period is taken as the time granularity, the task to be dispatched is generated according to the time granularity and the granularity information, and the task to be dispatched is sent to the task queue. The purpose of efficiently generating the tasks to be dispatched is achieved, and the tasks to be dispatched are sent to the task queue, so that the efficiency of processing the service data is improved.
Based on the foregoing method for data processing disclosed in the embodiments of the present invention, a data processing apparatus is also correspondingly disclosed in the embodiments of the present invention, as shown in fig. 6, which is a schematic structural diagram of a data processing apparatus disclosed in the embodiments of the present invention, and includes: a first obtaining module 60, a second obtaining module 61, an obtaining module 62 and a sending module 63.
A first obtaining module 60, configured to obtain service data to be processed in a data processing cycle.
The second obtaining module 61 is configured to perform granularity processing on the service data to be processed based on a pre-established data granularity model, and obtain the service data after data granularity processing.
The obtaining module 62 is configured to obtain granularity information of a corresponding data granularity interface in the data granularity model based on a predetermined data granularity interface.
And the sending module 63 is configured to use the data processing period as a time granularity, generate a task to be dispatched according to the time granularity and the granularity information, and send the task to the task queue.
An alternative configuration of the second obtaining module 61 in the embodiment of the apparatus of the present invention is: the second obtaining module 61 comprises a defining unit and a building unit.
The definition unit is configured to define granularity information used for describing the data-granulized service data, where the granularity information at least includes a granularity identifier, a granularity name, and a granularity type, the granularity identifier is used to indicate a unique identifier of the data-granulized service data, the granularity name is used to indicate a service meaning of the data-granulized service data, and the granularity type is used to indicate a data type of the data-granulized service data.
And the establishing unit is used for establishing a data granularity model based on the defined granularity information.
An optional structure of the obtaining module 62 in the embodiment of the apparatus of the present invention is: the acquisition module 62 includes a determination unit, a first acquisition unit, a second acquisition unit, and a third acquisition unit.
And the determining unit is used for determining the type of the granularity information corresponding to the data granularity interface used by the current execution call granularity information, and the data granularity interface is predetermined.
And the first obtaining unit is used for obtaining the granularity identification in the data processing period in the data granularity model based on the data granularity interface if the data granularity interface is used for calling the granularity identification.
And the second obtaining unit is used for obtaining the granularity name in the data processing period in the data granularity model based on the data granularity interface if the data granularity interface is used for calling the granularity name.
And if the data granularity interface is used for calling the granularity type, the third obtaining unit obtains the granularity type in the data processing period in the data granularity model based on the data granularity interface.
Another optional structure of the obtaining module 62 in the embodiment of the apparatus of the present invention is: the obtaining module 62 includes a first sending unit, a second sending unit, and a third sending unit.
And the first sending unit is used for generating a task to be dispatched according to the time granularity and the granularity identification by taking the data processing period as the time granularity if the granularity information is the granularity identification, and sending the task to be dispatched to the task queue.
And the second sending unit is used for taking the data processing period as the time granularity if the granularity information is the granularity name, generating the task to be dispatched according to the time granularity and the granularity name, and sending the task to the task queue.
And the third sending unit is used for taking the data processing period as the time granularity if the granularity information is the granularity type, generating the task to be dispatched according to the time granularity and the granularity type, and sending the task to the task queue.
According to the data processing device disclosed in the embodiment of the present invention, to-be-processed service data in a data processing period is obtained, data granularity processing is performed on the to-be-processed service data based on a pre-established data granularity model, the data-granularity-processed service data is obtained, the data-granularity-processed service data includes a granularity identifier, a granularity name, and a granularity type, granularity information corresponding to a data granularity interface in the data granularity model is obtained based on a pre-determined data granularity interface, the data processing period is used as time granularity, a task to be dispatched is generated according to the time granularity and the granularity information, and the task to be dispatched is sent to a task queue. The purpose of efficiently generating the tasks to be dispatched is achieved, and the tasks to be dispatched are sent to the task queue, so that the efficiency of processing the service data is improved.
Based on the data processing device disclosed in the above embodiment of the present invention, the data processing device further includes: the device comprises an establishing module, a setting module and a creating module.
And the establishing module is used for establishing the corresponding relation between the data granularity model and the scheduler.
And the setting module is used for setting the data processing period of the data in the data granularity model scheduled by the scheduler.
And the creating module is used for creating corresponding data granularity interfaces based on the granularity information, wherein different service types in the granularity names correspond to different types of data granularity interfaces.
According to the data processing device disclosed in the embodiment of the present invention, to-be-processed service data in a data processing period is obtained, data granularity processing is performed on the to-be-processed service data based on a pre-established data granularity model, the data-granularity-processed service data is obtained, the data-granularity-processed service data includes a granularity identifier, a granularity name, and a granularity type, granularity information corresponding to a data granularity interface in the data granularity model is obtained based on a pre-determined data granularity interface, the data processing period is used as time granularity, a task to be dispatched is generated according to the time granularity and the granularity information, and the task to be dispatched is sent to a task queue. The purpose of efficiently generating the tasks to be dispatched is achieved, and the tasks to be dispatched are sent to the task queue, so that the efficiency of processing the service data is improved.
Based on the data processing apparatus disclosed in the above embodiment of the present invention, the above modules and units may be implemented by a hardware device composed of a processor and a memory. The method specifically comprises the following steps: the modules and units are stored in a memory as program units, and a processor executes the program units stored in the memory to realize data processing.
The processor comprises a kernel, and the kernel calls a corresponding program unit from the memory. The kernel can be set to be one or more, and data processing is realized by adjusting kernel parameters.
An embodiment of the present invention provides a storage medium on which a program is stored, the program implementing data processing when executed by a processor.
An embodiment of the present invention provides a processor, where the processor is configured to execute a program, where the program executes a data processing method in any one of fig. 1 to 5 when running.
An embodiment of the present invention provides a data processing apparatus 70, and as shown in fig. 7, a schematic structural diagram of the data processing apparatus 70 provided in the embodiment of the present invention is shown.
The data processing device in the embodiment of the present invention may be a server, a PC, a PAD, a mobile phone, or the like.
The data processing device comprises at least one processor 701, and at least one memory 702 connected to the processor, and a bus 703.
The processor 701 and the memory 702 communicate with each other via a bus 703. A processor 701 for executing the program stored in the memory 702.
A memory 702 for storing a program for at least: acquiring service data to be processed in a data processing period; performing data granularity processing on service data to be processed based on a pre-established data granularity model to obtain the service data after data granularity processing; acquiring granularity information of a corresponding data granularity interface in a data granularity model based on a predetermined data granularity interface; and taking the data processing period as time granularity, generating a task to be dispatched according to the time granularity and the granularity information, and sending the task to the task queue.
The present application further provides a computer program product adapted to perform a program for initializing the following method steps when executed on a data processing device:
the method comprises the steps of obtaining service data to be processed in a data processing period, conducting data granularity processing on the service data to be processed based on a pre-established data granularity model, obtaining granularity information of the service data after data granularity processing based on a pre-determined data granularity interface, obtaining the granularity information of the corresponding data granularity interface in the data granularity model, taking the data processing period as time granularity, generating a task to be dispatched according to the time granularity and the granularity information, and sending the task to a task queue.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a device includes one or more processors (CPUs), memory, and a bus. The device may also include input/output interfaces, network interfaces, and the like.
The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should be noted that, in the present specification, the embodiments are all described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other. For the device-like embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (10)
1. A method of data processing, the method comprising:
acquiring service data to be processed in a data processing period;
performing data granularity processing on the to-be-processed service data based on a pre-established data granularity model to obtain service data after data granularity processing;
acquiring granularity information corresponding to the data granularity interface in the data granularity model based on a predetermined data granularity interface;
and taking the data processing period as time granularity, generating a task to be dispatched according to the time granularity and the granularity information, and sending the task to a task queue.
2. The method of claim 1, wherein pre-establishing a data granularity model comprises:
defining granularity information for describing service data after data granularity, wherein the granularity information at least comprises a granularity identifier, a granularity name and a granularity type, the granularity identifier is used for indicating a unique identifier of the service data after the data granularity, the granularity name is used for indicating the service significance of the service data after the data granularity, and the granularity type is used for indicating the data type of the service data after the data granularity;
and establishing the data granularity model based on the defined granularity identification, the granularity name and the granularity type.
3. The method of claim 2, further comprising:
establishing a corresponding relation between the data granularity model and a scheduler;
setting a data processing period of the scheduler for scheduling the data in the data granularity model;
and creating a corresponding data granularity interface based on the granularity information, wherein different service types in the granularity name correspond to different types of data granularity interfaces.
4. The method of claim 1, wherein obtaining granularity information corresponding to a data granularity interface in the data granularity model based on a predetermined data granularity interface comprises:
determining the type of granularity information corresponding to a data granularity interface used by the current execution calling granularity information, wherein the data granularity interface is predetermined;
if the data granularity interface is used for calling granularity identification, acquiring the granularity identification in the data processing period in the data granularity model based on the data granularity interface;
if the data granularity interface is used for calling the granularity name, acquiring the granularity name in the data processing period in the data granularity model based on the data granularity interface;
and if the data granularity interface is used for calling the granularity type, acquiring the granularity type in the data processing period in the data granularity model based on the data granularity interface.
5. The method according to claim 4, wherein taking the data processing cycle as a time granularity, generating a task to be served according to the time granularity and the granularity information, and sending the task to a task queue, comprises:
if the granularity information is a granularity identifier, taking the data processing period as time granularity, generating a task to be dispatched according to the time granularity and the granularity identifier, and sending the task to a task queue;
if the granularity information is a granularity name, taking the data processing period as time granularity, generating a task to be dispatched according to the time granularity and the granularity name, and sending the task to a task queue;
and if the granularity information is of a granularity type, taking the data processing period as time granularity, generating a task to be dispatched according to the time granularity and the granularity type, and sending the task to a task queue.
6. A data processing apparatus, characterized in that the apparatus comprises:
the first obtaining module is used for obtaining service data to be processed in a data processing period;
the second obtaining module is used for carrying out data granularity processing on the to-be-processed service data based on a pre-established data granularity model to obtain the service data after data granularity processing;
the acquisition module is used for acquiring granularity information corresponding to the data granularity interface in the data granularity model based on a predetermined data granularity interface;
and the sending module is used for taking the data processing period as time granularity, generating a task to be dispatched according to the time granularity and the granularity information, and sending the task to a task queue.
7. The apparatus of claim 6, wherein the second obtaining module comprises:
the defining unit is used for defining granularity information used for describing data-granulized service data, wherein the granularity information at least comprises a granularity identifier, a granularity name and a granularity type, the granularity identifier is used for indicating a unique identifier of the data-granulized service data, the granularity name is used for indicating the service significance of the data-granulized service data, and the granularity type is used for indicating the data type of the data-granulized service data;
and the establishing unit is used for establishing the data granularity model based on the defined granularity information.
8. The apparatus of claim 6, further comprising:
the establishing module is used for establishing the corresponding relation between the data granularity model and the scheduler;
the setting module is used for setting a data processing period of the data in the data granularity model scheduled by the scheduler;
and the creating module is used for creating corresponding data granularity interfaces based on the granularity information, wherein different service types in the granularity names correspond to different types of data granularity interfaces.
9. A storage medium, characterized in that the storage medium includes a stored program, wherein a device on which the storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 5 when the program runs.
10. A data processing apparatus comprising a processor and a memory, the memory having a program stored therein, the processor being configured to execute the program, wherein the program when executed performs the data processing method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910933441.XA CN112579590A (en) | 2019-09-29 | 2019-09-29 | Data processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910933441.XA CN112579590A (en) | 2019-09-29 | 2019-09-29 | Data processing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112579590A true CN112579590A (en) | 2021-03-30 |
Family
ID=75111075
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910933441.XA Pending CN112579590A (en) | 2019-09-29 | 2019-09-29 | Data processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112579590A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030063562A1 (en) * | 2001-09-21 | 2003-04-03 | Terago Communications, Inc. | Programmable multi-service queue scheduler |
| CN1897566A (en) * | 2005-07-14 | 2007-01-17 | 中兴通讯股份有限公司 | System and method for realizing convergent point service quality guarantee based on class grading |
| US20090248473A1 (en) * | 2008-03-31 | 2009-10-01 | Susanne Doenig | Managing Consistent Interfaces for Business Objects Across Heterogeneous Systems |
| CN101937370A (en) * | 2010-08-16 | 2011-01-05 | 中国科学技术大学 | Method and device supporting system-level resource distribution and task scheduling on FCMP (Flexible-core Chip Microprocessor) |
| WO2018171322A1 (en) * | 2017-03-20 | 2018-09-27 | 中兴通讯股份有限公司 | Virtual network function and method for implementing service processing thereof, and storage medium |
| CN108763482A (en) * | 2018-05-25 | 2018-11-06 | 联想(北京)有限公司 | Method, apparatus and server cluster for data analysis flow scheduling |
| CN110069334A (en) * | 2019-05-05 | 2019-07-30 | 重庆天蓬网络有限公司 | A kind of method and system based on the distributed data job scheduling for assuring reason |
| CN110275763A (en) * | 2019-05-09 | 2019-09-24 | 北京善义善美科技有限公司 | A kind of alert data processing system and method |
-
2019
- 2019-09-29 CN CN201910933441.XA patent/CN112579590A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030063562A1 (en) * | 2001-09-21 | 2003-04-03 | Terago Communications, Inc. | Programmable multi-service queue scheduler |
| CN1897566A (en) * | 2005-07-14 | 2007-01-17 | 中兴通讯股份有限公司 | System and method for realizing convergent point service quality guarantee based on class grading |
| US20090248473A1 (en) * | 2008-03-31 | 2009-10-01 | Susanne Doenig | Managing Consistent Interfaces for Business Objects Across Heterogeneous Systems |
| CN101937370A (en) * | 2010-08-16 | 2011-01-05 | 中国科学技术大学 | Method and device supporting system-level resource distribution and task scheduling on FCMP (Flexible-core Chip Microprocessor) |
| WO2018171322A1 (en) * | 2017-03-20 | 2018-09-27 | 中兴通讯股份有限公司 | Virtual network function and method for implementing service processing thereof, and storage medium |
| CN108763482A (en) * | 2018-05-25 | 2018-11-06 | 联想(北京)有限公司 | Method, apparatus and server cluster for data analysis flow scheduling |
| CN110069334A (en) * | 2019-05-05 | 2019-07-30 | 重庆天蓬网络有限公司 | A kind of method and system based on the distributed data job scheduling for assuring reason |
| CN110275763A (en) * | 2019-05-09 | 2019-09-24 | 北京善义善美科技有限公司 | A kind of alert data processing system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109102135B (en) | Order allocation method and device | |
| CN109408205B (en) | Task scheduling method and device based on hadoop cluster | |
| CN107276970B (en) | Unbinding and binding method and device | |
| CN112449750A (en) | Log data collection method, log data collection device, storage medium, and log data collection system | |
| CN110827002B (en) | Approval method based on workflow and workflow engine | |
| CN112801763B (en) | Touch and reach scheme generation method and device and electronic equipment | |
| CN107295052B (en) | Service processing method and device | |
| WO2016197853A1 (en) | Complexity-based service processing method and apparatus | |
| CN114036031A (en) | Scheduling system and method for resource service application in enterprise digital middleboxes | |
| CN110019260B (en) | User data updating method and related equipment | |
| CN110764930B (en) | Request or response processing method and device based on message mode | |
| CN113132429B (en) | Method and device for pushing messages | |
| CN113034178A (en) | Multi-system integral calculation method and device, terminal equipment and storage medium | |
| CN112114976A (en) | Service processing method, device, equipment and storage medium | |
| CN112579590A (en) | Data processing method, device, equipment and storage medium | |
| WO2016169420A1 (en) | Method, device and system for querying service request execution state | |
| CN111177093A (en) | Method, device and medium for sharing scientific and technological resources | |
| CN116028696A (en) | Resource information acquisition method and device, electronic equipment and storage medium | |
| CN116170502A (en) | Message service system, method and message service platform | |
| CN112509164A (en) | Attendance card-punching method, attendance card-punching device, attendance card-punching equipment and storage medium | |
| CN112507193A (en) | Data updating method, device, equipment and storage medium | |
| CN112860292A (en) | Configuration management method and device based on application | |
| CN112182507A (en) | Data quality measuring method, device and equipment | |
| CN111625866A (en) | Authority management method, system, equipment and storage medium | |
| CN117785486B (en) | Environment resource allocation method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |