CN110135917B

CN110135917B - Data processing method, data processing device and system

Info

Publication number: CN110135917B
Application number: CN201910454770.6A
Authority: CN
Inventors: 张蛟; 李豆豆; 李爽; 高廉墀; 刘彦伟; 李芳利; 李亚周
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Priority date: 2019-05-28
Filing date: 2019-05-28
Publication date: 2021-09-03
Anticipated expiration: 2039-05-28
Also published as: CN110135917A

Abstract

The present disclosure provides a data processing method, including: acquiring first order information of a real-time generated order; determining whether the combined order information is complete according to the order reference information, wherein the combined order information comprises first order information and second order information of a real-time generated order stored in the first storage space; and under the condition that the combined order information is complete, generating order data for generating the order in real time according to a preset rule. The present disclosure also provides a data processing apparatus and a data processing system.

Description

Data processing method, data processing device and system

Technical Field

The present disclosure relates to the field of electronic technologies, and in particular, to a data processing method, a data processing apparatus, and a data processing system.

Background

With the development of information technology, the data volume also has explosive growth, the way for people to acquire information becomes more diversified and more convenient, and the requirement of people on the timeliness of information becomes higher and higher. Especially for key data such as orders and the like, timeliness of the data plays an important role in analyzing purchasing behaviors of website visitors, recommending articles in real time, accurately delivering advertisements, analyzing delivering effects and the like.

In the course of implementing the disclosed concept, the inventors found that there are at least the following problems in the prior art: in the prior art, the calculation of order data and the like is mainly off-line calculation, that is, all order data in the previous day are uniformly calculated by adopting a mode of scheduling tasks regularly when a user accesses and places fewer orders every morning, and although the mode can better ensure the accuracy of the data. However, offline data calculation has a great disadvantage in timeliness, and the requirement for calculating real-time data cannot be met. Moreover, the prior art can only show the final correct data and cannot show the data in the order modification process.

Disclosure of Invention

In view of the above, the present disclosure provides a data processing method, a data processing apparatus and a system capable of generating real-time order data.

One aspect of the present disclosure provides a data processing method, including: acquiring first order information of a real-time generated order; determining whether the combined order information is complete according to the order reference information, wherein the combined order information comprises first order information and second order information of a real-time generated order stored in the first storage space; and under the condition that the combined order information is complete, generating order data for generating the order in real time according to a preset rule.

According to an embodiment of the present disclosure, the merged order information further includes one or more pieces of order detail information, and the order reference information includes a reference correspondence between the order and the number of the order detail information. The determining whether the consolidated order information is complete includes: determining the corresponding relation between the real-time generated order and the number of the order detail information included in the combined order information; and under the condition that the corresponding relation meets the reference corresponding relation, determining that the combined order information is complete.

According to an embodiment of the present disclosure, before generating order data for generating an order in real time according to a predetermined rule, the data processing method further includes: writing the first order information into the first storage space to update the second order information to obtain updated order information; and determining whether the updated order information and the second order information have different key information or not under the condition that the second order information is complete. And generating order data for generating the order in real time according to a preset rule under the condition that the combined order information is complete and different key information exists. The order data includes data information corresponding to the key information.

According to an embodiment of the present disclosure, the first order information includes order identification information for generating an order in real time, and before the data processing method generates order data for generating an order in real time according to a predetermined rule, the data processing method further includes: when the combined order information is complete and different key information exists, storing order identification information included in the first order information into a second storage space; generating order data for generating an order in real time according to a predetermined rule includes: acquiring order identification information in a second storage space; acquiring updated order information in the first storage space according to the order identification information; and generating order data for generating the order in real time according to the updated order information and the preset rule.

According to an embodiment of the present disclosure, the order identification information includes one or more identification codes, and the generating order data of the real-time generated order according to the updated order information and the predetermined rule includes: generating data information for generating the order in real time according to the updated order information and a predetermined algorithm; acquiring additional information corresponding to one or more identification codes included in the order identification information from the third storage space; and generating order data for generating the order in real time according to the data information and the additional information. The first storage space and the third storage space are different subspaces of the first cache space, and the first cache space and the second storage space are different storage spaces.

According to an embodiment of the present disclosure, the data processing method further includes: acquiring supplementary information of the additional information in real time; in case that the supplemental information is acquired, the additional information in the third storage space is updated.

According to an embodiment of the present disclosure, the obtaining the first order information of the real-time generated order includes: and obtaining a plurality of pieces of first order information of the order generated in real time in a multithreading mode, wherein the first order information of the same order obtained in the multithreading mode is sequentially cached in a second cache space according to the obtained time sequence. The writing the first order information into the first storage space includes: and sequentially writing the plurality of first order information into the first storage space according to the time sequence cached in the second cache space. The second cache space and the first storage space are different storage spaces.

Another aspect of the present disclosure provides a data processing apparatus including an order information acquisition module, a first determination module, and an order data generation module. The order information acquisition module is used for acquiring first order information of the real-time generated order. The first determining module is used for determining whether the combined order information is complete according to the order reference information, wherein the combined order information comprises the first order information and second order information of the real-time generated order stored in the first storage space. And the order data generation module is used for generating order data of the real-time generated order according to a preset rule under the condition that the combined order information is complete.

According to an embodiment of the present disclosure, the merged order information further includes one or more pieces of order detail information, and the order reference information includes a reference correspondence between the order and the number of the order detail information. The first determining module comprises a corresponding relation determining submodule and an information integrity determining submodule. And the corresponding relation determining submodule is used for determining the corresponding relation between the real-time generated order and the number of the order detail information included in the combined order information. And the information integrity determination submodule is used for determining that the information of the combined order is complete under the condition that the corresponding relation meets the reference corresponding relation.

According to an embodiment of the present disclosure, the data processing apparatus further includes an order information writing module and a second determining module. The order information writing module is used for writing the first order information into the first storage space so as to update the second order information and obtain the updated order information. The second determining module is used for determining whether the updated order information and the second order information have different key information or not under the condition that the second order information is complete. The order data generation module is specifically used for generating order data for generating an order in real time according to a preset rule under the condition that the combined order information is complete and different key information exists. The order data includes data information corresponding to the key information.

According to an embodiment of the present disclosure, the first order information includes order identification information of the real-time generated order. The data processing device further comprises an order identification information writing module, and the order identification information writing module is used for storing the order identification information included in the first order information into the second storage space under the condition that the combined order information is complete and different key information exists. The order data generation module comprises an order identification information acquisition submodule, an order information acquisition submodule and a data generation submodule. The order identification information acquisition submodule is used for acquiring the order identification information in the second storage space. The order information acquisition submodule is used for acquiring the updated order information in the first storage space according to the order identification information. And the data generation submodule is used for generating order data of the real-time generated order according to the updated order information and the preset rule.

According to an embodiment of the present disclosure, the order identification information includes one or more identification codes. The data generation submodule comprises a data information generation unit, an additional information acquisition unit and a data generation unit. And the data information generating unit is used for generating data information of the real-time generated order according to the updated order information and the preset algorithm. The additional information acquisition unit is used for acquiring additional information corresponding to one or more identification codes included by the order identification information from the third storage space. The data generating unit is used for generating order data of the order in real time according to the data information and the additional information. The first storage space and the third storage space are different subspaces of the first cache space, and the first cache space and the second storage space are different storage spaces.

According to an embodiment of the present disclosure, the data processing apparatus further includes a supplementary information obtaining module and an additional information updating module. The supplementary information acquisition module is used for acquiring supplementary information of the additional information in real time. The additional information updating module is used for updating the additional information in the third storage space under the condition that the supplemental information is acquired.

According to an embodiment of the present disclosure, the order information obtaining module is configured to obtain multiple pieces of first order information of an order generated in real time in a multithreading manner, and the multiple pieces of first order information of the same order obtained in the multithreading manner are sequentially cached in the second cache space according to the obtained time sequence. The order information writing module is specifically configured to sequentially write the plurality of first order information into the first storage space according to the time sequence cached in the second cache space. The second cache space and the first storage space are different storage spaces.

Another aspect of the present disclosure provides a data processing system comprising a data acquisition processing device and a data storage device. The data acquisition processing device is connected with the data source and used for acquiring first order information and order reference information of the real-time generated order from the data source and generating order data of the real-time generated order according to a preset rule under the condition that the combined order information is complete, wherein the combined order information comprises the first order information and second order information of the real-time generated order stored in the first storage space. The data storage device is provided with a coprocessor and a first storage space. The first storage space is used for storing second order information. The coprocessor is used for determining whether the combined order information is complete according to the order reference information.

Another aspect of the disclosure provides a data processing system comprising one or more processors, and a storage device. The storage device is used to store one or more programs. Wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the data processing methods described above.

Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions for implementing the data processing method as described above when executed.

Another aspect of the present disclosure provides a computer program comprising computer executable instructions for implementing the data processing method as described above when executed.

According to the embodiment of the disclosure, the defect of poor timeliness existing in calculation by adopting offline data in the prior art can be at least partially avoided, and therefore, the instantaneity problem of the offline data is better made up through real-time access and real-time calculation of order information, and the timeliness of the order data is improved.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:

fig. 1A schematically illustrates an application scenario of a data processing method, a data processing apparatus and a system according to an embodiment of the present disclosure;

FIG. 1B schematically illustrates an example diagram of a data processing apparatus and system suitable for implementing the data processing method, according to an embodiment of the disclosure;

fig. 2 schematically shows a flow chart of a data processing method according to a first embodiment of the present disclosure;

FIG. 3 schematically illustrates a flow chart for determining whether consolidated order information is complete, according to an embodiment of the disclosure;

fig. 4 schematically shows a flow chart of a data processing method according to a second embodiment of the present disclosure;

FIG. 5A schematically shows a flow chart of a data processing method according to a third embodiment of the present disclosure;

FIG. 5B schematically illustrates a flow chart for generating order data for a real-time generated order, according to an embodiment of the disclosure;

fig. 6 schematically shows a flow chart of a data processing method according to a fourth embodiment of the present disclosure;

FIG. 7 schematically illustrates a flow chart for generation of order data for a single order according to an embodiment of the present disclosure;

FIG. 8 schematically shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure; and

fig. 9 schematically shows a block diagram of a computer system suitable for implementing a data processing method according to an embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.

Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

The embodiment of the disclosure provides a data processing method, device and system capable of generating real-time order data. The data processing method comprises the following steps: acquiring first order information of a real-time generated order; determining whether the combined order information is complete according to the order reference information, wherein the combined order information comprises first order information and second order information of a real-time generated order stored in the first storage space; and under the condition that the combined order information is complete, generating order data for generating the order in real time according to a preset rule.

Fig. 1A schematically illustrates an application scenario 100 of a data processing method, a data processing apparatus and a system according to an embodiment of the present disclosure. FIG. 1B schematically illustrates an example diagram of a data processing apparatus and system suitable for implementing the data processing method, according to an embodiment of the disclosure. It is understood that fig. 1A is only an example of an application scenario in which the embodiment of the present disclosure may be applied, and fig. 1B is only an example of an example scenario in which the embodiment of the present disclosure may be implemented to help a person skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiment of the present disclosure may not be used in other devices, systems, environments or scenarios, nor does it mean that the embodiment of the present disclosure may not be implemented by other examples.

As shown in FIG. 1A, an application scenario according to this embodiment includes a first computing cluster 101, a second computing cluster 102, a network 103, and a database 104. The network 103 is used to provide a medium for communication links between the first computing cluster 101, the second computing cluster 102, and the database 104. Network 103 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

Therein, the first computing cluster 101 may for example integrate data processing and data acquisition functionality. The first computing cluster 101 is configured to interact with the database 104 through the network 103 to obtain the order information and the order reference information from the database 104. The first computing cluster 101 is further configured to interact with the second computing cluster 102 through the network 103, so as to write the obtained order information into the second computing cluster 102, and update the order information stored in the second computing cluster 102. The first computing cluster 101 is further configured to analyze and process order information stored in the second computing cluster 102, generate order data, and store the order data in the database 104 for a user to obtain and call.

According to an embodiment of the disclosure, as shown in fig. 1B, the first computing cluster 101 may specifically have a Flink cluster (distributed stream processing open source framework), and include a plurality of threads (e.g., thread 1 to thread 9), and the plurality of threads may interact with the database 104 and the second computing cluster 102 in parallel to improve throughput of the first computing cluster 101.

According to the embodiment of the disclosure, a plurality of programs may be run on the Flink cluster, and the plurality of programs are respectively used for executing different functions, for example, a first program to a third program may be run. Wherein the first program is used to interact with the database 104 to obtain order information. The second program is used to interact with the second computing cluster 102 to write order information to the second computing cluster 102. The third program is used for interacting with the second computing cluster 102 to obtain order information stored in the second computing cluster 102, and obtaining order data and the like through computation according to the order information. It is understood that the number and functions of the programs running on the Flink cluster are only examples to facilitate understanding of the present disclosure, and the present disclosure is not limited thereto.

The second computing cluster 102 integrates a data caching function, so as to cache the order information written by the first computing cluster 101. The second computing cluster 102 may further have a preprocessing function, for example, to determine whether the combined order information of the order information to be written by the Flink cluster and the order information stored in the second computing cluster is complete, and if so, determine that the Flink cluster can perform order data computation.

According to an embodiment of the present disclosure, as shown in fig. 1B, the second computing cluster 102 may specifically be, for example, an Hbase cluster (distributed, column-oriented open database) for storing order information written by the Flink cluster in real time. The Hbase cluster is for example further integrated with a co-processor having the described pre-processing functionality. According to the embodiment of the present disclosure, in order to facilitate unified management, the preprocessing function may also be integrated in a Flink cluster, for example, and is not described herein again.

As shown in fig. 1A, for example, the database 104 may also be connected to a database server 105 integrated with an order generation platform, so as to store order information of an order generated by the order generation platform in real time, and then the database 104 may serve as a data source to provide order information for the Flink cluster.

According to an embodiment of the present disclosure, the database 104 may be, for example, a database integrated with Kafka (high throughput distributed publish-subscribe messaging system for processing all action flow data in a consumer-scale website), and the order information stored by the database 104 may be put into a specific topic of Kafka. As shown in fig. 1B, this particular topic may include, for example, the following: the Order table (Order table) for storing Order time, payment information and the like, the Order Detail table (Order Detail table) for storing Order Detail information, the discount table (Ocs table) for storing Order discount information, the discount Detail table (Ocs _ Ext table) for storing discount Detail information, the first relation table (Ocs-Ebs table) for storing the corresponding relation between the discount information and the number of the discount Detail information, and the second relation table (Order-submit table) for storing the corresponding relation between the Order and the number of the Order Detail information.

According to an embodiment of the present disclosure, the preprocessing function is specifically configured to determine whether the order information is complete according to a corresponding relationship in the first relationship table and a corresponding relationship in the second relationship table, and when the order information satisfies the two corresponding relationships, it indicates that the order information is complete.

According to an embodiment of the present disclosure, in order to facilitate calculation of order data by the Flink cluster, the Hbase cluster may further communicate with the database 104 through the network 103, so as to store identification information of an order with complete order information as intermediate data topic into the database 104. The Flink cluster can obtain the order information of the order with complete order information from the Hbase cluster by obtaining the identification information in the intermediate data topic, and calculate to obtain the order data of the order. Meanwhile, order data obtained by calculation of the Flink cluster can be stored in the database 104 as real-time topic. Meanwhile, in order to make the order data persistent, the order data in the real-time topic can be formed into a real-time wide table.

According to the embodiment of the disclosure, the order data is generated for further improvement considering that the identification information of the order is often composed of a plurality of identification codes with specified meanings. As shown in fig. 1B, the database 104 may further store a plurality of dimension tables topic updated in real time, for example, an Rdim table for storing description fields of order types (third party orders, self-service orders, virtual orders, or the like), an XXX _ Sku _ Chain table for storing description fields of Product features (for example, Product colors, or the like), an XXX _ Product _ Chain table for storing description fields of commodity types (wash class, home class), and a Brands _ Chain table for storing description fields of Brands ("standing white", "carving cards", or the like). The user can add, delete and/or modify the topics and the specific information in the topics according to the actual requirement. Accordingly, the Flink cluster is also used to obtain the specific information in the dimension tables topic and maintain the information in the Hbase cluster. Accordingly, when an addition, deletion, and/or modification of information in the database 104 occurs, the Flink cluster may also synchronously update the changed information to the Hbase cluster for the Flink cluster to call when subsequently generating order data.

It is understood that the types and numbers of the first computing cluster 101, the second computing cluster 102, and the database 104 are merely illustrative, and the first computing cluster 101 and the second computing cluster 102 may be the same computing cluster according to implementation requirements. The first computing cluster 101 may also include other open source frameworks than the Flink cluster, the second computing cluster 102 may also include other open source databases than Hbase, and the database 104 may also integrate other messaging systems for processing action flow data besides Kafka.

It should be noted that the data processing method provided by the embodiment of the present disclosure may be generally executed by a first computing cluster, and accordingly, the data processing apparatus provided by the embodiment of the present disclosure may be generally disposed in the first computing cluster. The data processing method provided by the embodiment of the present disclosure may also be partially executed by the first computing cluster, and partially executed by the second computing cluster. Correspondingly, the data processing apparatus provided by the embodiment of the present disclosure may also be partially disposed in the first computing cluster, and partially integrated in the second computing cluster.

Fig. 2 schematically shows a flow chart of a data processing method according to a first embodiment of the present disclosure.

As shown in fig. 2, the data processing method of the embodiment of the present disclosure includes operations S201 to S203.

In operation S201, first order information of a real-time generated order is acquired.

According to an embodiment of the present disclosure, the first order information may be specifically obtained by the first computing cluster 101 in fig. 1A from the database 104. The order information stored in the database 104 may be obtained by: the online system provided by the database server 105 updates binlog (binary log) information related to an order when a general user places an order, and uses information of interest of a downstream user obtained by analyzing the binlog information as order information. The first order information may include, for example, the number of items in the order, the selling price of each item, and offer information. Specifically, the first Order information may include, for example, any one or more of a plurality of information stored in an Orders table, an Orders Detail table, an Ocs table, and an Ocs _ Ext table in fig. 1B.

According to the embodiment of the present disclosure, storing the order information into the database may specifically be to put the order information into a specific topic of Kafka, so that the first computing cluster 101 may obtain the order information in the specific topic in real time. Considering that there may be many new, modified, and deleted information stored in real time in the order generation process, when the first computing cluster 101 acquires the order information in real time, it may often only acquire partial information of the order generated in real time, that is, the first order information in the order information.

According to the embodiment of the disclosure, a multithreading mode can be adopted to obtain a plurality of pieces of first order information of the real-time generated order. In order to avoid the situation that the first order information obtained by multiple threads is stored in a disordered manner, the first order information of the same order obtained by multiple threads may be sequentially cached in the second cache space according to the obtained time sequence, for example. The second cache space may specifically be a cache space of the first computing cluster 101.

In operation S202, it is determined whether the consolidated order information is complete according to the order reference information.

The combined order information comprises first order information and second order information of the real-time generated order stored in the first storage space. According to the embodiment of the present disclosure, considering that in the process of generating a certain order, the process of the first computing cluster 101 obtaining the order information of the certain order and writing the order information of the certain order into the second computing cluster 102 is performed step by step, it is impossible to write all the order information of the certain order into the second computing cluster 102 synchronously at one time. Therefore, when determining whether the order information of the order is complete, it should be determined whether the consolidated order information composed of the first order information acquired in real time and the already stored second order information is complete. Accordingly, the first storage space may specifically be a subspace of the first cache space in the second computing cluster 102. According to the embodiment of the disclosure, in order to facilitate determining second order information which belongs to the real-time generated order together with the first order information and is stored in the first storage space, when the second order information is written into the first storage space, for example, order identification information which can uniquely represent the real-time generated order, such as an order number of the real-time generated order, can be written as an index. Accordingly, the first order information should also include, for example, order identification information of the real-time generated order to represent the order to which the first order information belongs.

According to an embodiment of the present disclosure, the order reference information may include a reference amount of order information that the order should include, for example, and the operation S202 determines whether the merged order information is complete, for example, may be: and comparing the order information quantity included in the combined order information with the reference quantity, and if the order information quantity included in the combined order information is not less than the reference quantity, determining that the combined order information is complete. According to an embodiment of the present disclosure, if the order information includes the order detail information and the benefit detail information described in fig. 1B, the order reference information may further include reference amounts of the order detail information and the benefit detail information that the order should include. Similarly, in operation S202, only when the number of the order detail information and the offer detail information included in the consolidated order information both satisfy the reference amount, the consolidated order information is complete. It is to be understood that the above-mentioned types of order reference information and the specific implementation method of operation S202 are only examples to facilitate understanding of the present disclosure, and the present disclosure does not limit this.

According to an embodiment of the present disclosure, the operation S202 may be specifically performed for a coprocessor in the Hbase cluster in fig. 1B, for example. Specifically, the coprocessor may determine, after writing order information into the Hbase cluster each time, whether all the order information of the real-time generated order has been written into the Hbase cluster according to the order information of the real-time generated order stored in the written Hbase cluster. Specifically, for example, whether the order details are all written and whether the offer details are all written may be determined according to the number of offer details generated in real time in the Ocs-Ebs table in fig. 1B and the number of order details in the Orderpipe-submit table. And if all the information is written, judging that the obtained merged order information is complete. According to the embodiment of the disclosure, after the merged order information is judged to be complete, in order to facilitate the call when the order data is generated subsequently, for example, an identifier for indicating the complete status may be added to the real-time generated order in the Hbase cluster.

According to an embodiment of the present disclosure, this operation S202 may also be implemented by the method described in fig. 3, for example, and is not described in detail herein.

In operation S203, in the case that the combined order information is complete, the order data of the real-time generated order is generated according to a predetermined rule.

According to the embodiment of the disclosure, the calculation of the order data can be performed under the condition that the combined order information is complete. The order data may specifically include, for example, an actual payment amount, a benefit amount, a freight charge, a group purchase amount of the real-time generated order, and/or a payment amount, a benefit amount, and the like of each item in the real-time generated order. The predetermined rule may be, for example, an algorithm for calculating an actual payment amount according to the order information, and the algorithm may be any algorithm in the prior art, which is not described herein again.

According to an embodiment of the present disclosure, the operation S203 may be specifically implemented by the method described in fig. 5A to 5B, and is not described in detail herein.

In summary, the data processing method according to the embodiment of the disclosure can better solve the problem of poor instantaneity of offline data by generating real-time access and acquisition of order information of an order and calculating order data in real time. Furthermore, by adopting multithreading, asynchronous and batch order information acquisition and calculation, the concurrent processing amount can be improved, and the requirement of a large peak value period and the like on the calculation processing capacity can be met.

Fig. 3 schematically illustrates a flow chart for determining whether consolidated order information is complete, according to an embodiment of the disclosure.

According to an embodiment of the present disclosure, the consolidated Order information may specifically include one or more Order Detail information in an Order Detail table. The order reference information may include, for example, a reference correspondence 1 between the order and the number of order detail information: m (second relationship stored in the Orderpipe-submit table), the reference correspondence may be the same or different for different orders. As shown in fig. 3, operation S202 in fig. 2 may specifically include operations S312 to S322.

In operation S312, a correspondence between the real-time generated order and the number of the order detail information included in the consolidated order information is determined. In operation S322, in the case where the correspondence satisfies the reference correspondence, it is determined that the consolidated order information is complete. According to an embodiment of the present disclosure, the operations S312 to S322 may specifically be: the number n of the order detail information in the combined order information belonging to the real-time generated order is determined. Then, the corresponding relation between the number of the real-time generated order and the order detail information is determined to be 1: n. It is then determined whether the correspondence 1: n is equal to the reference correspondence 1: m. If the order information is equal, the corresponding relation meets the reference corresponding relation, and the integrated order information is determined to be complete; if not, the corresponding relation is not satisfied with the reference corresponding relation, and the incomplete merged order information is determined. Wherein n and m are positive integers.

According to the embodiment of the disclosure, the merged Order information may include, for example, one or more offer Detail information in the Ocs _ Ext table in addition to one or more Order Detail information in the Order Detail table. The order reference information should include not only the second relationship stored in the Orderpipe-submit table but also the first relationship stored in the Ocs-Ebs table. Operation S202 in fig. 2 may determine whether the correspondence between the numbers of the order details information included in the real-time generated order and the consolidated order information satisfies the second relationship, and also determine whether the correspondence between the numbers of the benefit details information included in the real-time generated order and the consolidated order information satisfies the first relationship, and determine that the consolidated order information is complete only when both the first relationship and the second relationship are satisfied.

Fig. 4 schematically shows a flow chart of a data processing method according to a second embodiment of the present disclosure.

According to the embodiment of the present disclosure, in order to make the information in the first storage space record all the data generated in the order generation process, the first order information of the real-time generated order obtained in operation S201 should also be written in the first storage space in real time. As shown in fig. 4, the data processing method according to the embodiment of the present disclosure further includes an operation S404, writing the first order information into the first storage space to update the second order information, so as to obtain updated order information.

According to the embodiment of the disclosure, the first storage space may be provided with a plurality of partitions, for example, each partition is used for storing order information of one order, and each partition takes the order identification information of each order as an index. The operation S404 may specifically be that the first order information is written into a partition indexed by the order identification information of the order generated in real time in the first storage space. Furthermore, according to the embodiment of the present disclosure, in order to better store each piece of intermediate order information generated in the order generation process, when writing new order information, non-overlay writing may be adopted to avoid that old order information is overlaid. According to the embodiment of the present disclosure, in order to well distinguish newly written order information from earlier written order information, the written order information may also be time-stamped each time the first order information is written, for example, while writing.

According to an embodiment of the present disclosure, when performing details information obtaining in multiple threads and writing a plurality of first details information, in order to avoid data processing disorder caused by obtaining multiple threads, the operation S404 may be specifically implemented by: and sequentially writing the plurality of first order information into the first storage space according to the time sequence cached in the second cache space.

According to the embodiment of the present disclosure, it is considered that writing operation needs to be performed on the Hbase cluster if a mark indicating complete is added for a real-time generated order after it is determined that the merged order information is complete. In order to reduce the writing operation to the Hbase cluster, the obtained first order information may be written into the Hbase cluster after determining whether the order information is complete in operation S202, and when it is determined that the order information is complete, a flag indicating complete may be written at the same time. Accordingly, as shown in fig. 4, operation S404 may be performed between operation S202 and operation S203.

Fig. 5A schematically shows a flow chart of a data processing method according to a third embodiment of the present disclosure.

According to the embodiment of the disclosure, considering that the first order information is obtained in real time, after the second order information stored in the first storage space is complete order information, the order generated in real time may have some changes of addition and deletion due to the requirement of the ordinary user. In this case, although the order information is changed, the final order data may not be changed. Without causing a change in order data, if the order information is complete and is calculated again, unnecessary overhead is undoubtedly caused. Therefore, in order to solve the above problem, as shown in fig. 5A, the data processing method according to the embodiment of the present disclosure may further include operation S505, in addition to operations S201 to S203, and operation S404, where operation S505 is performed before operation S203.

In operation S505, in the case where the second order information is complete, it is determined whether there is different key information between the updated order information and the second order information.

According to an embodiment of the present disclosure, the key information may specifically be, for example, information that may affect data information in the calculated order data. Accordingly, the order data of the real-time generated order obtained in operation S203 includes data information corresponding to the key information. According to the embodiment of the present disclosure, the key information may specifically be, for example, information that may affect data related to the amount of money in the finally obtained order data, and may include offer detail information, the number of items, unit price of the items, and the like. The order information without affecting the data information may be information such as an article delivery address, consignee information, and the like.

Accordingly, operation S203 may be performed only if the consolidated order information is complete and different key information exists, for example.

According to the embodiment of the present disclosure, in order to facilitate distinguishing whether to calculate order data, for example, identification information of an order for which order data needs to be calculated may be placed in a predetermined space, and calculation of order data is performed as long as the order identification information exists in the predetermined space. Therefore, as shown in fig. 5A, the data processing method according to the embodiment of the present disclosure may further include an operation S506 of, in a case that the merged order information is complete and different key information exists, storing the order identification information included in the first order information into the second storage space.

The second storage space may be, for example, a storage space of the database in fig. 1A. Specifically, the operation S506 may be specifically executed by a coprocessor in the Hbase cluster, and is configured to write order identification information (specifically, an order number) meeting the condition order into the intermediate data topic described in fig. 1B, so that the Flink cluster acquires the order.

Accordingly, as shown in fig. 5A, operation S203 may include operations S513 to S533.

In operation S513, order identification information in the second storage space is acquired; in operation S523, according to the order identification information, the updated order information in the first storage space is obtained; in operation S533, order data for generating an order in real time is generated according to the updated order information and the predetermined rule.

According to an embodiment of the present disclosure, the operations S513 to S533 may specifically include: and the Flink cluster acquires order identification information in the intermediate data topic. And then according to the obtained order identification information, obtaining the latest order information of the order corresponding to the order identification information from the Hbase cluster. And finally, calculating order data of the order according to the obtained latest order information. According to an embodiment of the present disclosure, operation S533 may be specifically performed by, for example, operations S5133 to S5333 described in subsequent fig. 5B, and will not be described in detail herein.

In summary, in the embodiment of the present disclosure, the order information is regenerated only when the updated order information and the second order information stored in the Hbase cluster at an earlier stage have key information that affects the order data, so that redundancy of the finally obtained order data can be effectively avoided, and therefore, unnecessary calculation overhead can be saved. Furthermore, under the condition that order data are obtained, the order information is regenerated as long as the updated order information and second order information stored in the Hbase cluster at the earlier stage have key information which affects the order data, so that the finally obtained order data can reflect the final data of each order, and a downstream user can know the intermediate process of each order according to other order data except the final order data.

FIG. 5B schematically shows a flow diagram for generating order data for a real-time generated order, according to an embodiment of the disclosure.

According to an embodiment of the present disclosure, it is considered that the order identification information is generally composed of a plurality of identification codes, and each identification code indicates an assigned meaning. The information indicated by each identification code may also be obtained for further enrichment of the generated order data. Operation S533 in fig. 5A may include operations S5133 to S5333, as shown in fig. 5B.

In operation S5133, data information for generating an order in real time is generated according to the updated order information and a predetermined algorithm.

The predetermined algorithm may specifically be an algorithm for calculating the obtained data information, and may be, for example, an algorithm for calculating a preferential amount of each item in the order, an algorithm for calculating an actual payment amount of each item in the order, an algorithm for calculating a profit amount of the self-owned platform, an algorithm for calculating a profit amount of the third-party platform, or the like. The predetermined algorithm may adopt various existing algorithms, and accordingly, the data information of the real-time generated order may include the preferential amount information of each item, the actual payment amount information of each item, the profit amount information of the computing platform, the profit amount information of the third party platform, and the like, which is not described in detail herein.

In operation S5233, additional information corresponding to one or more identification codes included in the order identification information is acquired from the third storage space.

According to the embodiment of the disclosure, in order to facilitate use when order data is finally generated, the Flink cluster may also write additional information corresponding to all identification codes into the Hbase cluster in advance for cache maintenance. Then, when operation S5233 is performed, it is only necessary to obtain corresponding additional information from the Hbase cluster according to one or more identification codes included in the order identification information. Accordingly, the third storage space is a certain storage subspace of the Hbase cluster. Specifically, in order to maintain the additional information and the order information of the order, respectively, the first storage space storing the order information and the third storage space storing the additional information may be two different subspaces in the Hbase cluster (first cache space).

According to an embodiment of the present disclosure, the additional information stored in the third storage space may specifically include, for example, a description field indicating characteristics (e.g., type of item, brand of item, color of item) of each item included in the order, and may also include information such as the type of the order and the number of items included in the order.

After the additional information is obtained, operation S5333 may be performed to generate order data for generating an order in real time according to the data information and the additional information. Specific examples may be: and splicing the additional information and the data information to obtain order data for generating the order in real time. According to an embodiment of the present disclosure, the operation S5333 may specifically be to directly write the generated data information and additional information into the real-time topic in fig. 1B, so as to obtain browsing by a downstream user.

According to an embodiment of the present disclosure, in order to make order data persistent, operation S5333 may further include generating a real-time wide table according to the order data in the real-time topic, so that a downstream user directly browses.

Fig. 6 schematically shows a flow chart of a data processing method according to a fourth embodiment of the present disclosure.

In order to avoid the incompleteness or the error of the additional information in the third storage space caused by the addition, deletion or change of the meaning indicated by the identification code, the data processing method of the embodiment of the present disclosure should further include an operation of maintaining the additional information in the third storage space. The data processing method of the embodiment of the present disclosure further includes operations S607 to S608, as shown in fig. 6.

In operation S607, supplementary information of the additional information is acquired in real time. In operation S608, in the case where the supplementary information is acquired, the additional information in the third storage space is updated.

According to an embodiment of the present disclosure, the operations S607 to S608 may be specifically performed by a Flink cluster. The method can be realized by the following operations: the Flink cluster monitors whether the dimension table topic shown in fig. 1B has supplementary information of additional information, that is, whether the Rdim table, the XXX _ Sku _ Chain table, the XXX _ Product _ Chain table and/or the Brands _ Chain table in the dimension table topic stores information. And if the information is stored, the information in the dimension table topic is acquired and the acquired information is written into a third storage space in the Hbase cluster if the additional information corresponding to the identification code is changed. And if the information is not stored in each table in the dimension table topic, the additional information corresponding to the identification code is not changed, and the operation of adding, deleting or changing the additional information in the third storage space is not executed.

FIG. 7 schematically illustrates a flow chart for generating order data for a single order according to an embodiment of the disclosure.

As shown in fig. 7, combining the methods shown in fig. 2 to fig. 6, the generating operation of the order data of a single order in the embodiment of the present disclosure specifically includes: firstly, accessing a Flink cluster to generate order information of an order in real time, and then judging whether all the order information of the order generated in real time reaches an Hbase cluster by a coprocessor, namely judging whether the order information accessed by the Flink cluster and the order information of the order generated in real time stored in the Hbase cluster are complete. If all the order information is received, writing the order information accessed by the Flink cluster into the Hbase cluster, and calculating a generation index, namely calculating data information, according to the updated order information of the real-time generated order; and then inquiring related dimension table data stored in a third storage space in the Hbase cluster according to a plurality of identification codes included in the order number of the real-time generated order to obtain additional information so as to supplement the dimension table data in the order data. And finally, sending the obtained data information and the supplemented dimension table data to the real-time topic of kafka in the database to complete the generation of order data.

As can be seen from the methods described in fig. 2 to fig. 7, the data processing method provided in the embodiment of the present disclosure can perform real-time access and real-time calculation on order and dimension table data, can better make up for the instantaneity problem of offline data, and can implement calculation and generation of data meeting conditions in seconds. Meanwhile, writing and reading of the Hbase cluster are optimized, namely parallel processing amount of more than 8000 ten thousand pieces of data per 5 minutes can be supported by adopting multithreading, asynchronization and batch writing and reading modes, and 99.99% of data can be processed within 1 second as long as the computing capability of computing equipment can be met even in the operation peak period. Moreover, intermediate state data for all orders can be calculated and recorded, thereby facilitating tracking the entire process of the lifecycle of the order data.

Fig. 8 schematically shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure.

As shown in fig. 8, the data processing apparatus 800 of the embodiment of the present disclosure includes an order information acquisition module 810, a first determination module 820, and an order data generation module 830.

The order information acquiring module 810 is configured to acquire first order information of an order generated in real time (operation S201).

The first determining module 820 is used to determine whether the merged order information is complete according to the order reference information (operation S202). The merged order information comprises first order information and second order information of real-time generated orders stored in the first storage space.

The order data generating module 830 is configured to generate order data of the real-time generated order according to a predetermined rule when the combined order information is complete (operation S203).

According to an embodiment of the present disclosure, the merged order information includes one or more pieces of order detail information, and the order reference information includes a reference correspondence between the order and the number of the order detail information. As shown in fig. 8, the first determining module 820 may include a correspondence determining submodule 821 and an information integrity determining submodule 822. The correspondence relation determining sub-module 821 is configured to determine a correspondence relation between the real-time generated order and the number of the order detail information included in the consolidated order information (operation S312). The information integrity determination submodule 822 is configured to determine that the consolidated order information is complete if the correspondence satisfies the reference correspondence (operation S322).

According to an embodiment of the present disclosure, as shown in fig. 8, the data processing apparatus 800 may further include an order information writing module 840 and a second determining module 850. The order information writing module 840 is configured to write the first order information into the first storage space to update the second order information, so as to obtain updated order information (operation S404). The second determining module 850 is configured to determine whether different key information exists between the updated order information and the second order information if the second order information is complete (operation S505). The order data generating module 830 is specifically configured to generate order data for generating an order in real time according to a predetermined rule under the condition that the combined order information is complete and different key information exists. The order data includes data information corresponding to the key information.

According to an embodiment of the present disclosure, the first order information includes order identification information of the real-time generated order. As shown in fig. 8, the data processing apparatus 800 further includes an order identification information writing module 860, configured to store the order identification information included in the first order information into the second storage space when the merged order information is complete and different key information exists (operation S506). The order data generating module 830 includes an order identification information obtaining sub-module 831, an order information obtaining sub-module 832 and a data generating sub-module 833. The order identification information obtaining sub-module 831 is configured to obtain order identification information in the second storage space (operation S513). The order information obtaining sub-module 832 is configured to obtain the updated order information in the first storage space according to the order identification information (operation S523). The data generation sub-module 833 is configured to generate order data for generating an order in real time according to the updated order information and the predetermined rule (operation S533).

According to an embodiment of the present disclosure, the order identification information includes one or more identification codes. As shown in fig. 8, the data generation sub-module 833 includes a data information generation unit 8331, an additional information acquisition unit 8332, and a data generation unit 8333. The data information generating unit 8331 is configured to generate data information of the real-time generated order according to the updated order information and the predetermined algorithm (operation S5133). The additional information acquisition unit 8332 is configured to acquire additional information corresponding to one or more identification codes included in the order identification information from the third storage space (operation S5233). The data generating unit 8333 is configured to generate order data of the real-time generated order according to the data information and the additional information (operation S5333). The first storage space and the third storage space are different subspaces of the first cache space, and the first cache space and the second storage space are different storage spaces.

According to an embodiment of the present disclosure, as shown in fig. 8, the data processing apparatus 800 may further include a supplementary information obtaining module 870 and an additional information updating module 880. The supplementary information acquiring module 870 serves to acquire supplementary information of the additional information in real time (operation S607). The additional information updating module 880 is configured to update the additional information in the third storage space if the supplemental information is acquired (operation S608).

According to an embodiment of the present disclosure, the order information obtaining module 810 is configured to obtain multiple pieces of first order information of an order generated in real time in a multi-thread manner, and sequentially cache the multiple pieces of first order information of the same order obtained in the multi-thread manner in the second cache space according to the obtained time sequence. The order information writing module 840 is specifically configured to sequentially write the plurality of first order information into the first storage space according to the time sequence buffered in the second buffer space. The second cache space and the first storage space are different storage spaces.

The present disclosure also provides a data processing system comprising a data acquisition processing device and a data storage device. The data acquisition processing device is connected with the data source and used for acquiring first order information and order reference information of the real-time generated order from the data source and generating order data of the real-time generated order according to a preset rule under the condition that the combined order information is complete, wherein the combined order information comprises the first order information and second order information of the real-time generated order stored in the first storage space. The data storage device is provided with a coprocessor and a first storage space. The first storage space is used for storing second order information. The coprocessor is used for determining whether the combined order information is complete according to the order reference information. According to an embodiment of the present disclosure, the data acquisition processing device may specifically be the first computing cluster 101 described in fig. 1A, the data source may specifically be the database 104 or the database server 105 described in fig. 1A, and the data storage device may be the second computing cluster 102 described in fig. 1A, which is not described herein again.

Any number of modules, sub-modules, units, sub-units, or at least part of the functionality of any number thereof according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in any other reasonable manner of hardware or firmware by integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware implementations. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the disclosure may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.

For example, any plurality of the order information acquisition module 810, the first determination module 820, the order data generation module 830, the order information writing module 840, the second determination module 850, the order identification information writing module 860, the supplemental information acquisition module 870, the additional information update module 880, the correspondence determination sub-module 821, the information integrity determination sub-module 822, the order identification information acquisition sub-module 831, the order information acquisition sub-module 832, the data generation sub-module 833, the data information generation unit 8331, the additional information acquisition unit 8332, and the data generation unit 8333 may be combined and implemented in one module, or any one of them may be split into a plurality of modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. According to the embodiment of the present disclosure, at least one of the order information obtaining module 810, the first determining module 820, the order data generating module 830, the order information writing module 840, the second determining module 850, the order identification information writing module 860, the supplementary information obtaining module 870, the additional information updating module 880, the correspondence determining sub-module 821, the information integrity determining sub-module 822, the order identification information obtaining sub-module 831, the order information obtaining sub-module 832, the data generating sub-module 833, the data information generating unit 8331, the additional information obtaining unit 8332, and the data generating unit 8333 may be at least partially implemented as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or any other reasonable manner that may integrate or package a circuit, or in any one of three implementations, software, hardware and firmware, or in any suitable combination of any of them. Alternatively, at least one of the order information acquisition module 810, the first determination module 820, the order data generation module 830, the order information writing module 840, the second determination module 850, the order identification information writing module 860, the supplementary information acquisition module 870, the additional information update module 880, the correspondence relation determination submodule 821, the information integrity determination submodule 822, the order identification information acquisition submodule 831, the order information acquisition submodule 832, the data generation submodule 833, the data information generation unit 8331, the additional information acquisition unit 8332, and the data generation unit 8333 may be at least partially implemented as a computer program module, and when the computer program module is run, a corresponding function may be performed.

Fig. 9 schematically shows a block diagram of a computer system suitable for implementing a data processing method according to an embodiment of the present disclosure. The computer system illustrated in FIG. 9 is only one example and should not impose any limitations on the scope of use or functionality of embodiments of the disclosure.

As shown in fig. 9, a computer system 900 according to an embodiment of the present disclosure includes a processor 901 which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)902 or a program loaded from a storage section 908 into a Random Access Memory (RAM) 903. Processor 901 may comprise, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 901 may also include on-board memory for caching purposes. The processor 901 may comprise a single processing unit or a plurality of processing units for performing the different actions of the method flows according to embodiments of the present disclosure.

In the RAM 903, various programs and data necessary for the operation of the system 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. The processor 901 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM 902 and/or the RAM 903. Note that the programs may also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 may also perform various operations of the method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.

System 900 may also include an input/output (I/O) interface 905, input/output (I/O) interface 905 also connected to bus 904, according to an embodiment of the present disclosure. The system 900 may also include one or more of the following components connected to the I/O interface 905: an input portion 906 including a keyboard, a mouse, and the like; an output section 907 including components such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 908 including a hard disk and the like; and a communication section 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the internet. The drive 910 is also connected to the I/O interface 905 as necessary. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 910 as necessary, so that a computer program read out therefrom is mounted into the storage section 908 as necessary.

According to embodiments of the present disclosure, method flows according to embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 909, and/or installed from the removable medium 911. The computer program, when executed by the processor 901, performs the above-described functions defined in the system of the embodiment of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM 902 and/or the RAM 903 described above and/or one or more memories other than the ROM 902 and the RAM 903.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.

The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.

Claims

1. A method of data processing, comprising:

acquiring first order information of a real-time generated order;

determining whether the combined order information is complete according to the order reference information, wherein the combined order information comprises the first order information and second order information of the real-time generated order stored in the first storage space; the order reference information comprises reference quantities of order detail information and preference detail information; and

under the condition that the combined order information is complete, generating order data of the real-time generated order according to a preset rule; the predetermined rule includes an algorithm for calculating an actual payment amount according to the order information;

wherein the first order information includes order identification information of the real-time generated order, and before generating order data of the real-time generated order according to the predetermined rule, the method further includes:

when the merged order information is complete and different key information exists, storing order identification information included in the first order information into a second storage space;

the generating order data of the real-time generated order according to the predetermined rule comprises:

acquiring order identification information in the second storage space;

acquiring updated order information in the first storage space according to the order identification information; the updated order information is used for generating order data of the real-time generated order;

the key information is information of data related to money in order data; and the order identification information is used as an index of the order partition.

2. The method of claim 1, wherein consolidated order information further comprises one or more order detail information, said order reference information comprising a reference correspondence of orders to numbers of order detail information, said determining whether said consolidated order information is complete comprising:

determining the corresponding relation between the real-time generated order and the number of the order detail information included in the combined order information;

and determining that the merged order information is complete under the condition that the corresponding relation meets the reference corresponding relation.

3. The method of claim 1, wherein prior to generating order data for the real-time generated order according to the predetermined rule, the method further comprises:

writing the first order information into the first storage space to update the second order information to obtain updated order information; and

determining whether the updated order information and the second order information have different key information under the condition that the second order information is complete,

under the condition that the combined order information is complete and different key information exists, generating order data of the real-time generated order according to the preset rule; the order data includes data information corresponding to the key information.

4. The method of claim 1, wherein the order identification information comprises one or more identification codes, and generating the order data of the real-time generated order according to the updated order information and a predetermined rule comprises:

generating data information of the real-time generated order according to the updated order information and a preset algorithm;

acquiring additional information corresponding to one or more identification codes included in the order identification information from a third storage space; and

generating order data of the real-time generated order according to the data information and the additional information,

the first storage space and the third storage space are different subspaces of a first cache space, and the first cache space and the second storage space are different storage spaces.

5. The method of claim 4, further comprising:

acquiring supplementary information of the additional information in real time;

and updating the additional information in the third storage space when the supplementary information is acquired.

6. The method of claim 3, wherein:

the step of obtaining first order information of the real-time generated order comprises the following steps: obtaining a plurality of pieces of first order information of the real-time generated order in a multithreading mode, wherein the first order information of the same order obtained in the multithreading mode is sequentially cached in a second cache space according to the obtained time sequence;

the writing the first order information into the first storage space includes: sequentially writing the plurality of first order information into the first storage space according to the time sequence cached in the second cache space,

the second cache space and the first storage space are different storage spaces.

7. A data processing apparatus comprising:

the order information acquisition module is used for acquiring first order information of a real-time generated order;

the first determining module is used for determining whether the combined order information is complete according to the order reference information, wherein the combined order information comprises the first order information and second order information of the real-time generated order stored in the first storage space; the order reference information comprises reference quantities of order detail information and preference detail information; the combined order information comprises first order information and second order information of a real-time generated order stored in the first storage space; and

the order data generation module is used for generating the order data of the real-time generated order according to a preset rule under the condition that the combined order information is complete; the predetermined rule includes an algorithm for calculating an actual payment amount according to the order information;

wherein the first order information includes order identification information of the real-time generated order:

the device also comprises an order identification information writing module which is used for storing the order identification information included in the first order information into a second storage space under the condition that the combined order information is complete and different key information exists;

the order data generation module comprises:

the order identification information acquisition sub-module is used for acquiring order identification information in the second storage space;

the order information acquisition sub-module is used for acquiring the updated order information in the first storage space according to the order identification information; and

the data generation submodule is used for generating order data of the real-time generated order according to the updated order information and the preset rule;

8. The apparatus of claim 7, wherein the consolidated order information further comprises one or more order detail information, the order reference information comprising a reference correspondence of orders to numbers of order detail information, the first determining module comprising:

the corresponding relation determining submodule is used for determining the corresponding relation between the real-time generated order and the number of the order detail information included in the combined order information; and

and the information integrity determining submodule is used for determining that the merged order information is complete under the condition that the corresponding relation meets the reference corresponding relation.

9. The apparatus of claim 7, further comprising:

an order information writing module, configured to write the first order information into the first storage space, so as to update the second order information, and obtain updated order information;

a second determining module, configured to determine whether the updated order information and the second order information have different key information or not when the second order information is complete,

the order data generation module is specifically configured to generate order data of the real-time generated order according to the predetermined rule under the condition that the merged order information is complete and different key information exists, where the order data includes data information corresponding to the key information.

10. The apparatus of claim 7, wherein the order identification information includes one or more identification codes, the data generation submodule comprising:

the data information generating unit is used for generating the data information of the real-time generated order according to the updated order information and a preset algorithm;

an additional information acquisition unit configured to acquire additional information corresponding to one or more identification codes included in the order identification information from a third storage space; and

a data generating unit for generating order data of the real-time generated order according to the data information and the additional information,

11. The apparatus of claim 10, further comprising:

the supplementary information acquisition module is used for acquiring supplementary information of the additional information in real time; and

and the additional information updating module is used for updating the additional information in the third storage space under the condition of acquiring the supplementary information.

12. The apparatus of claim 7 or 9, wherein:

the order information acquisition module is used for acquiring a plurality of pieces of first order information of the real-time generated order in a multithreading mode, and the plurality of pieces of first order information of the same order acquired in the multithreading mode are sequentially cached in a second cache space according to the acquired time sequence;

the order information writing module is specifically configured to write the plurality of first order information into the first storage space in sequence according to the time sequence cached in the second cache space,

13. A data processing system comprising:

the data acquisition processing device is connected with the data source and is used for:

acquiring first order information and order reference information of a real-time generated order from the data source; the order reference information comprises reference quantities of order detail information and preference detail information; and

under the condition that the combined order information is complete, generating order data of the real-time generated order according to a preset rule, wherein the combined order information comprises the first order information and second order information of the real-time generated order stored in a first storage space; the combined order information comprises first order information and second order information of a real-time generated order stored in the first storage space; the predetermined rule includes an algorithm for calculating an actual payment amount according to the order information;

a data storage device provided with a coprocessor and the first storage space, wherein:

the first storage space is used for storing the second order information;

the coprocessor is used for determining whether the combined order information is complete according to the order reference information;

wherein the first order information includes order identification information of the real-time generated order, and before generating order data of the real-time generated order according to the predetermined rule, the system further includes:

acquiring order identification information in the second storage space;

14. A data processing system comprising:

one or more processors;

a storage device for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the data processing method of any of claims 1-6.