CN114253957A

CN114253957A - Data processing method, related device, storage medium and computer program product

Info

Publication number: CN114253957A
Application number: CN202111565889.4A
Authority: CN
Inventors: 纪永; 吴筱帆; 闵鑫鑫
Original assignee: Ping An Securities Co Ltd
Current assignee: Ping An Securities Co Ltd
Priority date: 2021-12-20
Filing date: 2021-12-20
Publication date: 2022-03-29

Abstract

The embodiment of the application discloses a data processing method, related equipment, a storage medium and a computer program product, wherein the method comprises the following steps: after receiving a business object to generate real-time business data in the security business, storing the real-time business data in a queue data structure by adopting message middleware; acquiring real-time service data from the message middleware by adopting a stream processing engine, analyzing and processing the real-time service data, and determining the clearing type of the real-time service data; if the clearing type of the real-time service data is the target clearing type, a stream processing engine is adopted to carry out data clearing processing on the real-time service data to obtain a data clearing result; the data clearing result is used for indicating the first asset change information of the business object, and real-time data clearing of real-time business data generated in the securities business can be achieved, so that the efficiency of clearing the securities business for the securities business is improved.

Description

Data processing method, related device, storage medium and computer program product

Technical Field

The present application relates to the field of computer technologies, and in particular, to a data processing method, a related device, a storage medium, and a computer program product.

Background

With the rapid development of the internet and the improvement of the quality of life of people, the number of users for carrying out security trading in the security trading market is greatly increased. It is understood that when there is a security trade in the security trade market, data clearing is required based on trade data (such as real-time business data mentioned in the embodiments of the present application) generated by a business object (or called a user), so as to realize security business clearing of the security business. The term "stock clearing" is understood to mean: in each business day, the number and price of the securities traded by each security company are respectively offset, and the receivable or payable net amount of the securities and funds is calculated. The computer equipment is adopted to carry out security service clearing on the security service, so that the manpower and material resources consumed by the service object in the security delivery process can be saved. Therefore, how to improve the clearing efficiency of the securities business becomes a current research hotspot.

Disclosure of Invention

The embodiment of the application provides a data processing method, related equipment, a storage medium and a computer program product, which can improve the data clearing efficiency in security business.

In one aspect, an embodiment of the present application provides a data processing method, including:

after receiving a business object to generate real-time business data in security business, storing the real-time business data in a queue data structure by adopting message middleware;

acquiring the real-time service data from the message middleware by adopting a stream processing engine, analyzing and processing the real-time service data, and determining the clearing type of the real-time service data;

if the clearing type of the real-time service data is the target clearing type, performing data clearing processing on the real-time service data by adopting the stream processing engine to obtain a data clearing result; the data clearing result is used for indicating first asset change information of the business object.

In another aspect, an embodiment of the present application provides a data processing apparatus, including:

the storage unit is used for storing the real-time service data in a queue data structure by adopting message middleware after receiving the real-time service data generated by the service object in the security service;

the data analysis unit is used for acquiring the real-time service data from the message middleware by adopting a stream processing engine, performing data analysis processing on the real-time service data and determining the clearing type of the real-time service data;

the data clearing unit is used for adopting the stream processing engine to carry out data clearing processing on the real-time service data to obtain a data clearing result if the clearing type of the real-time service data is a target clearing type; the data clearing result is used for indicating first asset change information of the business object.

In another aspect, an embodiment of the present application provides a computer device, including:

a processor adapted to implement one or more computer programs;

a computer storage medium storing one or more computer programs adapted to be loaded and executed by the processor to:

after receiving a business object to generate real-time business data in security business, storing the real-time business data in a queue data structure by adopting message middleware; acquiring the real-time service data from the message middleware by adopting a stream processing engine, analyzing and processing the real-time service data, and determining the clearing type of the real-time service data; if the clearing type of the real-time service data is the target clearing type, performing data clearing processing on the real-time service data by adopting the stream processing engine to obtain a data clearing result; the data clearing result is used for indicating first asset change information of the business object.

In another aspect, an embodiment of the present application provides a computer storage medium, where one or more computer programs are stored, and the one or more computer programs are adapted to be loaded and executed by the processor to:

In another aspect, the present application further provides a computer program product or a computer program, where the computer program product includes a computer program, and the computer program is adapted to be loaded by a processor and executed to:

In the embodiment of the application, after detecting that real-time service data is generated in the security service, the computer device may receive and store the real-time service data in real time by using the message middleware, and perform data analysis processing on the real-time service data by using the stream processing engine, so that the computer device may perform data clearing processing on the real-time service data. Then, since the real-time service data is obtained in real time, the data clearing performed on the real-time service data by the computer device may also be performed in real time. And because the business object in the securities business can continuously generate real-time business data in the opening process of the securities business, the real-time business data generated by the securities business is processed by the computer equipment in real time, so that the securities business can be cleared in the opening process of the securities business, and the clearing efficiency of the securities business can be effectively improved to a certain extent.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a block diagram of a data processing system according to an embodiment of the present disclosure;

fig. 2 is a schematic flow chart of a data processing method provided in an embodiment of the present application;

fig. 3 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application;

fig. 4 is a schematic structural diagram of a computer device according to an embodiment of the present application.

Detailed Description

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It should be understood that the specific embodiments described in the embodiments of the present application are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the various embodiments of the present application without making any creative effort, shall fall within the protection scope of the present application.

The embodiment of the application provides a data processing method in combination with a stream processing technology and a distributed processing technology, and can be specifically realized by combining a stream processing engine and a distributed publish-subscribe message system (also called message middleware). The streaming processing engine may be understood as a program implemented based on streaming processing technology for streaming processing data. The stream processing technology is an important big data processing means, and is mainly characterized in that: in the process of streaming data, the data to be processed arrives continuously and in real time. The distributed processing technology is a technology for dividing a huge calculation task into a plurality of small tasks after processing. In the embodiment of the application, by combining the distributed technology and the streaming technology, the securities business clearing task in the securities business can be divided into a plurality of data clearing tasks which are carried out in real time, so that the overall efficiency of the securities business clearing can be improved. It is understood that the data processing method provided by the embodiment of the application can be used for performing distributed stream processing on real-time service data generated in the certificate service.

Based on the above description, the following explains a general principle of the data processing method provided in the embodiment of the present application. In a specific implementation, the data processing method provided by the embodiment of the present application may be executed by a computer device. The general principle of the data processing method may then be as follows: after receiving the real-time service data generated by the service object in the security service, the computer equipment stores the real-time service data in a queue data structure by adopting message middleware in real time. Further, the computer device may employ a streaming processing engine to obtain real-time business data from the message middleware and determine a clearing type of the real-time business data. Further, after the computer device determines that the clearing type of the real-time service data is the target clearing type, the streaming processing engine can be continuously adopted to carry out data clearing processing on the real-time service data, so that real-time security clearing of the security service is realized, and the clearing rate in a security service clearing scene is improved.

In order to facilitate a clearer understanding of the principles of the data processing method provided in the embodiments of the present application, the following detailed explanation is made on each term or noun mentioned above. The securities business refers to a business for publicly buying and selling securities in the financial market. In the embodiment of the present application, a security service may be understood as a service related to security trading, such as: securities purchase services, securities sale services, securities commission services, and the like. The security service may be provided by a security company or a security exchange (the following embodiments take the security exchange as an example), and the security service may be opened to the service object so that the service object may perform a corresponding security transaction based on the opened security service. Securities, as referred to herein, are a collective term for various economic equity documents, primarily referring to security products in the securities market, including, but not limited to: stocks (securities used by stock companies to represent shares), bonds (securities due for return), bond certificates (right certificates to purchase a certain number of ordinary shares), fund certificates (certificates to record the number of units of fund held by investors), and the like. A business object may be understood as a user associated with a security business. In practical applications, the business object may be a user who has registered an account with a security company that provides security services. Then, the real-time service data can be understood as: transaction data generated by a business object when conducting a security transaction may be used to indicate transaction details of related security products, such as: the securities trade deals with (or entrusts) which security product, the price and trading number of the security product involved in the securities trade, etc. The clearing type is used to indicate whether the acquired real-time service data needs to be cleared, so that it can be understood that when the clearing type of the real-time service data is a target clearing type, the clearing type can be used to indicate that the real-time service data is service data that needs to be cleared. Data clearing can be understood as: a process for calculating accounts receivable and accounts payable related to related securities trades based on the trade details. It can be understood that, by performing data clearing on the real-time service data, asset change information of the service object corresponding to the real-time service data can be obtained, and the asset change information includes but is not limited to: the position change information (position price, position number, etc.), the earning change information, and the cumulative earning change information.

In a specific embodiment, the data processing method provided in the embodiment of the present application may be applied to the data processing system shown in fig. 1. As seen in fig. 1, the data processing system may include one or more business objects 10, one or more stock exchanges 11, and a streaming processing device 13. Wherein each of the one or more securities exchanges 11 is established with a streaming processing device 13, which streaming processing device 13 may be a computer device. The stock exchange can be understood as a place where the business object performs stock exchange, and the stock exchange can generate corresponding exchange data (i.e. real-time business data) after the business object generates exchange behavior. In the embodiment of the present application, the stock exchange may be a computer device, and may also be an application program, a client, a web page, and the like running on the computer device. The streaming processing device 13 may be configured to acquire real-time service data, and perform data clearing processing on the acquired real-time service data. As can be seen from fig. 1, in practical applications, the streaming processing device 13 may include a real-time service data receiving platform 131, a message middleware 132, and a streaming processing engine 133. It should be noted that the streaming processing engine 133 can listen to the message middleware 132 to obtain relevant data from the message middleware 132 in real time.

Then, based on the above-mentioned related description for the data processing system, the application flow of the data processing method provided by the embodiment of the present application in the data processing system may be as follows: the business object carries out security trading in a security exchange, and after the security exchange detects that security trading behavior exists in the business object, the business exchange acquires trading data corresponding to the business object and sends the trading data to the streaming processing equipment in real time. In particular, the stock exchange may send real-time business data to a real-time business data receiving platform in the streaming processing device. Further, after receiving the real-time transaction data, the real-time service data receiving platform in the streaming processing device may send the real-time service data to the message middleware, so that the message middleware stores the real-time service data according to the data structure of the queue, and thus the real-time service data may be processed as data in a data stream. When the stream processing engine monitors that newly arrived real-time service data exists in the message middleware, the stream processing engine can acquire the real-time service data from the message middleware and determine the clearing type of the real-time service data, and then can perform data clearing processing on the real-time service data after determining that the clearing type of the real-time service data is a target clearing type, namely: and after determining that the real-time service data is the service data which needs to be subjected to data clearing, performing data clearing processing on the real-time service data.

It should be noted that the above mentioned computer device (including the streaming processing device in the data processing system) may be a terminal device, and may also be a server. The terminal device may include, but is not limited to: smart phones, tablet computers, notebook computers, desktop computers, vehicle-mounted terminals, smart televisions, and the like. Various clients (APPs) can run in the terminal device, such as a data processing client, a social client, a browser client, an information flow client, an education client, and the like. Further, the server may include, but is not limited to: the cloud server may be a server cluster or a distributed system including an independent physical server and a plurality of physical servers, and the cloud server may provide basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, a middleware service, a domain name service, a security service, a CDN (Content Delivery Network), a big data and artificial intelligence platform, and the like.

An embodiment of the present application provides a data processing method, please refer to fig. 2, and fig. 2 is a schematic flowchart of another data processing method provided in the embodiment of the present application, which can also be executed by the aforementioned computer device. And as shown in fig. 2, the data processing method may include steps S201 to S204:

s201, after receiving real-time service data generated by a service object in security service, storing the real-time service data in a queue data structure by using message middleware.

In the actual stock trading scenario, the real-time business data may be entrusted data or transaction data. The entrusting data is generated by the stock exchange based on entrusting behaviors of the business objects in the stock exchange, and the trading data is generated by the stock exchange based on trading behaviors of the business objects in the stock exchange. The bargaining behavior can be understood as: trade activity has been completed for trading securities, such trade activity involving the actual income or expenditure of funds or transfer of ownership of the securities; the delegation behavior can then be understood as: the act of willingfully but not actually trading the securities, there is no transfer of security ownership. It should be noted that, in practical applications, a delegation behavior may correspond to a trading behavior, that is, a security exchange may actually trade a security product that is willing to trade. In addition, the stock business can be one or more, and the business object can also be one or more. It will be appreciated that the computer device may then store real-time business data in real-time using messaging middleware after detecting that any business object produces the real-time business data in any security business. In the embodiment of the present application, the principle of data clearing performed by computer equipment on real-time business data generated by any business object in any security business is consistent, and therefore for convenience of description, related steps involved in the data processing method provided by the embodiment of the present application are described in detail below with one security business and one business object without special description. The message middleware can be understood as a container for storing real-time service data, and specifically, the message middleware can include any one of the following: ActiveMQ, RabbitMQ, RocketMQ, Kafka, etc. Illustratively, for better distributed streaming processing of real-time traffic data, in the embodiment of the present application, the computer device mainly employs Kafka as message middleware.

Since the security service may be provided by a security exchange and the computer device may be communicatively connected to one or more security exchanges, it may also be understood that: the computer device may be associated with one or more stock exchanges. Then, in this case, the target exchange may be any one of the one or more stock exchanges, and the real-time business data may be generated by the business object in the target exchange. In a specific application, the manner for acquiring real-time service data by the computer device may be as follows: the target exchange may send the real-time business data to the computer device after generating the real-time business data, so that the computer device may store the real-time business data using message middleware. It will be appreciated that during the opening of a security service, one or more security exchanges may be simultaneously generating and transmitting real-time service data to a computer device. In this case, the computer device stores the real-time service data by using the message middleware, so that the computer device can be prevented from allocating a large amount of computing resources at one time to perform data clearing on the received real-time service data. And after receiving the real-time data, the computer equipment stores the real-time data in a queue data structure by adopting message middleware, and can integrate the generated real-time service data into a data stream, so that the computer equipment can perform distributed data clearing processing on the real-time service data in a stream processing mode. Therefore, the method can effectively improve the stability of the performance of the computer equipment, and further improve the security clearing efficiency of the security business.

S202, acquiring real-time service data from the message middleware by adopting a stream processing engine.

In particular embodiments, a computer device may employ a streaming processing engine to obtain real-time business data from message middleware. The streaming processing engine may include any one or more of: apache Storm, Apache Samza, Apache Spark, and Apache Flink, among others. The Apache Storm and Apache Samza are only stream processing engines, that is, a computer device can only perform stream processing on real-time service data by using any one of the two stream processing engines. By stream processing is understood: after receiving the real-time service data, the computer device quickly performs data processing (such as data analysis processing and data clearing processing) on the real-time service data in a short time. Apache Spark and Apache Flink are hybrid processing engines, that is, a computer device can implement stream processing and batch processing on real-time service data by using either one of the two stream processing engines. Batch processing can be understood as: and the computer equipment acquires a plurality of real-time service data and then performs a data processing process. In a security service clearing scene, the batch processing of the real-time service data by the computer device may be performed at the end of each trading day, or may be performed after the computer device acquires a preset number of real-time service data in the message middleware.

The computer equipment generally needs to support the batch flow and the batch flow simultaneously when the securities business clearing is carried out, so that the stability of the computer equipment when the securities business clearing is carried out is ensured to the maximum extent. Therefore, in the embodiment of the present application, the computer device may preferentially adopt the hybrid processing engine, that is, in the embodiment of the present application, the Apache Spark or the Apache flush may preferentially adopt. In the process of providing the data processing method according to the embodiment of the present application, in order to select an optimal stream processing engine, relevant personnel perform a POC test (namely Proof of Concept verification) on an Apache Spark or an Apache Flink, and the test result indicates that: apache Flink is superior to Apache Spark in comprehensive evaluation of functions, performance, development, community liveness and the like, so that the Apache Flink can better meet conditions of high availability, low delay, distributed memory calculation and the like, and the processing performance of computer equipment is further improved. In the embodiment of the present application, when the hybrid processing engine is used by the computer device, Apache Flink may be preferentially used as a streaming processing engine when the computer device is used for clearing securities business.

S203, performing data analysis processing on the real-time service data, and determining the clearing type of the real-time service data.

In a specific embodiment, when there are a plurality of securities businesses, the data clearing of the real-time business data by the computer device may involve the following situations: in part of the security business (such as the security business A), the computer equipment needs to carry out data clearing processing on the entrusted data; in another part of the security service (e.g., security service B), the computer device needs to perform data clearing processing on the transaction data. The following is described with reference to specific examples: assuming that the real-time service data acquired by the computer equipment is entrusted data generated by a service object in the security service A, the computer equipment can clear the acquired entrusted data; correspondingly, if the real-time service data acquired by the computer device is transaction data generated by the service object in the security service a, the computer device may not perform data clearing on the acquired commission data. Therefore, it can be understood that, in order to improve the accuracy of data clearing, before the computer device performs data clearing processing on the real-time service data, the computer device may determine a clearing type corresponding to the real-time service data, where the clearing type is used to indicate whether the computer device needs to perform data clearing on the real-time service data. Illustratively, in the embodiments of the present application, the clearing type may include any one or two of the following: a first clearing type for indicating that the computer device needs to perform data clearing on the real-time service data, and a second clearing type for indicating that the computer device does not need to perform data clearing on the real-time service data.

Based on the above description, that is, the data analysis process is mainly used for determining the clearing type of the real-time service data, so that the computer device can perform the data clearing process on the real-time service data in time after determining that the real-time service data needs to be subjected to the data clearing process. When the computer device determines the clearing type of the real-time service data, the data type of the real-time service data can be determined first, and then the computer device can further determine the clearing type of the real-time service data based on the determined data type. It should be noted that, the specific manner in which the computer device determines the clearing type of the real-time service data based on the data type of the real-time service data will be described in detail in the following embodiments, which is not described herein again.

The data type of the real-time service data in the embodiment of the present application is defined as follows. Since the real-time service data may be entrusted data or transaction data, exemplary data types of the real-time service data may include, but are not limited to: commission type, deal type. Wherein, the delegation type is used for indicating: the real-time business data is the business data generated after a business object entrusts a stock exchange to carry out stock exchange (namely entrusting data); the type of deal is used to indicate: real-time business data is business data (i.e., deal data) generated after a business object deals with a security in a security exchange. In addition, it should be noted that in other embodiments, when the computer device performs data analysis processing on the acquired real-time service data, the computer device may further identify a data source of the real-time service data, so that the security of the real-time service data can be ensured to a certain extent, and the security of the computer device and the stability of the performance of the computer device are further ensured.

In particular implementations, a computer device may employ a streaming processing engine to perform data analysis processing on real-time business data. Specifically, the computer device may first acquire the data type of the real-time service data by using the streaming processing engine, and acquire a target data type corresponding to the service data in the security service as the target clearing type by using the streaming processing engine. That is, the computer device may first determine the data type (e.g., transaction type or commission type) of the real-time service data, and determine which data type of service data needs to be subjected to data clearing processing (e.g., commission type or commission type) in the security service corresponding to the real-time service data. If the computer equipment determines that the entrusted data needs to be subjected to data clearing processing in the security service, the target data type is the entrusted type. Correspondingly, if the computer equipment determines that the data clearing processing needs to be carried out on the transaction data in the securities business, the target data type is the transaction type. Further, it can be understood that, if the data type of the real-time service data is the same as the target data type, the computer device may determine that the clearing type of the real-time service data is the target clearing type, so that the data clearing processing may be further performed on the real-time service data.

And S204, if the clearing type of the real-time service data is the target clearing type, carrying out data clearing processing on the real-time service data to obtain a data clearing result.

The data clearing result is used for indicating first asset change information of the business object, and it can be understood that the first asset change information is obtained by the computer device after data clearing is performed on related real-time business data. Further, the data clearing result in the embodiment of the present application may include, but is not limited to, any one or more of the following: accounts payable (such as handling fee, transaction commission, securities transaction fund, fee of the banker, etc.), and accounts payable (such as the amount of the return of today, the amount of the single return, the total amount of the return, etc.). Based on the foregoing, the clearing type may include a first clearing type and a second clearing type, where the first clearing type is used for indicating that the real-time service data needs to be subjected to data clearing, and the second clearing type is used for indicating that the real-time service data does not need to be subjected to data clearing. In this case, then, it is understood that the target clearing type herein may be the first clearing type.

In practical applications, when the computer device performs data clearing processing on the real-time business data, the data clearing processing may be performed based on corresponding data clearing rules, and the data clearing rules may be established by a stock exchange. In the embodiment of the application, the data clearing rule refers to corresponding data clearing logic which is pre-established in the computer device by the computer device according to data clearing rule documents of various stock exchanges associated with the computer device, and the essence of the data clearing rule in the embodiment of the application is code data. Therefore, it is understood that after the real-time service data is acquired, the computer device can automatically perform data clearing processing according to the data clearing logic of the stock exchange corresponding to the real-time service data, and the clearing rate of stock service clearing can be improved to a certain extent.

Furthermore, in an actual stock clearing scenario, when a plurality of stock trades are all associated with a computer device, the data formats of real-time business data generated by different stock trades may not be the same. For example, a may indicate that a is 10 in the real-time business data generated by the stock exchange a, and B is 10 in the real-time business data generated by the stock exchange B. The security net value can be understood as a data element in the real-time business data, and both a and b can be understood as an element identifier (hereinafter referred to as a first element identifier) corresponding to the data element of the security net value. It can be seen that, for data elements with the same semantic information (that is, data elements with the same expression meaning), different element identifiers may be used to represent real-time business data generated by different securities trades. In this case, in order to enable the computer device to have the capability of performing data clearing processing on the real-time service data generated by each stock exchange under the condition of ensuring a high processing rate, the computer device may first perform data format unification on the acquired real-time service data (that is, convert the acquired real-time service data into the same data format), so that the computer device only needs to have the capability of performing data clearing on the real-time service data in one data format in the process of performing data clearing on the real-time service data, thereby avoiding that the computer device simultaneously includes multiple data processing logics, and further reducing the complexity of the computer device, and further improving the stability of the performance of the computer device.

Based on the above description, it can be understood that, in a specific embodiment, the manner in which the computer device performs data clearing processing on the real-time service data may be as follows: and the computer equipment acquires the data clearing rule of the real-time service data in the target exchange and converts the real-time service data into the target service data in the target data format. Further, the computer device adopts a stream processing engine to perform data clearing processing on the target service data according to the obtained data clearing rule so as to obtain a data clearing result. When the computer device converts the real-time service data into the target service data in the target data format, semantic information indicated by a first element identifier of any data element in the real-time service data may be obtained first, and then further, the computer device may search for a second element identifier matching the semantic information from a target data dictionary, thereby obtaining the target element identifier. Finally, the computer device may generate target business data based on the target element identification and the corresponding data element.

Wherein the first element identifier exists in the data dictionary of the target exchange, the first element identifier is an identifier used by the target exchange to indicate a certain data element (such as data element a), and semantic information of the first element identifier can be used to indicate the meaning of the data element a. For example, it is assumed that a first element identifier a and a data element 10 corresponding to the first element identifier exist in the real-time service data. Then, if the semantic information of the first element identifier a obtained by the computer device is the price of a share of a security unit (i.e. the net value of the security), it can be understood that the meaning of 10 in the real-time business data is: the security net value is 10. And the second element identification exists in the target data dictionary, the second element identification is an identification which is used for indicating a data element after the computer equipment performs data format conversion on the real-time service data, and the target element identification is the second element identification matched with the semantic information of the first element identification.

Since the computer device may be associated with one or more securities exchanges, it will be readily appreciated that the target data dictionary may be generated based on the data dictionary of one or more securities exchanges (e.g., target exchanges) or may be generated based on the data dictionary of each security exchange associated with the computer device. The manner in which the computer device generates the target data dictionary is described in detail below, taking as an example that the computer device generates the target data dictionary based on the data dictionary of the target exchange. The manner in which the computer device generates the target data dictionary may be as follows: the computer equipment acquires a data dictionary of a target exchange, wherein the data dictionary comprises at least one first element identifier; then, the computer device may obtain semantic information indicated by each first element identifier, and generate a second element identifier based on the obtained semantic information. Further, the computer device may generate a target data dictionary based on a corresponding second element identification of each of the at least one first element identification. In this embodiment, each first element identifier corresponds to one second element identifier, and data elements corresponding to the two element identifiers represent the same meaning. It is understood that when the computer device generates the target data dictionary from a plurality of data dictionaries, the second element identifications corresponding to the element identifications for representing the same semantic information in each data dictionary are the same. For example, assuming that a denotes the net value of a security in a security exchange a, B denotes the net value of a security in B, and C denotes the net value of a security in C, the second elements generated by the computer device based on the semantic information of a, B, and C are identified as the same, or it can be understood that: and the second element marks corresponding to the first element marks a, b and c are the same.

In one embodiment, after the computer device completes data clearing processing on the real-time service data and obtains the first asset change information of the corresponding service object, in order to improve the accuracy of a data clearing result obtained by the computer device, a clearing data file generated by the target transaction can be further obtained, wherein the clearing data file comprises at least one piece of clearing data. The clearing data can be understood as: and adding data elements such as expense fields (tax stamp, user fee, certificate and administration fee), clearing date, clearing amount, payment date and the like on the basis of the real-time business data of the target exchange to obtain the data. The clearing data may be used to indicate second asset change information for the corresponding business object. It should be noted that the clearing data file includes at least one piece of clearing data including: and the target exchange carries out clearing processing on the real-time service clearing data to obtain target clearing data. Then, further, the computer device may perform a check based on the first asset transition information and the second asset transition information to further improve the accuracy of the data clearing result calculated by the computer device. Specifically, if the second asset change information is inconsistent with the first asset change information, the computer device may perform the step of performing the data clearing process on the real-time service data again to obtain the data clearing result.

In the embodiment of the application, after detecting that real-time service data is generated in security services, the computer device may acquire and store the real-time service data in real time by using the message middleware, and perform distributed data processing (such as data analysis processing and data clearing processing) on the real-time service data by using the stream processing engine, so that the computer device may further complete data clearing on the real-time service data quickly. In the embodiment of the present application, Apache Flink may be used as a stream processing engine to perform stream processing or batch processing on real-time service data, and the Apache Flink may better satisfy conditions of high availability, low latency, distributed memory calculation, and the like, so that a computer device using the data processing method provided in the embodiment of the present application will have a higher data processing performance. In addition, during the opening process of the security service, the service objects in the security service can continuously generate real-time service data, and a security service usually comprises a large number of service objects, so that the security service can generate a large amount of real-time service data during the opening process. In this case, the computer device can store the real-time service data in a queue form into the message middleware, and then the computer device can process the real-time service data generated by the security service in batch or in real time by adopting the stream processing engine according to the processing requirement or the actual processing capacity, so that the security service can be cleared in the opening process of the security service without waiting for the end of the security service (or the end of a trading day), and the efficiency of clearing the security service is effectively improved.

Based on the related description of the data processing method, the application also discloses a data processing device. The data processing means may be a computer program (comprising program code) running on a computer device as mentioned above. The data processing apparatus may execute the data processing method shown in fig. 2, please refer to fig. 3, the data processing apparatus may at least include: a storage unit 301, a data analysis unit 302, and a data clearing unit 303.

The storage unit 301 is configured to store, by using a message middleware, real-time service data generated by a service object in a security service in a data structure of a queue;

a data analysis unit 302, configured to acquire the real-time service data from the message middleware by using a stream processing engine, perform data analysis processing on the real-time service data, and determine a clearing type of the real-time service data;

the data clearing unit 303 is configured to perform data clearing processing on the real-time service data by using the streaming processing engine to obtain a data clearing result if the clearing type of the real-time service data is the target clearing type; the data clearing result is used for indicating first asset change information of the business object.

In an embodiment, the data analysis unit 302 may be configured to perform:

acquiring the data type of the real-time service data by adopting the streaming processing engine, and acquiring a target data type corresponding to the service data in the security service when the service data is a target clearing type;

and if the data type of the real-time service data is the same as the target data type, determining that the clearing type of the real-time service data is the target clearing type.

In yet another embodiment, the real-time business data is generated in a target exchange associated with a security business; the data clearing unit 303 may be configured to perform:

acquiring a data clearing rule of the real-time service data in the target exchange, and converting the real-time service data into target service data in a target data format;

and performing data clearing processing on the target service data according to the data clearing rule to obtain a data clearing result.

In yet another embodiment, the real-time business data includes at least one data element and a first element identifier for each data element, the first element identifier existing in a data dictionary of the target exchange; the data clearing unit 303 may be further configured to perform:

obtaining semantic information indicated by a first element identifier of any data element in the real-time service data;

searching a second element identifier matched with the semantic information from a target data dictionary to obtain a target element identifier, wherein the target data dictionary is generated based on the data dictionary of the target exchange;

and generating the target business data based on the target element identification and any data element.

In yet another embodiment, the data clearing unit 303 may be further configured to perform:

acquiring a data dictionary of the target exchange, wherein the data dictionary comprises at least one first element identifier;

obtaining semantic information indicated by each first element identifier, and generating a second element identifier based on the obtained semantic information; second element identifications generated by first element identifications with the same semantic information are the same;

and generating the target data dictionary based on the second element identification corresponding to each first element identification in the at least one first element identification.

In yet another embodiment, the real-time business data is generated in a target exchange associated with a security business; the data clearing unit 303 may be further configured to perform:

acquiring a clearing data file generated by the target trading place by adopting the streaming processing engine, wherein the clearing data file comprises second asset change information of the business object, and the second asset change information is obtained by performing data clearing on the real-time business data by the target trading place;

if the second asset change information is inconsistent with the first asset change information, re-executing: and performing data clearing processing on the real-time service data by adopting the streaming processing engine to obtain a data clearing result.

According to an embodiment of the application, the steps involved in the method shown in fig. 2 may be performed by various units in the data processing apparatus shown in fig. 3. For example, step S201 shown in fig. 2 may be performed by the storage unit 301 in the data processing apparatus shown in fig. 3, both step S202 and step S203 may be performed by the data analysis unit 302 shown in fig. 3, and step S204 may be performed by the data clearing unit 303 in the data processing apparatus shown in fig. 3.

According to another embodiment of the present application, the units in the data processing apparatus shown in fig. 3 are divided based on logical functions, and the units may be respectively or entirely combined into one or several other units to form the unit, or some unit(s) therein may be further split into multiple units smaller in function to form the unit(s), which may achieve the same operation without affecting the achievement of the technical effects of the embodiments of the present application. In other embodiments of the present application, the data processing apparatus may also include other units, and in practical applications, these functions may also be implemented by being assisted by other units, and may be implemented by cooperation of multiple units.

According to another embodiment of the present application, the data processing apparatus shown in fig. 3 may be constructed by running a computer program (including program codes) capable of executing the steps involved in the method shown in fig. 2 on a general-purpose computing device such as a computer device including a processing element such as a Central Processing Unit (CPU), a random access storage medium (RAM), a read-only storage medium (ROM), and a storage element, and implementing the data processing method of the embodiment of the present application. The computer program may be embodied on, for example, a computer storage medium, and loaded into and executed by the computing device described above via the computer storage medium.

In the embodiment of the application, the data processing device can acquire real-time business data in real time after detecting that the real-time business data is generated in the security business, and perform data analysis processing on the real-time business data, so that the data processing device can further perform data clearing processing on the real-time business data. Then, since the real-time service data is acquired in real time, the data processing apparatus performs data clearing on the real-time service data in real time. Because the business object in the securities business can continuously generate the real-time business data in the opening process of the securities business, the data processing device can process the real-time business data generated by the securities business in real time, and can realize the clearing of the securities business in the opening process of the securities business, thereby effectively improving the clearing efficiency of the securities business to a certain extent.

Based on the above description of the method embodiment and the apparatus embodiment, an embodiment of the present application further provides a computer device, please refer to fig. 4. The computer device comprises at least a processor 401 and a computer storage medium 402, and an input interface 403. Wherein the processor 401, the computer storage medium 402 and the input interface 403 of the computer device may be connected by a bus or other means.

The above-mentioned computer storage medium 402 is a memory device in a computer device for storing a computer program and data. It is understood that the computer storage medium 402 herein may comprise a built-in storage medium in the computer device, and may of course also comprise an extended storage medium supported by the computer device. The computer storage media 402 provides storage space that stores the operating system of the computer device. Also stored in this memory space are one or more computer programs, which may be one or more program codes, adapted to be loaded and executed by processor 401. The computer storage medium may be a high-speed RAM memory, or may be a non-volatile memory (non-volatile memory), such as at least one disk memory; and optionally at least one storage medium located remotely from the processor. The processor 401 (or CPU) is a computing core and a control core of the computer device, and is adapted to implement one or more computer programs, and in particular, is adapted to load and execute the one or more computer programs so as to implement corresponding method flows or corresponding functions. The input interface 403 may be used to receive data sent by other devices, such as: and receiving the real-time service data sent by the target exchange.

In one embodiment, one or more computer programs stored in the computer storage medium 402 may be loaded and executed by the processor 401 to implement the corresponding method steps described above in connection with the method embodiment shown in FIG. 2; in particular implementations, one or more computer programs in the computer storage medium 402 may be loaded and executed by the processor 401 to perform the steps of:

In an embodiment, the processor 401 may specifically load and execute:

In yet another embodiment, the real-time business data is generated in a target exchange associated with a security business; the processor 401 may specifically load and execute:

In yet another embodiment, the real-time business data includes at least one data element and a first element identifier for each data element, the first element identifier existing in a data dictionary of the target exchange; the processor 401 may specifically load and execute:

In another embodiment, the processor 401 may specifically load and execute:

if the second asset change information is inconsistent with the first asset change information, re-executing: and performing data clearing processing on the real-time service data to obtain a data clearing result.

In the embodiment of the application, after detecting that real-time service data is generated in the security service, the computer device may acquire the real-time service data in real time, and perform data analysis processing on the real-time service data, so as to further enable the computer device to perform data clearing processing on the real-time service data. Then, since the real-time service data is obtained in real time, the data clearing of the real-time service data by the computer device is also performed in real time. Because the business object in the securities business can continuously generate the real-time business data in the opening process of the securities business, the real-time business data generated by the securities business is processed by the computer equipment in real time, the securities business can be cleared in the opening process of the securities business, and the clearing efficiency of the securities business can be effectively improved to a certain extent.

The present application further provides a computer storage medium, where one or more computer programs corresponding to the data processing method are stored in the computer storage medium, and when one or more processors load and execute the one or more computer programs, descriptions of the data processing method in the embodiments can be implemented, which are not described herein again. The description of the beneficial effects of the same method is not repeated herein. It will be appreciated that the computer program may be deployed to be executed on one or more devices that are capable of communicating with each other.

It should be noted that according to an aspect of the present application, a computer program product or a computer program is also provided, the computer program product including a computer program, and the computer program is stored in a computer storage medium. The processor in the terminal device reads the computer program from the computer storage medium and then executes the computer program, thereby enabling the terminal device to perform the methods provided in the various alternatives in the aspect of the data processing method embodiment shown in fig. 2 described above.

It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments may be implemented by a computer program, which can be stored in a computer storage medium and can include the processes of the above embodiments of the data processing method when executed. The computer storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.

While the invention has been described with reference to a particular embodiment, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A data processing method, comprising:

2. The method according to claim 1, wherein the performing data analysis processing on the real-time service data and determining the clearing type of the real-time service data comprises:

3. The method of claim 1, wherein the real-time business data is generated in a target exchange associated with a security business; the data clearing processing is carried out on the real-time service data, and the mode of obtaining the data clearing result comprises the following steps:

4. The method of claim 3, wherein the real-time business data comprises at least one data element and a first element identifier for each data element, the first element identifier being present in the data dictionary of the target exchange; the method for converting the real-time service data into the target service data in the target data format comprises the following steps:

5. The method of claim 4, wherein generating the target data dictionary based on the data dictionary of the target exchange comprises:

6. The method of claim 1, wherein the real-time business data is generated in a target exchange associated with a security business; the method further comprises the following steps:

7. A data processing apparatus, comprising:

8. A computer device, comprising:

a processor adapted to implement one or more computer programs;

computer storage medium storing one or more computer programs adapted to be loaded by said processor and to perform the data processing method according to any of claims 1-6.

9. A computer storage medium storing one or more computer programs adapted to be loaded by a processor and to perform the data processing method of any of claims 1-6.

10. A computer program product or computer program, characterized in that the computer product comprises a computer program adapted to be loaded by a processor and to perform the data processing method according to any of claims 1-6.