CN107123047B - Data acquisition system based on bond transaction and data acquisition method thereof - Google Patents
Data acquisition system based on bond transaction and data acquisition method thereof Download PDFInfo
- Publication number
- CN107123047B CN107123047B CN201710244125.2A CN201710244125A CN107123047B CN 107123047 B CN107123047 B CN 107123047B CN 201710244125 A CN201710244125 A CN 201710244125A CN 107123047 B CN107123047 B CN 107123047B
- Authority
- CN
- China
- Prior art keywords
- data
- module
- information
- transaction
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Abstract
Disclosed is a data acquisition system based on bond transaction and a data acquisition method thereof, the data acquisition system comprises a client (1), a server (2), a management terminal (3) and a data acquisition server (23), the data acquisition server (23) comprises a data extraction module (24), a data processing module (25), a distributed data storage management module (26) and a query module (27), the data processing module (25) comprises a data association module (29) for associating by using parameter characteristics, a classification module (30) based on time sequence classification and a clustering module (31) for removing data duplicate, the distributed data storage management module (26) comprises a time sequence data memory (32) for storing time sequence data and a data memory (33) for storing non-time sequence data, the query module (27) performs a query in the distributed data storage management module (26) based on a query command.
Description
Technical Field
The invention relates to the field of data acquisition, in particular to a data acquisition system based on bond transaction and a data acquisition method thereof.
Background
The bond trading system firstly completes the processing of the whole trading process, including functions of quotation processing, quotation display, click request processing, trading check and the like, and provides trading information services of bond position taking, available funds, market quotation and the like. The whole course of the market transaction process is recorded, so that market control functions such as inquiry, monitoring and emergency are provided for transaction managers, and the stable operation of a transaction system is guaranteed. With the adoption of the method, as the bond transaction faces huge and large-scale data impact such as millions of concurrent user access, thousands of concurrent transaction processing per second and the like, data acquisition is needed to be optimized, the technical problems of large data volume, fast data change and the like in the current business system are effectively solved, the problems of data acquisition repetition and low quality are solved, and the reliability and the expansibility of data acquisition are improved.
A. acquiring historical settlement data of a user terminal;
B. collecting current data of a user terminal;
C. carrying out merging calculation by utilizing the historical settlement data and the current data of the user terminal, receiving a settlement request of the user terminal according to the calculation result, and obtaining new historical settlement data according to the settlement request;
D. and synchronizing the new historical settlement data to the first management subsystem and the second management subsystem. The patent can realize the instant data acquisition, settlement and system data synchronization in the data processing process, can fully utilize network resources to complete the system required data processing operation, and improves the utilization efficiency of the network resources and the data processing efficiency based on the network. However, the patent is not suitable for bond transaction processing, and does not have the processing for completing the whole transaction process, including quotation processing, quotation display, click request processing, transaction inspection, bond holding, available funds, market quotation and the like, and the whole process of the market transaction is recorded, so that the stable operation of a transaction system is ensured, and the problems that the data volume in the system is large, the data change is fast, the data acquisition efficiency, the data acquisition quality and the data acquisition reliability cannot be improved under the condition of keeping the smooth operation of bond transaction processing can not be avoided, the reliability of the whole operation of the system is improved, and the data acquisition repetition and the like cannot be avoided.
A data acquisition apparatus disclosed in patent document 2 includes: the downloading unit is used for acquiring basic data and/or service data; the updating unit is used for updating the data acquired by the downloading unit; and the uploading unit is used for uploading the data updated by the updating unit to the central server. This patent has avoided repeated collection, has realized the data sharing of collection end. However, the patent is not suitable for bond transaction processing, and does not have the processing for completing the whole transaction process, including quotation processing, quotation display, click request processing, transaction inspection, bond holding, available funds, market quotation and the like, and the whole process of the market transaction is recorded, so that the stable operation of a transaction system is ensured, and the problems that the data volume in the system is large, the data change is fast, the data acquisition efficiency, the data acquisition quality and the data acquisition reliability cannot be improved under the condition of keeping the smooth operation of bond transaction processing are further avoided, the reliability of the whole operation of the system is improved, and the problem of data acquisition repetition under the large data volume and high-frequency change of bond transaction cannot be avoided.
A method of data acquisition disclosed in patent document 3 includes: acquiring a data structure table of each service system to be associated; establishing a synchronous mapping relation for the synonymous fields in the data structure table; and updating the field contents stored in each service system according to the recorded field contents and the synchronous mapping relation of the fields. The patent realizes single acquisition of data and sharing of multiple service platforms, and improves data entry efficiency. However, the patent is not suitable for bond transaction processing, and does not have the processing for completing the whole transaction process, including quotation processing, quotation display, click request processing, transaction inspection, bond holding, available funds, market quotation and the like, and the whole process of the market transaction is recorded, so that the stable operation of a transaction system is ensured, and the problems that the data volume in the system is large, the data change is fast, the data acquisition efficiency, the data acquisition quality and the data acquisition reliability cannot be improved under the condition of keeping the smooth operation of bond transaction processing are further avoided, the reliability of the whole operation of the system is improved, and the problem of data acquisition repetition under the large data volume and high-frequency change of bond transaction cannot be avoided.
Documents of the prior art
Patent document
Patent document 1: chinese patent publication No. CN1741059A
Patent document 2: chinese patent publication No. CN101038597A
Patent document 3: chinese patent publication No. CN104657430A
Disclosure of Invention
Problems to be solved by the invention
In view of the fact that the method is suitable for bond transaction processing, the method has the advantages that on the basis of completing the processing of the whole transaction process including quotation processing, quotation display, click request processing, transaction inspection, bond position providing, available funds, market quotation and the like, the whole transaction process of the market is recorded, the stable operation of a transaction system is guaranteed, under the conditions that the data volume in the system is large and the data change is fast, the smooth operation of the bond transaction processing is kept, the data acquisition efficiency, the data acquisition quality and the data acquisition reliability are improved, the reliability of the whole operation of the system is improved, and the problem of data acquisition repetition is avoided under the large data volume and high-frequency change of the bond transaction.
Means for solving the problems
The present inventors have made intensive studies to achieve the above objects, and more particularly, to a data collecting system and a data collecting method thereof based on bond transactions, wherein the data collecting system based on bond transactions includes a client, a server, a manager, and a data collecting server.
The client comprises a quotation information module for providing quotation information, a quotation query module for querying the quotation information, a click-to-deal processing module for click-to-deal, a transaction query module for querying balance and deal intention and a login module for logging in and logging out and modifying a password of a client.
The server side comprises a quotation processing module used for quotation information receiving, sorting, merging and pushing, a transaction processing module used for quotation interval monitoring, temporary commitment instruction generation and commitment instruction transmission, a deal processing module used for receiving deal information, deal result processing and market data generation, a book-keeping module used for book-keeping management, a market-keeping module used for market-keeping calculation, a time-keeping module used for providing time-keeping service, a day-time processing module used for processing opening signals and bond information and a day-end processing module used for synchronizing the balance of bond of investors, the balance of fund and incremental information of investors.
The management terminal comprises an information query module for providing information, a transaction monitoring module for monitoring quotation information, a commitment instruction and/or transaction application, an information maintenance module for maintaining bond information and quotation information, a day-end monitoring module for monitoring investor information, bond balance information and fund balance information day-end, an emergency information maintenance module for processing investor information import, emergency transaction intention application input and emergency deal result input and a system maintenance module for setting system parameters.
The data acquisition server comprises a data extraction module, a data processing module, a distributed data storage management module and a query module, wherein, the data extraction module comprises a front-end processor and an ETL conversion module which are used for collecting data generated by the client, the server and the management end in real time, the data processing module comprises a data association module for association by using parameter characteristics, a classification module based on time sequence classification and a clustering module for de-duplication of data, the distributed data storage management module comprises a time sequence data storage for storing time sequence data and a data storage for storing non-time sequence data, the distributed data storage management module stores the time sequence data in the time sequence data storage and stores the non-time sequence data in the data storage, the query module queries the distributed data storage management module based on a query command.
In the data acquisition system based on bond transaction, the parameter characteristics comprise data generation time, a data generation module, a data generation IP address, a data format and/or a data type, wherein the data generation module comprises one or more of a market quotation query module, a click deal processing module, a transaction query module, a quotation processing module and a transaction processing module.
In the data acquisition system based on bond transaction, the front-end processor comprises a transaction message format processing module, the transaction message comprises a notification type and a request/response type, and the ETL conversion module comprises a data extraction unit, a data cleaning unit, a null value processing unit, a data format processing unit, a data splitting unit and a data replacement unit.
In the data acquisition system based on bond transaction, the ETL conversion module is a Datastage ETL, and the clustering module is a Kmeans clustering device.
In the data acquisition system based on bond transaction, the data association module performs association by using an Apriori algorithm based on parameter characteristics.
In the data acquisition system based on bond transaction, the data classification module classifies the data into time sequence data and non-time sequence data based on time sequence.
According to another aspect of the present invention, a data collecting method using the data collecting system based on bond transaction includes:
in the first step: the front-end processor collects data generated by the client, the server and the management terminal in real time.
In the second step: and the ETL conversion module extracts, cleans and converts the data.
In the third step: the data association module associates the data with the parameter characteristics.
In the fourth step: the classification module separates the data into time-series data and non-time-series data based on parameter characteristics.
In the fifth step: the clustering module performs deduplication on the data.
In the sixth step: the distributed data storage management module stores time series data in the time series data storage and stores non-time series data in the data storage.
In the seventh step: and the query module queries in the distributed data storage management module based on the query command.
In the data acquisition method of the present invention, the third step is: and the data association module associates the data based on the parameter characteristics by using a multi-valued attribute MAQA algorithm.
In the data acquisition method of the present invention, in the fourth step: the classification module classifies the data into time-series data and non-time-series data based on parameter feature pairs by using a GARCH algorithm.
In the data acquisition method of the present invention, the fifth step is: and the clustering module adopts a CLARA algorithm to perform data deduplication.
ADVANTAGEOUS EFFECTS OF INVENTION
In the data acquisition system based on bond transaction, the client, the server and the management end complete the processing of the whole transaction process, including functions of quotation processing, quotation display, click request processing, transaction check and the like, and simultaneously provide transaction information services such as bond position taking, available funds, market quotation and the like.
The above description is only an overview of the technical solutions of the present invention, and in order to make the technical means of the present invention more clearly apparent, and to make the implementation of the content of the description possible for those skilled in the art, and to make the above and other objects, features and advantages of the present invention more obvious, the following description is given by way of example of the specific embodiments of the present invention.
Drawings
Fig. 1 shows a schematic structural diagram of a data acquisition system based on bond transaction according to the present invention.
Fig. 2 shows a schematic step diagram of a data collection method using a data collection system based on bond transaction according to the present invention.
Description of the symbols
1 client
2 service end
3 management terminal
4 quotation information module
5 market information inquiry module
6 click friendship processing module
7 transaction query module
8 login module
9 quotation processing module
10 transaction processing module
11 transaction processing module
12 note module
13 market information module
14 time synchronization module
15 daytime processing module
16-day final processing module
17 information inquiry module
18 transaction monitoring module
19 information maintenance module
20-day end monitoring module
21 emergent information maintenance module
22 system maintenance module
23 data acquisition server
24 data extraction module
25 data processing module
26 distributed data storage management module
27 query module
28 front-end processor
29 data association module
30 classification module
31 clustering module
32 time series data memory
33 data memory
34 ETL conversion module
Detailed Description
Specific embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While specific embodiments of the invention are shown in the drawings, it should be understood that the invention may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
It should be noted that certain terms are used throughout the description and claims to refer to particular components. As one skilled in the art will appreciate, various names may be used to refer to a component. This specification and claims do not intend to distinguish between components that differ in name but not function. In the following description and in the claims, the terms "include" and "comprise" are used in an open-ended fashion, and thus should be interpreted to mean "include, but not limited to. The description which follows is a preferred embodiment of the invention, but is made for the purpose of illustrating the general principles of the invention and not for the purpose of limiting the scope of the invention. The scope of the present invention is defined by the appended claims.
As will be appreciated by one skilled in the art, the present invention may be embodied as a system, method or computer program product. Accordingly, the present disclosure may be embodied in the form of: may be embodied entirely in hardware, entirely in software (including firmware, resident software, micro-code, etc.) or in a combination of hardware and software, and may be referred to herein generally as a "circuit," module "or" system. Furthermore, in some embodiments, the invention may also be embodied in the form of a computer program product in one or more computer-readable media having computer-readable program code embodied in the medium.
Any combination of one or more computer-readable media may be employed. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The present invention is described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instruction means (instructions) which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
For the purpose of facilitating an understanding of the embodiments of the present invention, the following description will be made in terms of several specific embodiments with reference to the accompanying drawings, and the drawings are not intended to limit the embodiments of the present invention.
As shown in fig. 1, the data acquisition system based on bond transaction is a schematic structural diagram, and specifically, the data acquisition system includes a client 1, a server 2, a management end 3, and a log management server 23.
The client 1 comprises a quotation information module 4 for providing quotation information, a quotation inquiry module 5 for inquiring the quotation information, a click-to-deal processing module 6 for clicking to deal, a transaction inquiry module 7 for inquiring balance and deal intention and a login module 8 for logging in and logging out a client and modifying a password.
Further, the investor logs in the login module 8 and can also check basic information of traders, the quotation information module 4 can further check detailed quotation information, the quotation inquiry module 5 can also inquire bond information, and the trade inquiry module 7 can also be provided with a function of printing a delivery receipt. Further, the client 1 may further extend the corresponding modules according to the transaction requirements to meet the transaction requirements, for example, a financing module for lending funds.
The server 2 comprises a quotation processing module 9 for receiving, sorting, merging and pushing quotation information, a transaction processing module 10 for monitoring quotation intervals, generating temporary commitment instructions and generating and sending commitment instructions, a deal processing module 11 for receiving deal information, processing deal results and generating quotation data, a bookkeeping module 12 for bookkeeping management, a quotation module 13 for quotation calculation, a time-setting module 14 for providing time-setting service, a time-of-day processing module 15 for processing opening signals and bond information and a time-of-day end processing module 16 for synchronizing the balance of bond of investors, the balance of funds and sending investment increment information.
In one embodiment, the quotation processing module 9 receives quotation merchant price information from the foreign exchange trading center, splits the quotation merchant price information into two queues according to different buying and selling directions to receive the quotation information, can also perform quotation classification sequencing according to the principles of price priority and time priority, merges quotation quantities of the same bond and the same price, hides quotation merchant and quotation time information, and pushes the anonymously merged quotation information to the client and the management terminal in real time. In one embodiment, the transaction processing module 10 may receive the interest-in-transaction message from the investor, after clicking for the interest-in-transaction, split the paired interest-in-transaction message, and for the received interest-in-transaction message, conduct credit checking on the quotation provider and the investor, check on the sufficiency and sufficiency certificate, and generate a temporary commitment instruction to check the available quota, so as to ensure that the quota of the interest-in-transaction message meets the requirements of the sufficiency or sufficiency certificate. The transaction processing module 10 may also generate a corresponding commitment instruction for the meeting-checked intention-to-deal message; meanwhile, generating a debt account or a fund account of the investor, and sending a commitment instruction to the foreign exchange transaction center. In one embodiment, the deal processing module 11 receives deal information from the foreign exchange transaction center, checks the deal information identification: success or failure, failure with an associated cause. Marking the success or failure of the deal of the corresponding commitment instruction, adjusting the available amount when the deal fails to generate the bond account of the investor or fund account of the investor, and generating the corresponding market basic information by the deal processing module 11 according to the deal information. In one embodiment, the time synchronization module 12 has a network time synchronization function at a system level, and ensures that clocks of all modules, such as a client and a server, are consistent, and further, the clocks are consistent with an external foreign exchange transaction center.
In one embodiment, the daytime processing module 15 receives an opening signal from a foreign exchange trading center. After receiving, the system marks that the opening signal is received, waits for the field crew to start the opening preparation signal, and the daytime processing module 15 receives the real-time bond-by-bond basic information from the Shanghai clearing house.
In one embodiment, the end-of-day processing module 16 receives, for example, investor bond balance information from the Shanghai clearing house. The end-of-day processing module 16 receives a post-system flag indicating that the investor's bond balance has been received, receives investor's fund balance information from the Shanghai clearing house, and receives a post-system flag indicating that the investor's fund balance has been received. The day end processing module 16 sends the incremental investor basic information to the foreign exchange trading center, and after the incremental investor basic information is sent, the system marks the incremental investor basic information to indicate that the investor basic information is sent on the same day.
The management terminal 3 comprises an information query module 17 for providing information, a transaction monitoring module 18 for monitoring quotation information, commitment instructions and/or transaction applications, an information maintenance module 19 for maintaining bond information and quotation information, a day end monitoring module 20 for daily end monitoring investor information, bond balance information and fund balance information, an emergency information maintenance module 21 for processing investor information import, emergency transaction intention application input and emergency deal result input, and a system maintenance module 22 for setting system parameters.
In one embodiment, the basic query conditions of the information query module 17 include offer instructions, product categories, product names including codes, quotation providers, transaction directions, and time, and the information query module 17 displays all quotation information, and the quotation information is sorted by time by default. The information inquiry module 17 checks the change conditions such as the prior limit, the post-check limit, the deduction limit and the like through intention numbering, the click transaction application number, the product type, the product name containing codes, investors, quotation providers, time and check state passing and failing to check; the information inquiry module 17 inquires the transaction information through a commitment instruction, an offer instruction, an application number, a product type, a product name containing code, an investor, a quotation provider, time, a state of transaction, non-transaction and non-feedback, and the information inquiry module 17 further inquires the current day and the historical condition through the investor, a transaction intention number and a time inquiry showing the click transaction application, the check before the transaction, the commitment instruction, the transaction feedback, the settlement failure relevant state and the share change condition which are generated by the investor from the click transaction; the information inquiry module 17 may inquire the bond balance of the investor, the fund balance of the investor, and investor information.
In one embodiment, the transaction monitoring module 18 displays the time of receiving the latest quote information from the transaction center, and the time is used as the basis for the abnormal receiving condition of the quote information judged by the service post, and displays all the committed instruction records which do not receive the return information of the deal in more than 1 minute, and reserves the remark item, so that the service post can indicate the processing condition. The information maintenance module 19 may add, modify, and query bond information and bidder information.
In one embodiment, the end-of-day monitoring module 20 may query whether the investor incremental information has been sent on the day. If there is an abnormality, abnormality information and the cause should be displayed, which can be used as warning log information. The end-of-day monitoring module 20 may query whether the end-of-day treatment has been completed. If there is an abnormality, the cause should be prompted and may be used as a warning message.
In one embodiment, the client 1, the server 2, and/or the administrator 3 may be a cell phone, a pad, a computer, or a server.
In one embodiment, each module of the client 1, the server 2 and the manager 3 completes the processing of the whole transaction flow, and generates data including but not limited to quote data, click request data, transaction data, position data, market quotation data, etc.; the modules of the client 1, the server 2 and the management end 3 also record the whole transaction process, and generate interaction data including but not limited to query data, monitoring data, management data and external platforms such as a foreign exchange transaction center and a clearing house.
With the adoption of the method, as the bond transaction faces huge and large-scale data impact such as millions of concurrent user access, thousands of concurrent transaction processing per second and the like, data acquisition is needed to be optimized, the technical problems of large data volume, fast data change and the like in the current business system are effectively solved, the problems of data acquisition repetition and low quality are solved, and the reliability and the expansibility of data acquisition are improved.
The data acquisition server 23 includes a data extraction module 24, a data processing module 25, a distributed data storage management module 26 and a query module 27, wherein, the data extraction module 24 comprises a front-end processor 28 and an ETL conversion module 34 for collecting data generated by the client 1, the server 2 and the management end 3 in real time, the data processing module 25 comprises a data association module 29 for association by using parameter characteristics, a classification module 30 based on time-series classification, and a clustering module 31 for de-duplicating data, the distributed data storage management module 26 comprises a time-series data storage 32 for storing time-series data and a data storage 33 for storing non-time-series data, the distributed data storage management module 26 stores the time-series data in the time-series data storage 32 and stores the non-time-series data in the data storage 33, the query module 27 queries the distributed data storage management module 26 based on a query command.
The data acquisition server 23 of the present invention is provided with a front-end processor 28 that acquires data generated by the client 1, the server 2, and the management terminal 3 in real time. The data collection server 23 may use a plurality of front-end computers as intermediate devices of the client 1, the server 2 and the management 3. Further, the data acquisition server 23 employs one or more integrated front-end processors to simplify the system structure, save system investment, reduce the labor intensity of system maintenance, and reduce the consumption of system resources by the multiple front-end processors. The hardware of the integrated front-end processor comprises a PC server, an Ethernet card, a multifunctional card, a voice/data card, a host communication card, a network controller, a router and other network connection equipment. Because the communication protocols between the client 1, the server 2 and the management terminal 3 are different greatly and the network structure is complex, the client 1, the server 2 and the management terminal 3 can be connected easily for data exchange by taking the integrated front-end processor as an intermediary. The integrated front-end processor can also conveniently convert different formats in the client 1, the server 2 and the management end 3, for example, the integrated front-end processor can also realize character code conversion among hosts. Furthermore, the comprehensive front-end processor performs authentication processing on the transaction message, verifies all received message confidentiality and authenticates all system-sending messages. By the mechanism, financial risks to the system caused by the appearance of fake transaction messages can be effectively avoided. Similarly, because the transaction message transmitted by the network can be stolen, the integrated front-end processor needs to encrypt/decrypt the personal password entering/exiting the host system, so that the fund security of the trader can be ensured. The integrated front-end processor can record transaction flow and display transaction completion, which can help analyze and solve potential and occurred problems between systems. In addition, the flow recording and data statistics functions of the integrated front-end processor can provide basis for account checking and fund clearing among systems.
In an embodiment of the data collection system based on bond transaction of the present invention, the front-end processor preferably includes a transaction message format processing module, and the transaction message includes a notification type and a request/response type.
In one embodiment, the data collection server 23 employs a message exchange-based integrated front-end processor that processes all transactions based on financial transaction messages. The messages can be used for easily and clearly expressing various requirements of financial transactions. Any details of the financial transaction may be included in the message, as long as the message format is well defined. The transaction message may be formulated according to ISO 8583 international standards. Notification type messages can be simply employed if the financial transaction involves only system/network management; if the financial transaction involves accounting, a request/response type message may be used. The format of the transaction message may be such that the first part of the message is of the message type, 1 byte long. And the system transaction processing master control process appoints a corresponding message processing program according to the message type. The second part of the message is the message content, and the length is not fixed. It is the specific content of the financial transaction, and its generation is completed by the server 2 sending the message. After receiving the notification message, the message receiving process sorts the message content, then sends the message to the system main message queue, after receiving the message, the transaction processing main control process assigns the message to a corresponding notification message processing program for processing according to the message type, then forwards the message to the message sending process, and after sending the message, the transaction is finished. After receiving the transaction request message, the message receiving process 1 sorts the message content, then sends the message to the system main message queue, after receiving the request message, the transaction processing main control process assigns the message to a corresponding request message processing program for processing according to the message type, then forwards the message to the message sending process 2, and after sending the message, the transaction request processing is finished. After receiving the transaction request, the system processes and sends out a transaction response to the message receiving process 2, the process sorts the message content and then sends the response message to the system main message queue, after receiving the response message, the transaction processing main control process assigns the response message to a corresponding response message processing program for processing according to the message type, then forwards the message to the message sending process 1, and after sending the message, the transaction processing is finished. The integrated front-end processor can preprocess the transaction request and reject the undesirable transaction request. In this way, the transaction request is directly refused to respond in the front-end stage, thereby reducing the system load to a certain extent. After receiving the transaction request message, the message receiving process 1 sorts the message content, then sends the message to the system main message queue, after receiving the message, the transaction processing main control process assigns the message to a corresponding transaction request processing program for processing according to the message type, then forwards the response refusing message to the message sending process 1, and after sending the message, the transaction is finished.
The embodiment of the data collection system based on bond transaction of the present invention preferably includes an ETL conversion module 34 including a data extraction unit, a data washing unit, a null value processing unit, a data format processing unit, a data splitting unit, and a data replacing unit.
In one embodiment, the data extraction unit adopts full-table comparison data extraction, the data extraction unit extracts all data from the client 1, the server 2 and the management terminal 3, performs corresponding rule conversion, and performs target table comparison on each piece of data without inserting a target after the conversion is completed. And judging whether the primary key value exists in the target table or not according to the primary key value, wherein the primary key value exists in the target table, the record exists, comparing other fields, and if the fields are different, performing an updating operation, wherein if the primary key value does not exist in the target table, the record does not exist, and then performing an inserting operation. The data extraction unit has no influence on the table structure of the existing system, does not need to modify a service operation program, and can realize the incremental loading of data without risk. The method is suitable for extraction of a transaction system. In one embodiment, the data extraction unit adds a time field as a timestamp in the online transaction processing uniformly in the service table, and if the corresponding time field is already in the table, the time field may not be added, and the value of the timestamp field must be modified at the same time when the service data is updated and modified. When the ETL is loaded, the comparison of the system time and the timestamp field determines what data extraction is to be performed. The data extraction unit is relatively clear and simple and has high speed. In one embodiment, the data extraction unit classifies and extracts the data according to the transaction requirements. For example, the user transaction running amount is classified and extracted in the transaction module.
In one embodiment, the data cleansing unit employs structure cleansing to filter some redundant data. For example, multiple interpolation or multiple regression algorithm is used to remove invalid and repeated data, thereby improving data quality.
In one embodiment, the null processing unit captures field null values, loads or replaces with other meaning data.
In one embodiment, the data format processing unit implements field format constraint definitions, which are customizable for time, value, character, etc. data in the data source.
In one embodiment, the data splitting unit decomposes the fields according to business requirements.
In one embodiment, the data replacement unit implements replacement of invalid data, missing data.
In an embodiment of the data collection system based on bond transaction of the present invention, the ETL conversion module 34 is preferably a Datastage ETL.
In the invention, the data of the client 1, the server 2 and the management terminal 3 are extracted and preprocessed by the data extraction module 24 and then sent to the data processing module 25 for processing, and the data processing module 25 comprises a data association module 29 for associating by using parameter characteristics, a classification module 30 based on time sequence classification and a clustering module 31 for removing the duplicate of the data.
The embodiment of the data collection system based on bond transaction of the present invention preferably includes, but is not limited to, the time of data generation, the module of data generation, the IP address of data generation, the data format and/or the data type. The data generation module comprises one or more of a market condition query module 5, a click-to-deal processing module 6, a transaction query module 7, a quotation processing module 9 and a transaction processing module 10, and of course, the data generation module can also come from other modules in the client 1, the server 2 and the management terminal 3.
The data correlation module 29 uses the parameter features to correlate frequent patterns, correlations, or causal structures among the data. In one embodiment, the data association module associates using an Apriori algorithm based on the parameter characteristics. The Apriori algorithm uses an iterative method of layer-by-layer search, where a k-term set is used to explore a k + 1-term set, and first, finds a set of frequent 1-term sets, denoted as L1, and L1 is used to find a set of frequent 2-term sets, L2, and then finds L3, and so on until a frequent k-term set cannot be found, data needs to be scanned once for finding each Lk. The data association module logically divides data into a plurality of mutually disjoint blocks, considers one block at a time and generates all frequent item sets for the block, then combines the generated frequent item sets to generate all possible frequent item sets, and finally calculates the support of the item sets. The size of the blocks is here chosen such that each block can be put into main memory and only needs to be scanned once per stage. The correctness of the algorithm is ensured by that each possible frequent item set is a frequent item set at least in a certain block. The data in the bond transaction can be associated with the data with the cause relation and the higher relevance through the data association module. For example, the data association module associates data having transaction attributes for the same time period.
The classification module 30 classifies the above-described related data based on time series, which is a data sequence recorded in time series by the same uniform index. The data in the same data column must be of the same aperture, requiring comparability. The time series data can be the number of epochs or the number of epochs. The classification module 30 classifies the associated data into time-series data and non-time-series data.
The clustering module 31 deduplicates the data. The clustering module 31 adopts a random search clustering algorithm, firstly randomly selects a point as a current point, then randomly checks some adjacent points around the current point which do not exceed the parameter Max neighbor number, if an adjacent point which is better than the current point is found, moves the adjacent point into the adjacent point, otherwise, takes the point as a local minimum. Then randomly selecting a point to find another local minimum until the found local minimum reaches the requirement of the user.
In one embodiment, the clustering module is a Kmeans clusterer. The Kmeans clustering device randomly selects k objects from n data objects as an initial clustering center; and for the rest other objects, respectively allocating the other objects to the most similar clusters according to the similarity between the other objects and the cluster centers; then calculating the clustering center of each obtained new cluster; this process is repeated until the standard measure function begins to converge. The Kmeans clusterer uses the mean square error as a standard measure function.
The distributed data storage management module 26 includes a time series data storage 32 that stores time series data and a data storage 33 that stores non-time series data. The distributed data storage management module improves the storage, retrieval and management efficiency of mass data and files, performs structured division on the mass data and the files, and stores data belonging to the structured data in a specified database, such as a data encapsulation storage system. The distributed data storage management module 26 stores time series data in a time series data store 32 and non-time series data in a data store 33. The time-series database may be provided in the time-series database 32, and the time-series database may store time-series data, and the non-time database may be provided in the data memory 33, and the non-time-series database may store non-time-series data.
In one embodiment, the time series data store 32 is used to store a memory that collectively caches frequently accessed, performance-critical data into the time series data store.
In one embodiment, the databases involved in the time series data storage 32 are mainly a distributed file system HDFS, a columnar database HBase, an in-memory database Redis, a relational database Oracle, and the like. The Oracle database is mainly used for storing configuration data and part of service data, the HDFS serves as a distributed file system unit of a bottom layer of a big data platform and provides support for HBASE of an upper layer, and a non-time sequence part in the service data can also be directly stored, the HBASE is a distributed storage system which is high in reliability and performance, is column-oriented and telescopic and is mainly used for storing the time sequence part in the service data, and the Redis is a key-value storage system based on a memory and is mainly used for storing cache data.
The query module 27 queries the distributed data storage management module 26 based on a query command.
In one embodiment, the query command may be based on an SQL command.
The data acquisition system based on bond transaction is stable, reliable and efficient, takes the open source distributed memory and the server of the parallel computing service as the core, and provides stable and reliable bottom data support for time sequence data storage and non-time sequence data oriented packaging; the real-time and punctual data acquisition has high timeliness, the data acquisition frequency is optimally designed, the acquisition efficiency is high, the energy efficiency is low, but the efficiency is high, the processing timeliness is higher, the equipment loss is reduced due to the fact that the processing time is shortened, the cost is saved, the service life is longer, and the performance is more stable.
Fig. 2 is a schematic diagram showing the steps of the data acquisition method of the data acquisition system based on bond transaction according to the present invention, and the steps of a data acquisition method using the data acquisition system based on bond transaction include:
in the first step S1: the front-end processor 28 collects data generated by the client 1, the server 2 and the management end 3 in real time.
In the second step S2: the ETL transformation module 34 extracts, cleans, and transforms the data.
In the third step S3: the data association module associates the data with the parameter characteristics.
In the fourth step S4: the classification module separates the data into time-series data and non-time-series data based on parameter characteristics.
In the fifth step S5: the clustering module performs deduplication on the data.
In the sixth step S6: the distributed data storage management module stores time series data in the time series data storage and stores non-time series data in the data storage.
In the seventh step S7: and the query module queries in the distributed data storage management module based on the query command.
The data collecting method of the data collecting system based on bond transaction of the present invention is preferably implemented in the third step S3: and the data association module associates the data based on the parameter characteristics by using a multi-valued attribute MAQA algorithm. The multi-value attributes of the multi-value attribute MAQA algorithm may be divided into a number attribute and a category attribute. The mining of the multi-value attribute association rule is converted into the mining of the Boolean type association rule, namely, the value of the multi-value attribute is divided into a plurality of intervals, each interval is used as an attribute, and each category of the category attribute is used as an attribute.
In the data collecting method of the data collecting system based on bond transaction according to the embodiment of the present invention, preferably, in the fourth step S4: the classification module classifies the data into time-series data and non-time-series data based on parameter feature pairs by using a GARCH algorithm. The GARCH algorithm further models the variance of the error, and is particularly suitable for classification of transaction data.
In the data collecting method of the data collecting system based on bond transaction according to the present invention, preferably, in the fifth step S5: and the clustering module adopts a CLARA algorithm to perform data deduplication. In one embodiment, the CLARA algorithm extracts multiple sample sets from the data set, uses PAM for each sample set, and takes the best cluster as output. (1) for i 1to v (number of samplings), the following steps ((2) to (4)) are repeatedly performed; (2) randomly extracting a sample of N objects from the whole database, and calling a PAM (pulse amplitude modulation) method to find k optimal central points of the sample from the sample; (3) applying the k central points to the whole database, and judging which representative object selected from the sample is closest to each non-representative object Oj; (4) if the value is smaller than the current minimum value, replacing the current minimum value with the value, and keeping k representative objects obtained in the sample selection as the set of the best representative objects obtained so far; (5) and (4) returning to the step (1), starting the next cycle, and outputting the best clustering result after the algorithm is finished.
Industrial applicability
The data acquisition system based on bond transaction and the data acquisition method thereof can be manufactured and used in the field of data acquisition.
Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to the above-described embodiments and application fields, and the above-described embodiments are illustrative, instructive, and not restrictive. Those skilled in the art, having the benefit of this disclosure, may effect numerous modifications thereto without departing from the scope of the invention as defined by the appended claims.
Claims (7)
1. A data acquisition system based on bond transaction, the data acquisition system comprises a client (1), a server (2), a management end (3) and a data acquisition server (23), and is characterized in that:
client (1) comprising:
a quotation information module (4) for providing quotation information;
a market information inquiry module (5) for inquiring market information;
a click deal processing module (6) for click deal;
a transaction inquiry module (7) used for inquiring the balance and the transaction intention, wherein the transaction inquiry module (7) can also be arranged to print a transaction order;
a financing module for lending funds; and
a login module (8) for logging in and logging out the customer and modifying the password, wherein the login module can also view basic information of the trader;
a server (2) comprising:
the quotation processing module (9) is used for receiving, sequencing, merging and pushing quotation information, the quotation processing module (9) receives quotation merchant and quotation price information from a foreign exchange trading center, splits the quotation merchant and the quotation price information into two queues according to different buying and selling directions to receive the quotation information, classifies and sequences the quotation according to the principles of price priority and time priority, merges the quotation amount of the same bond and the same price, hides the quotation merchant and the quotation time information and pushes the anonymously merged quotation information to a client and a management terminal in real time;
the trading processing module (10) is used for monitoring quotation intervals, generating a temporary commitment instruction and generating and sending the commitment instruction, the trading processing module (10) can receive a trading interest message which is split and paired after an investor clicks a trade, and aiming at the received trading interest message of the investor, credit checking is carried out on a quotation provider and the investor, a full-mark and sufficient-mark check is carried out, the temporary commitment instruction is generated for checking the available amount, so that the amount of the trading interest message meets the requirements of the full mark or the sufficient mark, the trading processing module (10) also generates a corresponding commitment instruction aiming at the trading interest message which is met by checking, and simultaneously generates a debt account or an investor fund account, and sends the commitment instruction to a foreign exchange trading center;
the deal processing module (11) is used for receiving deal information, processing deal results and generating quotation data, the deal processing module (11) receives deal information from the foreign exchange transaction center, checks whether the deal information is marked as successful or failed, has a relevant reason description for failure, marks the success or failure of a corresponding commitment instruction, adjusts the available amount when the deal information fails, generates an investor bond account or an investor fund account, and the deal processing module (11) generates corresponding quotation basic information according to the deal information;
a bookkeeping module (12) for bookkeeping management;
a market module (13) for market calculation;
the time setting module (14) provides time setting service, the time setting module (14) has the network time setting function of a system level, all module clocks are ensured to be consistent, and the clocks are consistent with an external foreign exchange trading center;
the daytime processing module (15) is used for processing the opening signal and the bond information, the daytime processing module (15) receives the opening signal from the foreign exchange transaction center, the system marks after receiving the opening signal and indicates that the opening signal is received, a field crew starts an opening preparation signal is waited, and the daytime processing module (15) receives real-time bond-by-bond basic information from the Shanghai clearing house; and
the daily end processing module (16) is used for synchronizing the bond balance and the fund balance of the investor and sending the increment information of the investor, the daily end processing module (16) receives the bond balance information of the investor from the Shanghai clearing house, the received system marks the bond balance of the investor on the same day, the received fund balance information of the investor from the Shanghai clearing house, the received system marks the bond mark on the same day, the daily end processing module (16) sends the increment basic information of the investor to the foreign exchange trading center, and after the sending, the system marks the basic information of the investor on the same day;
a management terminal (3) comprising:
the information inquiry module (17) is used for providing information, the basic inquiry conditions of the information inquiry module (17) comprise an offer instruction, a product type, a product name containing code, a quotation provider, a transaction direction and time, the information inquiry module (17) displays all quotation information and defaults to sort according to time, the information inquiry module (17) inquires the change conditions of the quota through intention number, click-to-exchange application number, the product type, the product name containing code, investors, quotation provider, time and check state, the quota before check is not inquired, the quota after check and the deduction quota, the information inquiry module (17) inquires the transaction information through a commitment instruction, the offer instruction, the application number, the product type, the product name containing code, investors, quotation provider, time and state, the transaction is already done, the transaction is not done, and the transaction information is not fed back, and the information inquiry module (17) inquires, The trading interest number and the time inquiry show click deal application generated by an investor from the start of click deal, pre-trade check, a commitment instruction, deal feedback, settlement failure related state and share change conditions, the current day and historical conditions are inquired, and an information inquiry module (17) inquires bond balance of the investor, fund balance of the investor and investor information;
the transaction monitoring module (18) is used for monitoring quotation information, a commitment instruction and/or transaction application, the transaction monitoring module (18) displays the time of receiving the latest quotation information of the transaction center as the basis of judging the abnormal receiving condition of the quotation information at the service post, displays all the commitment instruction records which do not receive the return information of the transaction in more than 1 minute, and reserves a remark item so as to indicate the processing condition at the service post;
the information maintenance module (19) is used for maintaining the bond information and the quotation provider information, and the information maintenance module (19) can increase, change and inquire the bond information and the quotation provider information;
the day end monitoring module (20) is used for monitoring investor information, bond balance information and fund balance information day end, the day end monitoring module (20) can inquire whether the investor increment information day end is sent or not, if the investor increment information day end is abnormal, abnormal information and reasons are displayed to be used as warning log information, the day end monitoring module (20) can inquire whether day end processing is finished or not, and if the investor increment information day end is abnormal, the reasons are prompted to be used as warning information;
the emergency information maintenance module (21) is used for processing investor information import, emergency transaction intention application input and emergency transaction result input; and
a system maintenance module (22) for setting system parameters;
the data acquisition server (23) comprises:
a data extraction module (24), the data extraction module (24) comprising:
the front-end processor (28) is used for acquiring data generated by the client (1), the server (2) and the management terminal (3) in real time; the front-end processor (28) comprises a transaction message format processing module, the transaction message comprises a notification type and a request/response type, and the front-end processor authenticates the transaction message, verifies all received message confidentiality and authenticates all messages of a sending system; and
the ETL conversion module (34), the ETL conversion module (34) includes data extraction unit, data cleaning unit, null value processing unit, data format processing unit, data splitting unit and data replacement unit; wherein the content of the first and second substances,
the data extraction unit adopts full-table comparison data extraction, extracts all data from the client 1, the server 2 and the management terminal 3, performs corresponding rule conversion, does not insert a target after the conversion, performs target table comparison on each piece of data, performs judgment of insertion and update according to a primary key value, wherein the primary key value exists in a target table, indicates that the record exists, performs comparison on other fields, and performs update operation if the primary key value does not exist in the target table, indicates that the record does not exist, i.e. performs insertion operation; the data extraction unit adds time field as time stamp in the business table in the online transaction processing, if the corresponding time field is existed in the table, the time stamp field value is not added, when the business data is updated and modified, the time stamp field value is modified at the same time, when ETL is loaded, the comparison between the system time and the time stamp field is used to determine which data extraction is performed; the data extraction unit classifies and extracts data according to transaction requirements;
the data cleaning unit removes invalid and repetitive data from the data;
the null value processing unit captures field null values and loads or replaces the field null values with other meaning data;
the data format processing unit realizes the field format constraint definition, and can define the format of time, numerical value and character data in a data source by user;
the data splitting unit decomposes the fields according to the service requirements;
the data replacement unit realizes the replacement of invalid data and missing data;
a data processing module (25), wherein the data processing module (25) comprises a data association module (29) for associating by using parameter characteristics, a classification module (30) based on time sequence classification and a clustering module (31) for removing the duplicate of the data; wherein the content of the first and second substances,
the parameter characteristics comprise data generation time, a data generation module, a data generation IP address, a data format and/or a data type;
a data association module (29) associates frequent patterns, associations, correlations, or causal structures among the data using the parameter features; the data association module logically divides data into a plurality of mutually disjoint blocks, considers one block each time and generates all frequent item sets, then combines the generated frequent item sets to generate all possible frequent item sets, and finally calculates the support of the item sets;
the classification module (30) based on time sequence classification is used for classifying the related data based on time sequence, and the time sequence data is a data column recorded by a unified index according to time sequence;
a clustering module (31) for de-duplicating data;
a distributed data storage management module (26), the distributed data storage management module (26) comprises a time sequence data storage (32) for storing time sequence data and a non-time sequence data storage (33) for storing non-time sequence data, the distributed data storage management module (26) stores the time sequence data in the time sequence data storage (32) and stores the non-time sequence data in the non-time sequence data storage (33), a time sequence database can be arranged in the time sequence data storage (32), the time sequence data is stored in the time sequence database, a non-time sequence database is arranged in the non-time sequence data storage (33), the non-time sequence database is stored in the non-time sequence database, and
a query module (27), the query module (27) querying in the distributed data storage management module (26) based on a query command.
2. The bond transaction based data collection system of claim 1, wherein: the ETL conversion module (34) is a Datastage ETL, and the clustering module is a Kmeans clustering device.
3. The bond transaction based data collection system of claim 1, wherein: and the data association module performs association by using an Apriori algorithm based on the parameter characteristics.
4. A data collection method using the data collection system based on bond transaction of any one of claims 1to 3, comprising the steps of:
in the first step (S1): the front-end processor (28) collects data generated by the client (1), the server (2) and the management terminal (3) in real time;
in the second step (S2): an ETL conversion module (34) extracts, cleans and converts the data;
in the third step (S3): the data association module associates the data by using the parameter characteristics;
in the fourth step (S4): a classification module separates the data into time-series data and non-time-series data based on parameter characteristics;
in the fifth step (S5): the clustering module is used for carrying out duplicate removal on the data;
in the sixth step (S6): the distributed data storage management module stores time sequence data in a time sequence data memory and stores non-time sequence data in a data memory;
in the seventh step (S7): and the query module queries in the distributed data storage management module based on the query command.
5. The data acquisition method of claim 4, wherein:
in the third step (S3): and the data association module associates the data based on the parameter characteristics by using a multi-valued attribute MAQA algorithm.
6. The data acquisition method of claim 4, wherein:
in the fourth step (S4): the classification module classifies the data into time-series data and non-time-series data based on parameter feature pairs by using a GARCH algorithm.
7. The data acquisition method of claim 4, wherein:
in the fifth step (S5): and the clustering module adopts a CLARA algorithm to perform data deduplication.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710244125.2A CN107123047B (en) | 2017-04-14 | 2017-04-14 | Data acquisition system based on bond transaction and data acquisition method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710244125.2A CN107123047B (en) | 2017-04-14 | 2017-04-14 | Data acquisition system based on bond transaction and data acquisition method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107123047A CN107123047A (en) | 2017-09-01 |
| CN107123047B true CN107123047B (en) | 2020-12-29 |
Family
ID=59725282
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710244125.2A Active CN107123047B (en) | 2017-04-14 | 2017-04-14 | Data acquisition system based on bond transaction and data acquisition method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107123047B (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107944866B (en) * | 2017-10-17 | 2021-08-31 | 厦门市美亚柏科信息股份有限公司 | Transaction record duplication elimination method and computer-readable storage medium |
| CN108304443A (en) * | 2017-11-29 | 2018-07-20 | 上海金融期货信息技术有限公司 | Data commission playback monitoring system based on flow data inverting |
| CN110009502B (en) * | 2019-04-03 | 2023-10-27 | 平安科技（深圳）有限公司 | Financial data analysis method, device, computer equipment and storage medium |
| CN110490586A (en) * | 2019-08-22 | 2019-11-22 | 北京金融资产交易所有限公司 | Equity class transaction in assets system |
| CN110880131A (en) * | 2019-11-11 | 2020-03-13 | 深圳前海微众银行股份有限公司 | Invoice generation method and device |
| CN111241074B (en) * | 2019-12-27 | 2023-07-04 | 冶金自动化研究设计院 | Steel enterprise data center application system based on time sequence data and relation data |
| CN111598470B (en) * | 2020-05-20 | 2023-03-24 | 贵州电网有限责任公司 | Distribution network material market price acquisition, monitoring and early warning method and system |
| CN112631159B (en) * | 2020-11-24 | 2022-06-21 | 泰康保险集团股份有限公司 | Monitoring method, monitoring device, storage medium and electronic equipment |
| CN112613717A (en) * | 2020-12-17 | 2021-04-06 | 安徽兆尹信息科技股份有限公司 | Financial data processing method and storage medium |
| CN112632127B (en) * | 2020-12-29 | 2022-07-15 | 国华卫星数据科技有限公司 | Data processing method for real-time data acquisition and time sequence of equipment operation |
| CN113190583B (en) * | 2021-05-14 | 2024-02-20 | 长春理工大学 | Data acquisition system, method, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102663649A (en) * | 2012-05-18 | 2012-09-12 | 苏州工业园区凌志软件有限公司 | Financial derivative transaction system |
| CN106156350A (en) * | 2016-07-25 | 2016-11-23 | 恒安嘉新(北京)科技有限公司 | The big data analysing method of a kind of visualization and system |
| CN106407216A (en) * | 2015-07-31 | 2017-02-15 | 国网能源研究院 | Clue tracing audition system developed on basis of semantic net construction path and construction method of clue tracing audition system |
-
2017
- 2017-04-14 CN CN201710244125.2A patent/CN107123047B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102663649A (en) * | 2012-05-18 | 2012-09-12 | 苏州工业园区凌志软件有限公司 | Financial derivative transaction system |
| CN106407216A (en) * | 2015-07-31 | 2017-02-15 | 国网能源研究院 | Clue tracing audition system developed on basis of semantic net construction path and construction method of clue tracing audition system |
| CN106156350A (en) * | 2016-07-25 | 2016-11-23 | 恒安嘉新(北京)科技有限公司 | The big data analysing method of a kind of visualization and system |
Non-Patent Citations (1)
| Title |
|---|
| "数据挖掘在图书馆个性化服务中的应用研究";潘小凤;《中国优秀硕士学位论文全文数据库 信息科技辑》;20140715(第7期);正文第4.1节 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107123047A (en) | 2017-09-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107123047B (en) | Data acquisition system based on bond transaction and data acquisition method thereof | |
| US10715598B1 (en) | Implementation of a web-scale data fabric | |
| CN103620601B (en) | Joining tables in a mapreduce procedure | |
| US20210092160A1 (en) | Data set creation with crowd-based reinforcement | |
| US20190050435A1 (en) | Object data association index system and methods for the construction and applications thereof | |
| CN103064933A (en) | Data query method and system | |
| US10783453B2 (en) | Systems and methods for automated incident response | |
| JP2022118108A (en) | Log auditing method, device, electronic apparatus, medium and computer program | |
| CN112365355B (en) | Method, device and readable medium for calculating foundation valuation and risk index in real time | |
| US10679230B2 (en) | Associative memory-based project management system | |
| CN111666346A (en) | Information merging method, transaction query method, device, computer and storage medium | |
| CN103077192A (en) | Data processing method and system thereof | |
| CN110046188A (en) | Method for processing business and its system | |
| CN111897790A (en) | Wind control log collection method and device, electronic equipment and storage medium | |
| CN111800292A (en) | Early warning method and device based on historical flow, computer equipment and storage medium | |
| CN113221570A (en) | Processing method, device, equipment and storage medium based on-line inquiry information | |
| CN113535677A (en) | Data analysis query management method and device, computer equipment and storage medium | |
| US11822578B2 (en) | Matching machine generated data entries to pattern clusters | |
| EP4348441A1 (en) | Systems and methods for ensuring quality of search system data | |
| CN114155076A (en) | Method, device and equipment for checking business data and financial data | |
| CN111695077A (en) | Asset information pushing method, terminal equipment and readable storage medium | |
| KR20150077669A (en) | Data Analysis Method and System Using MapReduce Approach | |
| CN112328960B (en) | Optimization method and device for data operation, electronic equipment and storage medium | |
| CN117112556A (en) | Data processing method, device, equipment, medium and product | |
| CN116629816A (en) | Human resource management and decision-making aid system and method based on big data, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |