WO2017181872A1 - Système et procédé de traitement de données - Google Patents

Système et procédé de traitement de données Download PDF

Info

Publication number
WO2017181872A1
WO2017181872A1 PCT/CN2017/080093 CN2017080093W WO2017181872A1 WO 2017181872 A1 WO2017181872 A1 WO 2017181872A1 CN 2017080093 W CN2017080093 W CN 2017080093W WO 2017181872 A1 WO2017181872 A1 WO 2017181872A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
module
processing module
source
access terminal
Prior art date
Application number
PCT/CN2017/080093
Other languages
English (en)
Chinese (zh)
Inventor
杨燕明
赵根苗
茅毓铭
徐湧
占卫平
Original Assignee
中国银联股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中国银联股份有限公司 filed Critical 中国银联股份有限公司
Publication of WO2017181872A1 publication Critical patent/WO2017181872A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses

Definitions

  • the present application proposes a method for data distribution and extraction, which separates a source access end from a target access end.
  • the source access end and the target access end can access the data processing system by using the same communication protocol, and the source access end performs the data format (for example, the first format) and the publishing mode agreed upon with the data processing system. Data release.
  • the target access end performs data extraction according to a data format (for example, a second format) and a subscription mode agreed upon with the data processing system.
  • the source access point and the target access end are not directly coupled to each other and therefore do not affect each other.
  • the method of the present application also supports simultaneous access of multiple sources and multiple targets, supports one data multiple extraction, supports data format conversion, and can realize smooth release and extraction of data under high transaction volume.
  • the task scheduling module is configured to schedule the first data segment to the sending process based on subscription task information from the target access terminal Module.
  • the task scheduling module is configured to periodically poll each data segment stored in the data cache.
  • the task scheduling module is configured to schedule the first data segment to the transmission processing module in a case where it is determined that the first data segment is not full when the predetermined time is exceeded.
  • the first data stores a check value in the header for the receiving processing module to verify the integrity and correctness of the first data.
  • the second data also has a check value in the header, so that the target access end verifies the integrity and correctness of the second data.
  • a method for data distribution and extraction between a source access terminal and a target access terminal includes: receiving first data having a first format from the source access terminal (step A); performing checksum decompression and reduction on the first data to obtain source data (step B); And decompressing the restored source data according to its type and storing the data segment in the data cache (step C); scheduling the data to be sent according to the data segment (step D); and converting the data to be sent into having The second data of the second format is sent to the target access terminal (step E).
  • step D may include scheduling the first data segment based on subscription task information from the target access terminal when determining that the first data segment stored in the data cache is full.
  • step D may further include periodically polling the numbers stored in the data cache. According to paragraph.
  • the first data segment is scheduled in a case where it is determined that the first data segment is not full when the predetermined time is exceeded.
  • the first data stores a check value in the header for the receiving processing module to verify the integrity and correctness of the first data.
  • FIG. 1 is a flow chart showing a method of data distribution and extraction according to an embodiment of the present application
  • FIG. 2 is a schematic structural diagram of a data processing system according to an embodiment of the present application.
  • FIG. 3 is a schematic diagram of data caching in accordance with an embodiment of the present application.
  • a data distribution and data extraction method is illustrated.
  • the source access terminal (referred to as the source end) performs data distribution, and sends a message to the data processing system according to the data format and the distribution mode agreed with the data processing system.
  • the data processing system feeds back the processing situation to the source end, and the source end determines the subsequent data location according to the feedback situation.
  • the data processing system sends the data subscribed by the target access terminal to the target access terminal according to the data format and the extraction mode agreed with the target access terminal.
  • the target access end (referred to as the target end) feeds back the processing to the data processing system, and the data processing system determines the subsequent data processing according to the feedback situation.
  • FIG. 2 further illustrates a block diagram of a data processing system in accordance with an embodiment of the present application.
  • the data processing system includes a communication interaction module, a data cache module, a reception processing module, a task scheduling module, and a transmission processing module.
  • each module will be introduced separately.
  • the communication interaction module is an intermediary between the data source and the target.
  • the source access terminal sends the data to be released to the communication interaction module, and the target access terminal also performs data extraction from the communication interaction module.
  • the communication interaction module is responsible for connecting the source end and the target end, and is responsible for invoking the subsequent module, and sends the processing status to the source end and the target end.
  • all data sources (systems) and data destinations (systems) need to adopt a uniform access standard when connecting to a data processing system.
  • the data processing system supports communication with the source data terminal or the target data terminal by means of TCP/IP, file, IPC, etc., but the content of the communication must follow the unified standard and format.
  • the content of the message/file may comprise two components, in order: a header, a data field. Its structure is shown in the following table:
  • the header is the first data element of the message/file, and records basic information such as data type, data length, number of data, data source identifier, data destination identifier, and check code.
  • One or more data records can be carried in the data field.
  • the target end (identified as SUB) and the data processing system agree that the source data type needs to be extracted as part of A1 (assuming the converted length is 30 bytes), then the data processing system will pass the TCP/IP method to the target end. Transfer the following message:
  • the data processing system After the source and the data processing system agree on the basic information such as the data type, the data processing system generates a series of data caches for the source data.
  • the main purpose of data caching is to coordinate the processing efficiency of upstream and downstream systems and improve the performance of data processing.
  • the cache uses a segmented scrolling approach.
  • the data sent by the upstream source is stored in the data cache in order after receiving, and the data received by the downstream target is sent out in order from the data cache.
  • the data buffer is in an equilibrium state and acts as a buffer.
  • the data processing capability of the downstream target end cannot keep up with the data transmission capability of the upstream source, the data cache may be blocked after a period of time. At this time, the data processing system may instruct the upstream source system to suspend transmission (for example, requesting a certain time to resend or Is to reduce the speed of sending commands and other instructions).
  • the data customized by the downstream target end the data of the first format agreed by the source end and the data processing system may not necessarily be the same in format and order. Therefore, it is necessary to perform certain processing and cutting on the data stored in the data cache to generate a conformity.
  • the data processing system implements the common location-based data processing and cutting rules, such as copying, adding, modifying, deleting, etc., and the data content customized by the downstream target end can be obtained by configuring the parameters.
  • the receiving processing module is configured to receive the packet sent by the communication interaction module. Since the communication interaction module is only responsible for data transmission, the receiving processing module needs to parse the externally sent data. In an embodiment, the receiving processing module needs to perform operations such as source data verification, source data decompression and restoration, and data cache processing on the data sent by the source end.
  • the source data check verifies the data content (data to be processed) sent by the source end to ensure the integrity and correctness of the data sent.
  • a KEY value is generated for each access end (including the source and the target), and the access end adopts an algorithm agreed with the data processing system according to the KEY value, for example, an algorithm such as MD5/DES may be used.
  • the data field in the sent message (after compression) is calculated, and the calculated result is filled in the check value field in the message header.
  • the data processing system rejects the source data of the failed verification (returns a response message, where the data field indicates the reason for the rejection).
  • the source data is decompressed and restored, and the source data packet sent by the source end is decompressed, and the decompression is adopted by the access end.
  • a good compression/decompression algorithm for example, an algorithm such as zlib/lzma. Compression can significantly reduce the amount of data transfer and greatly improve the overall performance of the data processing system.
  • the source data cache process stores the source data that has been verified and decompressed and restored in the data cache, and the data processing system sequentially writes the data write type according to the type of the data sent.
  • the subsequent task scheduling module When it is found that the segment is full, it will immediately call the subsequent task scheduling module to process the data in the segment; when it finds that there is no available segment, it will return the instruction to suspend the transmission to the source, and the source receives the response. , will pause the work to resend this data after the specified time.
  • the task scheduling module is responsible for processing the data after the receiving processing module writes the cache, and its main function is to route the subscription task information of the target end according to the agreement between the target end and the data processing system, and to process the data segment (not every data).
  • the scheduling is dispatched to subsequent transmission processing modules for data distribution.
  • the data buffer segment is scheduled to be dispatched in two cases. One is that the receiving processing module finds that the segment is full and notifies the task scheduling module to perform scheduling, and the other is that the task scheduling module polls for discovery. If there is data in the segment but it is not filled for more than the specified time, the design is to make the data dispatched and dispatched in time.
  • this task scheduling module is designed to handle such one-to-many, many-to-many data task routing and conversion. .
  • the task scheduling module determines, according to the subscription task information table, which target receiving end to send the data segment to be processed, and sends the target end data by calling the sending processing module.
  • the task scheduling module also monitors the processing of the specified data segment to all associated target terminals that have integrity requirements for the data, and re-calls when the data on a segment is found to have no feedback for a long time.
  • the sending processing module continues the data distribution of the segment cache to prevent the sending process from exiting abnormally.
  • the transmit processing module can have two functions, one is responsible for processing the task scheduling module.
  • the processing request indicates that the data in the specified data buffer segment is converted and processed according to the rule and sent to the target end; the second is responsible for processing the response returned by the target end, and if the data sent last time receives a failure response or no response timeout Will be sent again.
  • the sending processing module needs to perform operations such as target data verification, cache processing, and the like on the feedback response sent by the target end.
  • the target data check verifies the feedback response content sent by the target end, ensures that the response is consistent with the request, ensures the integrity and correctness of the data, and the verification method is consistent with the source data check. deal with.
  • the target data cache processing is processed for the feedback response content sent by the target end. If the feedback response sent by the target end is successfully processed, it indicates that the data processing system can continue to process the remaining data to be processed in the segment. The target can be placed in the "Completed” state until all data in the segment has been processed. When all target-side states that have subscribed to the data become "completed", the segment can be freed from the data cache. If the feedback response sent by the target end is an abnormal response, subsequent processing is performed according to the exception handling method agreed with the target end (such as skipping, resending several times, or isolating the specified target end).
  • the exception handling method agreed with the target end (such as skipping, resending several times, or isolating the specified target end).
  • the sending processing module processes only the specified cached data segment.
  • the sending processing module converts to the specified target data type one by one according to the conversion rule, and the data to be processed in the segment ends or reaches
  • the communication interaction module is called to send out.
  • the data transformation system supports simple location-based copying, replacement, adding, content filtering and other functions, and can flexibly extend the deformation transformation rules by developing dynamic libraries.
  • Rule number Rule step behavior specific contents Rule 1 1 filter 141–143 is equal to 'S20' Rule 2 1 copy 1–20 Rule 2 2 copy 40–50 Rule 2 3 replace 60–65 for *** Rule 3 1 copy 1–20 Rule 3 2 replace 40–50 for ABCD
  • a data processing system can include a communication interaction module configured to couple with the source access terminal and the target access terminal and receive the first format from the source access terminal a data processing unit, configured to receive the first data having the first format from the communication interaction module, perform verification and decompression on the first data to obtain source data, and verify And the source data after being restored by the decompression is stored in the data cache according to the type thereof; the sending processing module converts the data to be sent into the second data having the second format, and a communication interaction module is sent to the target access terminal; and a task scheduling module is disposed between the receiving processing module and the sending processing module, and configured to schedule data to be sent according to a data segment Send the processing module to the said.
  • a method of data distribution and extraction between a source access terminal and a target access terminal may include: receiving, by the source access terminal, first data having a first format; performing checksum decompression and reduction on the first data to obtain source data; and passing the verification and decompressing the restored data
  • the source data is stored in the data cache according to the type of the data segment; the data to be sent is scheduled according to the data segment; the data to be transmitted is converted into the second data having the second format, and sent to the target access end.

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Computer And Data Communications (AREA)

Abstract

L'invention concerne un système de traitement de données et un procédé de distribution et d'extraction de données, le système de traitement de données comprenant : un module d'échange de communication, le module d'échange de communication étant configuré pour être accouplé à un terminal d'accès source et à un terminal d'accès cible, et pour recevoir du terminal d'accès source des premières données avec un premier format; un module de traitement de réception, le module de traitement et de réception étant configuré pour recevoir du module d'échange de communication les premières données avec le premier format, vérifier les premières données et décompresser et restaurer celles-ci pour obtenir les données de source, et stocker les données de source vérifiées, décompressées et restaurées dans des segments de données dans un cache de données en fonction du type de données de source; un module de traitement d'expédition, le module de traitement d'expédition convertissant des données à envoyer en des deuxièmes données avec un deuxième format, et envoyant celles-ci au terminal d'accès cible par l'intermédiaire du module d'échange de communication; et un module d'ordonnancement de tâches, le module d'ordonnancement de tâches étant agencé entre le module de traitement de réception et le module de traitement d'expédition, et étant configuré pour ordonnancer chaque segment de données des données à envoyer au module de traitement d'expédition.
PCT/CN2017/080093 2016-04-22 2017-04-11 Système et procédé de traitement de données WO2017181872A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610252966.3 2016-04-22
CN201610252966.3A CN105843966A (zh) 2016-04-22 2016-04-22 数据处理系统以及方法

Publications (1)

Publication Number Publication Date
WO2017181872A1 true WO2017181872A1 (fr) 2017-10-26

Family

ID=56588673

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/080093 WO2017181872A1 (fr) 2016-04-22 2017-04-11 Système et procédé de traitement de données

Country Status (2)

Country Link
CN (1) CN105843966A (fr)
WO (1) WO2017181872A1 (fr)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105843966A (zh) * 2016-04-22 2016-08-10 中国银联股份有限公司 数据处理系统以及方法
CN106775977B (zh) * 2016-12-09 2020-06-02 北京小米移动软件有限公司 任务调度方法、装置及系统
CN108243146B (zh) * 2016-12-23 2020-01-24 中科星图股份有限公司 一种信息提交方法
CN107368370A (zh) * 2017-07-04 2017-11-21 合肥市乐腾科技咨询有限公司 一种高效率数据处理系统
CN109558448B (zh) * 2018-10-10 2021-04-06 北京海数宝科技有限公司 数据处理方法、装置、计算机设备和存储介质
CN110572373A (zh) * 2019-08-20 2019-12-13 北京安盟信息技术股份有限公司 一种数据交换平台和文件数据流处理方法
CN111629026A (zh) * 2020-04-07 2020-09-04 青岛奥利普自动化控制系统有限公司 一种基于mes的数据交互方法和设备
CN112148782B (zh) * 2020-09-24 2023-01-20 建信金融科技有限责任公司 市场数据接入方法及装置
CN112925557B (zh) * 2021-01-28 2023-09-05 长沙市到家悠享网络科技有限公司 数据处理装置、方法及计算设备
CN114928576A (zh) * 2022-05-19 2022-08-19 广东电网有限责任公司 数据交换机、数据交换方法及计算机可读存储介质
CN115150466B (zh) * 2022-06-29 2023-08-15 北京百度网讯科技有限公司 一种数据分发的实现方法、装置、电子设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080215586A1 (en) * 2005-02-18 2008-09-04 International Business Machines Corporation Simulating Multi-User Activity While Maintaining Original Linear Request Order for Asynchronous Transactional Events
CN102541942A (zh) * 2010-12-31 2012-07-04 中国银联股份有限公司 一种数据批量转移系统及其方法
CN103944875A (zh) * 2014-02-21 2014-07-23 中国人民解放军装甲兵工程学院 数据交换方法及数据交换系统
CN104933173A (zh) * 2015-06-30 2015-09-23 北京京东尚科信息技术有限公司 一种用于异构多数据源的数据处理方法、装置和服务器
CN105843966A (zh) * 2016-04-22 2016-08-10 中国银联股份有限公司 数据处理系统以及方法

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102917020B (zh) * 2011-09-24 2016-02-17 国网电力科学研究院 一种基于数据包的移动终端与业务系统数据同步的方法
CN104809201B (zh) * 2015-04-24 2018-11-16 联动优势科技有限公司 一种数据库同步的方法和装置
CN104809202B (zh) * 2015-04-24 2019-01-18 联动优势科技有限公司 一种数据库同步的方法和装置
CN104809200B (zh) * 2015-04-24 2018-11-16 联动优势科技有限公司 一种数据库同步的方法和装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080215586A1 (en) * 2005-02-18 2008-09-04 International Business Machines Corporation Simulating Multi-User Activity While Maintaining Original Linear Request Order for Asynchronous Transactional Events
CN102541942A (zh) * 2010-12-31 2012-07-04 中国银联股份有限公司 一种数据批量转移系统及其方法
CN103944875A (zh) * 2014-02-21 2014-07-23 中国人民解放军装甲兵工程学院 数据交换方法及数据交换系统
CN104933173A (zh) * 2015-06-30 2015-09-23 北京京东尚科信息技术有限公司 一种用于异构多数据源的数据处理方法、装置和服务器
CN105843966A (zh) * 2016-04-22 2016-08-10 中国银联股份有限公司 数据处理系统以及方法

Also Published As

Publication number Publication date
CN105843966A (zh) 2016-08-10

Similar Documents

Publication Publication Date Title
WO2017181872A1 (fr) Système et procédé de traitement de données
US10999367B2 (en) Communication apparatus, control device, and communication method
US9009234B2 (en) Complex event processing system having multiple redundant event processing engines
JP5128111B2 (ja) メッセージに関連付けられた順序を保存するためのシステム、並びに、その方法及びコンピュータ・プログラム
CN114338651B (zh) 文件传输方法、装置、电子设备及可读存储介质
US10075549B2 (en) Optimizer module in high load client/server systems
US9367261B2 (en) Computer system, data management method and data management program
US10362131B1 (en) Fault tolerant message delivery
CN103971687A (zh) 一种语音识别系统中的负载均衡实现方法和装置
EP2995028B1 (fr) Récupération de n-uplets
EP3031172B1 (fr) Gestion de sources de données
CN112217847A (zh) 微服务平台及其实现方法、电子设备及存储介质
US11218371B2 (en) Method and apparatus for performing communication in internet of things
WO2012130162A1 (fr) Système et procédé de génération de fichier d'informations sur la base d'un traitement parallèle
WO2020232892A1 (fr) Procédé et appareil de gestion de clé pour chaîne de blocs, dispositif et support de stockage
CN116405547A (zh) 消息推送方法、装置及处理器、电子设备、存储介质
CN113014618B (zh) 消息处理方法、系统和电子设备
US20240020297A1 (en) Metrics and events infrastructure
CN108614820B (zh) 实现流式源数据解析的方法和装置
US11831711B2 (en) System and method for sending and receiving remote procedure calls
CN109492038B (zh) 一种基于微内核和异步队列的异构系统间数据发布系统
WO2024021280A1 (fr) Procédé et dispositif de transmission de données pour compagnie gazière industrielle, et serveur et système
CN113778786B (zh) 一种监控中间件
CN106664217B (zh) 候选问题网络实体的标识的方法、系统和介质
US10523588B2 (en) Technique for processing messages in a message-based communication scenario

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17785357

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 17785357

Country of ref document: EP

Kind code of ref document: A1