WO2020151209A1 - 一种基于多维数据变量的数据交互方法及平台 - Google Patents
一种基于多维数据变量的数据交互方法及平台 Download PDFInfo
- Publication number
- WO2020151209A1 WO2020151209A1 PCT/CN2019/097172 CN2019097172W WO2020151209A1 WO 2020151209 A1 WO2020151209 A1 WO 2020151209A1 CN 2019097172 W CN2019097172 W CN 2019097172W WO 2020151209 A1 WO2020151209 A1 WO 2020151209A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- interface
- legal
- dimensional
- fields
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/448—Execution paradigms, e.g. implementations of programming paradigms
Definitions
- the invention relates to the technical field of data interaction information, in particular to a data interaction method and platform based on multi-dimensional data variables.
- the present invention provides a data interaction method and platform based on multi-dimensional data variables, which can effectively filter junk data and realize efficient real-time data interaction between systems, providing efficient and stable data support for various data services .
- a data interaction platform based on multi-dimensional data variables is divided into three modules: 1, verification module 2, interface engine 3, configuration analysis engine.
- the verification module implements the filtering of junk data through the Bloom filter; the interface engine concurrently calls multiple data provider interfaces to interactively obtain multi-dimensional data variables, and can return the function of valid data release; the configuration
- the parsing engine has the function of automatically and privately customizing the parsing of the data returned by the interface engine.
- the verification module maps all the security fields provided by itself according to the parsed fields and data types, so as to filter out safe and reliable fields and avoid polluting field injection.
- the interface of a reasonable data provider is matched according to the field. If the interface of a specific data provider is not specified, an optimal matching strategy is formulated here. First, filter out all interfaces that meet the required fields. Second, filter out all the most stable and efficient interfaces. Finally, give priority to the interface that has the fastest access.
- the interface engine concurrently accesses each data provider interface according to the interface type, so as to perform interactive acquisition of multi-dimensional data variables.
- Concurrency in the operating system, means that several programs in a period of time are between started and run to completion, and these programs are all running on the same processor, but there is only one at any one time. The program runs on the processor, but because the current CPU processing performance is efficient, the intuitive feeling of processing tasks will be performed at the same time. For large-scale or slow single access, the performance of concurrent processing will far exceed the performance of linear processing.
- parsing engine is configured to parse and encapsulate the data returned by the interface engine.
- Multiple sets of parsing and packaging solutions can be configured at the beginning, and private methods can also be abstracted for users to customize parsing and packaging. The user only needs to pass in the data structure configuration when accessing the request, and the configuration analysis engine will parse the packaged data according to the specified data structure configuration.
- the present invention also provides a data interaction method based on multi-dimensional data variables, which includes the following steps:
- the data requester requests access
- Concurrently request provider interface data that is, make a multi-dimensional concurrent request to each data provider interface, and execute (6);
- step (5) Based on the data result obtained in step (5), analyze and encapsulate the data, and execute (7);
- the field legality check is to pass the mapping check of all the legal fields given and automatically screen.
- the matching provider interface is a field mapping interface based on the legal field interface data dictionary provided by the data provider. There may be a field that maps to multiple interfaces, and a priority strategy is adopted to Choice, efficiency first, stability first.
- the access to the provider interface is accessed by adopting a concurrent strategy, and concurrently calls the multi-dimensional interface provided by multiple providers.
- parsing the package result data includes two strategies: customizable package and system automatic package.
- the data variables are filtered based on the Bloom filter, which can filter out safe and reliable fields to avoid pollution field injection; match the interface of the reasonable data provider according to the field to formulate the optimal matching strategy; access each item concurrently according to the interface type
- the data provider interface is used for interactive acquisition of multi-dimensional data variables. For large-scale or low-efficiency single access, the performance of concurrent processing will far exceed the performance of linear processing. Therefore, the present invention can realize the functions of effectively filtering junk data and realizing efficient real-time data interaction between various systems, and provide efficient and stable data support for various data services.
- Figure 1 is an architecture diagram of a data interaction platform based on multi-dimensional data variables provided by the present invention
- Fig. 2 is a flowchart of a data interaction method based on multi-dimensional data variables provided by the present invention.
- a data interaction platform (interaction system) based on multi-dimensional data variables is divided into three modules: 1, verification module 2, interface engine 3, configuration analysis engine.
- the data requester initiates a data request, performs security verification on the required data fields, and uses the Bloom filter algorithm to filter junk data.
- Bloom Filter is a huge bit array and several hash functions.
- the length of the bit array is m
- the number of hash functions is k.
- the specific operation process Assume that there are 3 elements ⁇ x,y,z ⁇ in the set, and the number of hash functions is 3.
- the elements are mapped through 3 hash functions in turn, each mapping will generate a hash value, this value corresponds to a point on the bit array, and then the position corresponding to the bit array is marked as 1 .
- the same method maps W to 3 points on the bit array by hash.
- the verification module uses Bloom filters to filter junk data, and maps all the security fields provided by itself according to the parsed fields and data types, so as to filter out safe and reliable fields and avoid polluting field injection.
- an optimal matching strategy is formulated here. First, filter out all interfaces that meet the required fields. Second, filter out all the most stable and efficient interfaces. Finally, give priority to the interface that has the fastest access.
- the interface engine concurrently accesses the interfaces of various data providers according to the interface type, so as to obtain multi-dimensional data variables interactively.
- Concurrency in the operating system, means that several programs in a period of time are between started and run to completion, and these programs are all running on the same processor, but there is only one at any time The program runs on the processor, but because the current CPU processing performance is efficient, the intuitive feeling of processing tasks will be performed at the same time. For large-scale or slow single access, the performance of concurrent processing will far exceed the performance of linear processing.
- parsing engine Configure the parsing engine to parse and encapsulate the data returned by the interface engine.
- Multiple sets of parsing and packaging solutions can be configured at the beginning, and private methods can also be abstracted for users to customize parsing and packaging. The user only needs to pass in the data structure configuration when accessing the request, and the configuration analysis engine will parse the packaged data according to the specified data structure configuration.
- a data interaction method based on multi-dimensional data variables includes the following steps:
- the interface provided by the specific data provider is mapped according to the field matching, specifically through the mapping and matching with the field interface data dictionary of the legal data provider
- the data obtained from various data providers is analyzed and packaged, and the data is analyzed and packaged through multiple packaged analysis and package methods.
- this step provides a privatized custom encapsulation method. If the data requester carries the encapsulation data structure parameter when accessing the request, the data can be encapsulated according to the data structure at this time. Specifically configure data through the key-value mapping data structure.
- Verification result final verification of the final packaging result. If the verification fails, the abnormal result information is returned to the data requester.
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
- Computer And Data Communications (AREA)
Abstract
Description
Claims (6)
- 一种基于多维数据变量的数据交互平台,其特征在于,包括:校验模块、接口引擎和配置解析引擎;其中所述校验模块,通过布隆过滤器实现对垃圾数据进行过滤,筛选出合法字段;所述接口引擎,并发调用多个数据提供方接口以交互获取多维数据变量,并能返回有效数据放的功能;所述配置解析引擎,对所述接口引擎返回的数据进行解析封装,即自动以及私有化定制解析。
- 一种基于多维数据变量的数据交互方法,其特征在于,该方法包括以下步骤:(1)数据请求方请求接入;(2)利用布隆过滤器对接入的数据进行过滤,以过滤掉垃圾数据变量,解析请求参数,获取字段及其数据类型;(3)通过与合法字段列表进行匹配,从而对(2)获取的所有字段进行合法校验,如果合法,则筛选出合法字段,然后执行(4);如果不合法,则执行(8);(4)将(3)校验的合法字段,匹配提供商接口,如果与合法接口列表相匹配,执行(5);否则执行(8);(5)并发请求提供商接口数据,即向各个数据提供商接口提出多维度并发请求, 执行(6);(6)基于所述步骤(5)获取的数据结果,对数据进行解析封装,执行(7);(7)对结果进行合理性校验,校验合理,则返回结果给数据请求方,校验不合理执行(8);(8)将包含异常原因的结果返回给数据请求方。
- 根据权利要求2所述的基于多维数据变量的数据交互方法,其特征在于:所述步骤(3)中,字段合法性校验是通过给定所有合法字段映射校验,自动筛选。
- 根据权利要求2所述的基于多维数据变量的数据交互方法,其特征在于:所述步骤(4)中,匹配提供商接口是根据数据提供商提供的合法字段接口数据字典来为字段映射接口,其中可能存在一个字段会映射到多个接口,是采取优先策略来选择,效率优先、稳定性优先。
- 根据权利要求2所述的基于多维数据变量的数据交互方法,其特征在于:所述步骤(5)中,访问提供商接口是通过采取并发策略访问的,并发调用多个提供商提供的多维度接口。
- 根据权利要求2所述的基于多维数据变量的数据交互方法,其特征在于:所述步骤(6)中,解析封装结果数据包括可定制化封装和系统自动封装两个策略。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910062225.2A CN109783163A (zh) | 2019-01-23 | 2019-01-23 | 一种基于多维数据变量的数据交互方法及平台 |
CN201910062225.2 | 2019-01-23 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020151209A1 true WO2020151209A1 (zh) | 2020-07-30 |
Family
ID=66501052
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2019/097172 WO2020151209A1 (zh) | 2019-01-23 | 2019-07-23 | 一种基于多维数据变量的数据交互方法及平台 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN109783163A (zh) |
WO (1) | WO2020151209A1 (zh) |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109783163A (zh) * | 2019-01-23 | 2019-05-21 | 集奥聚合(北京)人工智能科技有限公司 | 一种基于多维数据变量的数据交互方法及平台 |
CN110647571A (zh) * | 2019-09-24 | 2020-01-03 | 集奥聚合(北京)人工智能科技有限公司 | 一种基于多维数据的交互方法及平台 |
CN110727670B (zh) * | 2019-10-11 | 2022-08-09 | 北京小向创新人工智能科技有限公司 | 基于流程图的数据结构预测传递及自动化数据处理方法 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104601557A (zh) * | 2014-12-29 | 2015-05-06 | 广东顺德中山大学卡内基梅隆大学国际联合研究院 | 一种基于软件定义网络的恶意网站防护方法及系统 |
US20160004605A1 (en) * | 2014-07-01 | 2016-01-07 | Commvault Systems, Inc. | Lightweight data reconstruction based on backup data |
CN108733724A (zh) * | 2017-04-24 | 2018-11-02 | 北京京东尚科信息技术有限公司 | 一种跨异构数据源实时连接方法及装置 |
CN109783163A (zh) * | 2019-01-23 | 2019-05-21 | 集奥聚合(北京)人工智能科技有限公司 | 一种基于多维数据变量的数据交互方法及平台 |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2005011042A (ja) * | 2003-06-18 | 2005-01-13 | Shinfuoomu:Kk | データ検索方法、データ検索装置、データ検索プログラムおよびコンピュータで読み取り可能な記録媒体 |
CN102571715B (zh) * | 2010-12-27 | 2015-07-29 | 腾讯科技(深圳)有限公司 | 多维数据查询方法及系统 |
CN102682037B (zh) * | 2011-03-18 | 2016-09-28 | 阿里巴巴集团控股有限公司 | 一种数据获取方法、系统及装置 |
CN106503217B (zh) * | 2016-11-03 | 2019-10-22 | 用友网络科技股份有限公司 | 基于业务驱动的数据交互装置及方法 |
-
2019
- 2019-01-23 CN CN201910062225.2A patent/CN109783163A/zh active Pending
- 2019-07-23 WO PCT/CN2019/097172 patent/WO2020151209A1/zh active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20160004605A1 (en) * | 2014-07-01 | 2016-01-07 | Commvault Systems, Inc. | Lightweight data reconstruction based on backup data |
CN104601557A (zh) * | 2014-12-29 | 2015-05-06 | 广东顺德中山大学卡内基梅隆大学国际联合研究院 | 一种基于软件定义网络的恶意网站防护方法及系统 |
CN108733724A (zh) * | 2017-04-24 | 2018-11-02 | 北京京东尚科信息技术有限公司 | 一种跨异构数据源实时连接方法及装置 |
CN109783163A (zh) * | 2019-01-23 | 2019-05-21 | 集奥聚合(北京)人工智能科技有限公司 | 一种基于多维数据变量的数据交互方法及平台 |
Also Published As
Publication number | Publication date |
---|---|
CN109783163A (zh) | 2019-05-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11481396B2 (en) | Executing untrusted commands from a distributed execution model | |
US20220171759A1 (en) | Detecting schema incompatibilities for generating views at target data stores | |
US10698897B2 (en) | Executing a distributed execution model with untrusted commands | |
KR101365832B1 (ko) | 데이터 액세스 계층 클래스 생성기 | |
US9165034B2 (en) | Heterogeneous data source management | |
WO2020151209A1 (zh) | 一种基于多维数据变量的数据交互方法及平台 | |
US10120928B2 (en) | Method and system for clustering event messages and managing event-message clusters | |
US10997062B1 (en) | System and method for implementing an automated regression testing module | |
WO2003048994A2 (en) | System and method for actively managing an enterprise of configurable components | |
WO2008154032A1 (en) | Secure hosted databases | |
US10983815B1 (en) | System and method for implementing a generic parser module | |
US10802801B1 (en) | Grand unified processor | |
WO2021042849A1 (zh) | 一种云平台、业务处理方法、命令接口及计算机设备 | |
CN115309566B (zh) | 一种服务接口的动态管理方法及系统 | |
US7428756B2 (en) | Access control over dynamic intellectual capital content | |
CN109669976A (zh) | 基于etl的数据服务方法及设备 | |
EP3462330A1 (en) | Fault tolerant adapter system to consume database as a service | |
CN111966692A (zh) | 针对数据仓库的数据处理方法、介质、装置和计算设备 | |
US20240143576A1 (en) | Registering additional type systems using a hub data model for data processing | |
CN103220167A (zh) | 一种分布式监控系统及其数据存储方法 | |
US20230196199A1 (en) | Querying databases with machine learning model references | |
US10951540B1 (en) | Capture and execution of provider network tasks | |
CN111048164A (zh) | 一种医学大数据长期保存系统 | |
CN112035466B (zh) | 一种区块链查询外置索引开发框架 | |
CN112187509A (zh) | 多架构云平台执行日志管理方法、系统、终端及存储介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19911135 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 12.11.2021) |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 12.11.2021) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19911135 Country of ref document: EP Kind code of ref document: A1 |