WO2012088769A1 - 数据过滤系统和方法 - Google Patents
数据过滤系统和方法 Download PDFInfo
- Publication number
- WO2012088769A1 WO2012088769A1 PCT/CN2011/002225 CN2011002225W WO2012088769A1 WO 2012088769 A1 WO2012088769 A1 WO 2012088769A1 CN 2011002225 W CN2011002225 W CN 2011002225W WO 2012088769 A1 WO2012088769 A1 WO 2012088769A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- filtering
- filter
- module
- item
- data
- Prior art date
Links
- 238000001914 filtration Methods 0.000 title claims abstract description 200
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000004458 analytical method Methods 0.000 claims abstract description 7
- 238000000605 extraction Methods 0.000 claims abstract description 5
- 230000014759 maintenance of location Effects 0.000 claims description 10
- 238000005457 optimization Methods 0.000 claims 1
- 238000005215 recombination Methods 0.000 abstract description 2
- 230000006798 recombination Effects 0.000 abstract description 2
- 230000008859 change Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000008521 reorganization Effects 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/9035—Filtering based on additional data, e.g. user or group profiles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/90335—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
Definitions
- the present invention relates to the field of information processing technologies, and in particular, to a data filtering system and method based on module free recombination.
- the first filtering method is completely static, and is filtered based on the established filtering conditions in the program instructions during data filtering;
- the second is that the filtering method is full-open dynamic filtering, for example, one filtering condition is
- the filtering component is dynamically set to be smaller than an integer value
- the third method is that the filtering method is semi-open dynamic filtering, that is, the external input of the filtering parameter is defined by the template prototype, and the input of the filtering parameter value is input, and the filtering parameter is often inconsistent with the parameter of the data template prototype, that is, the filtering parameter is ensured to be consistent.
- the logical attribute of the data item being filtered See, for example, the data filtering method disclosed in Chinese Patent Application No. 200910249725.3.
- Existing data typically has multiple attribute characteristics.
- the bank card transaction data is exemplified, which has a transaction amount attribute, a transaction type attribute, a card issuer attribute, and the like. Users often need to filter data for different attribute characteristics under different conditions as needed.
- the filtering parameter selection is limited by the template prototype method, and the filtering method for receiving the filtering parameter value dynamic input system is the most common, but the filtering template is usually customized in advance and closely coupled with the application system itself. , can't be based on reality
- an object of the present invention is to provide a data filtering method and system capable of dynamically generating filtering conditions to implement filtering analysis of data.
- the invention provides a data filtering system, comprising:
- Extracting unit configured to extract attribute features of the source data and send to the selecting unit; selecting a unit, selecting at least one attribute feature according to which data filtering is performed, and sending the selected attribute feature to the filtering condition generating unit;
- the filter condition generating unit generates at least one filtering module that is in one-to-one correspondence with the at least one attribute feature according to the selected at least one attribute feature, and combines the at least one filtering module to generate a filtering condition;
- the data filtering analysis unit filters the source data according to the filtering conditions to generate target data.
- the filtering condition generating unit comprises:
- the filter item generating module receives the selected at least one attribute feature, generates at least one filter item, and sends the filter element to the filter module generation module, where each filter item includes a filter item name, a filtering method, and a corresponding attribute feature. Filter fields, match methods, and filter parameter values;
- the filtering module generates a module, generates at least one filtering module and sends it to the control module Block, wherein the filtering module has a one-to-one correspondence with the filtering item;
- a control module combining the at least one filtering module to generate a filtering condition.
- the filtering condition generating unit includes:
- the filter item generating module receives the selected at least one attribute feature and generates at least one filter item and sends it to the filter module generating module, where each filter item includes a filter item name, a filter item priority, a filtering method, and a selected one.
- a filtering module generating module, generating at least one filtering module and sending the same to the control module, wherein the filtering module has a one-to-one correspondence with the filtering item;
- control module combines the at least one filtering module according to the filter item priority to generate a filtering condition.
- the filtering method includes filtering retention parameters and filtering rejection parameters, wherein
- Filtering the retention parameter means retaining data that matches the filter item
- Filtering the cull parameter means culling data that matches the filter item.
- the matching manner includes whole word matching, head matching, tail matching, fuzzy matching or interval matching.
- the system further includes an input device coupled to the selection unit and the filter condition generating unit for inputting the required information.
- the system further includes a display unit connected to the extraction unit and the filter condition generation unit for displaying information from the extraction unit and the filter condition generation module.
- the invention also provides a data filtering method, comprising the following steps:
- A extracts the attribute characteristics of the source data
- A3. Generate a filtering module according to the selected attribute feature, and combine the filtering module to generate a filtering condition.
- A4. Filter the source data according to the filtering condition to generate target data.
- step A3 further includes: generating at least one filter item, wherein each filter item includes a filter item name, a filtering method, a filtering domain corresponding to the selected attribute feature, a matching manner, and Filtering the parameter values; generating at least one filtering module, wherein the filtering module is in one-to-one correspondence with the filtering items;
- At least the one of the filter modules is combined to generate the filter condition.
- step A3 further includes: generating at least one filter item, wherein each filter item includes a filter item name, a filter item priority, a filtering mode, and a selected attribute feature. Filter fields, matching methods, and filter parameter values;
- the at least one filtering module is combined according to the filter item priority to generate a filtering condition.
- the filtering method includes filtering retention parameters and filtering rejection parameters, wherein
- Filtering the retention parameter means retaining data that matches the filter item
- Filtering the culling parameter means culling the data that matches the filter item
- the matching manner includes a full word matching, a head matching, a tail matching, a fuzzy matching or an interval matching.
- the method of the prototype of the data template limits the flexibility of the filter item to a certain extent, and the method can automatically decompose the data according to the data by using the filtering mechanism and the decoupling of the application system, the flexible definition of the filtering item, and the filtering method of the filtering module freely reorganizing.
- the characteristics of the business and the change arrangement of the filter logic and the filtering module of the business change adjustment filter item, and the filtering analysis of the data are flexibly performed. This ensures that when data characteristics change, it can be freely reorganized to efficiently adapt to data characteristics. And changes in stress.
- FIG. 1 is a schematic structural view of a data filtering system according to the present invention.
- FIG. 2 is a schematic routine diagram of a data filtering method in accordance with the present invention. detailed description
- the data filtering system includes an extracting unit 1, a selecting unit 2, a filter condition generating unit 3, a data filtering analyzing unit 4, a display unit 5, and an input unit 6.
- the filter generation unit 3 includes a filter item generation module 30, a filter module generation module 31, and a control module 32.
- the extracting unit 1 is used to extract the attribute features of the source data and send them to the selecting unit 2 and the display unit 5.
- the selection unit 2 is connected to the input unit 5, so that the user selects at least one attribute feature for which data filtering is performed by the input unit 5 and transmits the selected attribute feature to the filter item generation module 30 in the filter condition generation unit 3.
- the filter item generating module 30 receives the selected at least one attribute feature, generates a filter item 1, a filter item 2, ... a filter item n, where n is greater than or equal to 1, and sends it to the filter module generation module 31, wherein each Filter items include filter item name, filtering method, and The filter domain, matching method, and filter parameter value corresponding to the selected attribute feature.
- the filter item may further include a filter item priority. The user can input the content of each part in the filtering item through the input unit 6.
- Filtering methods include filtering retention parameters and filtering rejection parameters, where
- Filtering the retention parameter means retaining data that matches the filter item
- Filtering the cull parameter means culling data that matches the filter item.
- Matching methods include whole word matching, head matching, tail matching, fuzzy matching or interval matching.
- the full word matching method can be filtered in an integer or character mode.
- Head matching can perform filtering from any number of bits starting from the first digit of the matched value.
- Fuzzy matching can accomplish a match similar to the general "like" way, that is, data filtering that is similar in value or contains the value of the filter parameter.
- Interval matching can complete integer matching like ">" " ⁇ " mode, or character type belongs to relationship matching.
- This matching method covers all possible filtering parameter logics without being limited by the limitations of fixed filtering parameters in the template mode.
- the foregoing matching manner is merely an exemplary description, and those skilled in the art can add new matching manners according to the actual without departing from the scope of protection of the present invention.
- the filter item generation module 31 also sends the filter item to the display unit 5, which displays the filter item in the form of a page.
- the filtering module generating module 31 generates a filtering module 1, a filtering module 2, ... a filtering module n and sends it to the control module, wherein the filtering module is paired with the filtering item.
- the filtering module can, for example, be a code instruction module that is generated based on the filter item and loaded into the control module for filtering.
- Control module 32 combines the at least one filter module to generate filter conditions.
- the at least one filtering module is combined according to the filter item priority to generate a filter condition.
- the data filtering analysis unit 4 filters the source data according to the filtering conditions to generate target data.
- 2 is a schematic routine diagram of a data filtering method in accordance with the present invention. As shown, the method includes the steps of:
- A3. Generate a filtering module according to the selected attribute feature, and combine the filtering module to generate a filtering condition;
- step A3 further comprises:
- each filter item includes a filter item name, a filter mode, a filter field corresponding to the selected attribute feature, a matching mode, and a filter parameter value; generating at least one filter module, where the filter module and the The filter items correspond one-to-one;
- At least the one of the filter modules is combined to generate the filter element.
- step A3 further includes:
- each filter item includes a filter item name, a filter item priority, a filtering mode, a filter field, a matching mode, and a filter parameter value corresponding to the selected attribute feature;
- the at least one filtering module is combined according to the filter item priority to generate a filtering condition.
- the filtering module can be combined according to the user's requirements, and the rearrangement and sequence combination of the digital filtering can be realized to obtain better system performance.
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Description
Claims
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP11853360.3A EP2660730A4 (en) | 2010-12-31 | 2011-12-30 | Data filtering system and method |
US13/977,650 US20140019463A1 (en) | 2010-12-31 | 2011-12-30 | Data filtering system and method |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201010619385.1 | 2010-12-31 | ||
CN201010619385.1A CN102567413B (zh) | 2010-12-31 | 2010-12-31 | 数据过滤系统和方法 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2012088769A1 true WO2012088769A1 (zh) | 2012-07-05 |
Family
ID=46382253
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2011/002225 WO2012088769A1 (zh) | 2010-12-31 | 2011-12-30 | 数据过滤系统和方法 |
Country Status (4)
Country | Link |
---|---|
US (1) | US20140019463A1 (zh) |
EP (1) | EP2660730A4 (zh) |
CN (1) | CN102567413B (zh) |
WO (1) | WO2012088769A1 (zh) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102930053A (zh) * | 2012-11-19 | 2013-02-13 | 西北大学 | 一种多维度智能的资源筛选方法 |
CN105245601A (zh) * | 2015-10-15 | 2016-01-13 | 桂林电子科技大学 | 一种数据过滤方法及数据过滤系统 |
CN110991813A (zh) * | 2019-11-07 | 2020-04-10 | 上海数禾信息科技有限公司 | 用于风控业务的数据处理方法及装置 |
Families Citing this family (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104484351B (zh) * | 2014-11-28 | 2018-07-20 | 上海百事通信息技术股份有限公司 | 大数据量号码过滤装置及方法 |
CN105930258B (zh) * | 2015-11-13 | 2019-04-26 | 中国银联股份有限公司 | 一种参数过滤的方法及装置 |
CN106534184B (zh) * | 2016-12-12 | 2019-08-06 | 南京富岛信息工程有限公司 | 一种基于深度数据过滤的控制系统信息安全网关 |
CN106649831B (zh) * | 2016-12-29 | 2020-09-04 | 北京奇艺世纪科技有限公司 | 一种数据过滤方法及装置 |
CN107391532B (zh) * | 2017-04-14 | 2020-08-04 | 阿里巴巴集团控股有限公司 | 数据过滤的方法和装置 |
CN108932300B (zh) * | 2018-06-06 | 2022-05-27 | 成都深思科技有限公司 | 一种无限迭代的过滤分析方法、设备及存储介质 |
CN109063199B (zh) * | 2018-09-11 | 2022-10-25 | 优视科技有限公司 | 资源过滤方法及其装置、电子设备、计算机可读介质 |
CN113837154B (zh) * | 2021-11-25 | 2022-03-25 | 之江实验室 | 一种基于多任务辅助的开放集过滤系统及方法 |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101599087A (zh) * | 2009-07-02 | 2009-12-09 | 金蝶软件(中国)有限公司 | 数据查询方法和装置 |
CN101719157A (zh) * | 2009-12-01 | 2010-06-02 | 山东中创软件工程股份有限公司 | 数据过滤方法、系统和用于该系统的数据处理装置 |
CN101930433A (zh) * | 2009-06-26 | 2010-12-29 | 上海汉光知识产权数据科技有限公司 | 专利查找系统及方法 |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020091685A1 (en) * | 2001-01-10 | 2002-07-11 | Letmeknow Technology | System and method for filtering data events |
US7590310B2 (en) * | 2004-05-05 | 2009-09-15 | Facet Technology Corp. | Methods and apparatus for automated true object-based image analysis and retrieval |
CN101071416A (zh) * | 2007-07-02 | 2007-11-14 | 华为技术有限公司 | 报表模板生成系统、开发工具、开发方法和报表客户端 |
CN101334794A (zh) * | 2008-08-07 | 2008-12-31 | 金蝶软件(中国)有限公司 | 一种生成单据过滤条件的方法及装置 |
-
2010
- 2010-12-31 CN CN201010619385.1A patent/CN102567413B/zh active Active
-
2011
- 2011-12-30 WO PCT/CN2011/002225 patent/WO2012088769A1/zh active Application Filing
- 2011-12-30 US US13/977,650 patent/US20140019463A1/en not_active Abandoned
- 2011-12-30 EP EP11853360.3A patent/EP2660730A4/en not_active Ceased
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101930433A (zh) * | 2009-06-26 | 2010-12-29 | 上海汉光知识产权数据科技有限公司 | 专利查找系统及方法 |
CN101599087A (zh) * | 2009-07-02 | 2009-12-09 | 金蝶软件(中国)有限公司 | 数据查询方法和装置 |
CN101719157A (zh) * | 2009-12-01 | 2010-06-02 | 山东中创软件工程股份有限公司 | 数据过滤方法、系统和用于该系统的数据处理装置 |
Non-Patent Citations (2)
Title |
---|
See also references of EP2660730A4 |
ZHAO LIANJUN ET AL.: "A data filtrating method based on data property description", COMPUTER APPLICATIONS AND SOFTWARE, vol. 22, no. 1, January 2005 (2005-01-01), pages 38 - 39, 136, XP008171035 * |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102930053A (zh) * | 2012-11-19 | 2013-02-13 | 西北大学 | 一种多维度智能的资源筛选方法 |
CN102930053B (zh) * | 2012-11-19 | 2014-08-13 | 西北大学 | 一种多维度智能的资源筛选方法 |
CN105245601A (zh) * | 2015-10-15 | 2016-01-13 | 桂林电子科技大学 | 一种数据过滤方法及数据过滤系统 |
CN110991813A (zh) * | 2019-11-07 | 2020-04-10 | 上海数禾信息科技有限公司 | 用于风控业务的数据处理方法及装置 |
Also Published As
Publication number | Publication date |
---|---|
US20140019463A1 (en) | 2014-01-16 |
CN102567413B (zh) | 2015-06-17 |
EP2660730A4 (en) | 2017-03-29 |
EP2660730A1 (en) | 2013-11-06 |
CN102567413A (zh) | 2012-07-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2012088769A1 (zh) | 数据过滤系统和方法 | |
US10915664B2 (en) | Data masking systems and methods | |
US20210109934A1 (en) | Configurable and incremental database migration framework for heterogeneous databases | |
JP2024045394A (ja) | ブロックチェーン実装スクリプトベースのバイト解釈のための方法とシステム | |
CN110032369A (zh) | 一种代码自动生成方法、装置及介质 | |
CN107015804B (zh) | 一种通过配置api快速开发项目的方法及系统 | |
CN106341444B (zh) | 数据访问方法和装置 | |
CN102591654B (zh) | 一种基于ssh架构的代码生成器及方法 | |
CN107273122A (zh) | 基于解耦机制的可迭代组建业务系统的方法及其终端 | |
US20170236130A1 (en) | Emulating Manual System of Filing Using Electronic Document and Electronic File | |
JP5939123B2 (ja) | 実行制御プログラム、実行制御方法および情報処理装置 | |
CN104317587A (zh) | 面向安卓移动设备的程序自动生成方法 | |
CN110389832A (zh) | ZooKeeper容器集群部署方法、系统、设备及存储介质 | |
WO2013029399A1 (zh) | 代码生成方法及系统 | |
CN109062568A (zh) | 一种项目生成方法及工具 | |
WO2016060552A1 (en) | System generator module for electronic document and electronic file | |
CN106528896B (zh) | 一种数据库优化方法和装置 | |
CN110321337A (zh) | 一种实现数据库动态切换的方法及计算机可读存储介质 | |
CN104346378B (zh) | 一种实现复杂数据处理的方法、装置及系统 | |
CN106843818A (zh) | 一种业务模型生成方法及装置 | |
CN105302556A (zh) | 实现计算的方法和系统以及服务器装置 | |
JP2018018323A (ja) | 分散処理管理方法、分散処理管理プログラム、および分散処理管理装置 | |
CN113126991A (zh) | 业务数据处理方法及装置 | |
CN109271448A (zh) | 基于数据库为平台的数据同步系统及方法 | |
CN109101229A (zh) | 一种面向对象的代码自动生成方法、装置及设备 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 11853360 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2011853360 Country of ref document: EP |
|
ENP | Entry into the national phase |
Ref document number: 2013555723 Country of ref document: JP Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 13977650 Country of ref document: US |
|
NENP | Non-entry into the national phase |
Ref country code: JP |