CN107766538A - Data filtering processing module and synchronous, asynchronous filter method based on java - Google Patents
Data filtering processing module and synchronous, asynchronous filter method based on java Download PDFInfo
- Publication number
- CN107766538A CN107766538A CN201711037833.5A CN201711037833A CN107766538A CN 107766538 A CN107766538 A CN 107766538A CN 201711037833 A CN201711037833 A CN 201711037833A CN 107766538 A CN107766538 A CN 107766538A
- Authority
- CN
- China
- Prior art keywords
- data
- unit
- filter
- processing
- processor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24561—Intermediate data storage techniques for performance improvement
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9535—Search customisation based on user profiles and personalisation
Abstract
The present invention relates to a kind of data filtering processing module based on java, including data receipt unit, serialization unit, message queue unit, unserializing unit and data processing unit, data processing unit includes some filters and processor to being equipped with；Using the either synchronously or asynchronously filter method of aforementioned data filter processing module, filter and processor, the tupe of data receipt unit, serialization unit and unserializing unit are initialized；Data receipt unit is sent directly to data processing unit or by serialization unit, message queue unit, unserializing unit until data processing unit, data receipt unit return after data are filtered and handled after receiving data.The present invention only can be embedded in java codes in the operation of reality with jar bags, it is not necessary to extra server or large-scale engine, be absorbed in data filtering processing, lightweight, the efficiency high of data filtering processing.
Description
Technical field
The present invention relates to the technical field of electric Digital data processing, more particularly to a kind of lightweight and distributed it is based on
Java data filtering processing module and synchronous, asynchronous filter method.
Background technology
With the development of information technology, data turn into the core asset of many enterprises, the particularly enterprise such as finance, internet
Industry, each enterprise store substantial amounts of data, if it is desired to producing value using data, then must have the interaction of data, transmit.
Requirement or standard of each company for data are different, for same part data, the focus of each company
Also it is different, meanwhile, all more or less there is flaw in many data, such data are values that is not interactive or transmitting, and
The stability of influence system, Consumer's Experience is influenceed, only treated data just have value.
In the prior art, the service of heavyweight is belonged to for the engine that data processing uses, emphasis is all huge
Data analysis aspect, the data filtering being not embedded in java codes and processing product, and existing data processing uses
Engine be all special and independent service, be not suitable in web projects embedded use, also product provides data inspection in addition
The function of looking into, but processing business is not provided.
The content of the invention
In order to solve problems of the prior art, the present invention is provided at a kind of data filtering based on java of optimization
Module and synchronous, asynchronous filter method, lightweight and distribution are managed, can synchronously also can the asynchronous filtering and processing for carrying out data.
The technical solution adopted in the present invention is a kind of data filtering processing module based on java, the data filtering
Processing module includes being used for receiving user and transmits the data receipt unit of data after data and filtration treatment, for by data receiver
The serialization unit for the Data Serialization processing that unit receives, for the data object after temporary serializing and theme is supported to subscribe to
Message queue unit with user grouping function, subscribed to according to theme for the data after serializing is handled and user grouping
Difference carries out the unserializing unit of unserializing processing and is handled or be used for for the data after handling unserializing
The data processing unit for the data that direct processing data receiving unit receives；The data processing unit includes some to coordinating
The filter and processor of setting.
Preferably, the filter includes character string filter, date filter and digital filter.
Preferably, the processor includes being used to delete field, deletes object and replace the string processor of field, day
Phase processor and digital processing unit, the string processor, date processor and digital processing unit and character string filter, day
Phase filter and digital filter fits are set.
Preferably, the character string filter includes string length filter, string encoding filter, formulates character
Judge filter, judge air filter and character string custom filter.
Preferably, the date filter includes date range filter, special time filter, date precision filter
With date custom filter.
Preferably, the date custom filter be for add and subtract the time, month, week, when, minute, second, integrate it is all, whole
Close the filter of the moon, integration year or form conversion.
Preferably, the digital filter includes digital scope filter, numeric type filter, numerical accuracy filter
With digital custom filter.
Preferably, the digital processing unit coordinated with the digital filter is for taking negative, taking absolute value, evolution fortune
The processor of calculation, power operation or exponent arithmetic.
A kind of synchronous filtering method of the data filtering processing module based on java described in use, methods described include with
Lower step：
Step 1.1：Initialize some filters and processor to being equipped with；Initialization data receiving unit is synchronization process
Pattern；
Step 1.2：Data receipt unit is synchronously sent to data processing unit after data are received；
Step 1.3：Data processing unit calls some filters and processor to being equipped with after data are received；
Step 1.4：Filter and processor are filtered and handled to data, returned data receiving unit；
Step 1.5：Data after filtering and processing are back to user by data receipt unit as return value.
A kind of asynchronous filter method of the data filtering processing module based on java described in use, methods described include with
Lower step：
Step 2.1：Initialize some filters and processor to being equipped with；Initialization data receiving unit is asynchronous process
Pattern；Initialization sequence unit and unserializing unit；
Step 2.2：After data receipt unit receives data, serializing processing is carried out to data using serialization unit；
Step 2.3：Data object after serializing is put into message queue unit by data receipt unit；
Step 2.4：Data processing unit obtains data object at message queue unit, and using unserializing unit to data
Object is reduced, and gradually data are filtered and handled by some filters to being equipped with and processor；
Step 2.5：The data filtered and processing is completed are back to data receipt unit by message queue unit；
Step 2.6：Data after processing are back to user by data receipt unit as return value.
The invention provides a kind of data filtering processing module of optimization based on java and synchronous, asynchronous filter method,
By set data receipt unit be used for receive user data and confirm either synchronously or asynchronously tupe and receive filtration treatment after
Data, set serialization unit be used for serialize handle asynchronous filtering when reception data, set message queue unit keep in
Data and the subscription of offer theme and user grouping after serializing, set unserializing unit to carry out unserializing to data object
Processing, data processing unit is set to handle or for direct processing data receiving unit the data after unserializing processing
The data of reception, data processing unit after receiving the data, using some filters and processing to being equipped with set in it
Device carries out filtration treatment to data；By for the default of data receipt unit, confirming either synchronously or asynchronously to carry out data filtering
Data are docked and transmitted from suitable unit by processing operation.The present invention can be only with jar bags in the operation of reality
Java codes can be embedded in, it is not necessary to extra server or large-scale engine, be absorbed in data filtering processing, lightweight, data
The efficiency high of filtration treatment.
Brief description of the drawings
Fig. 1 is the structural representation of the data filtering processing module of the present invention；Wherein, solid arrow is shown in the present invention
The flow chart of synchronous filtering method, dotted arrow show the flow chart of asynchronous filter method in the present invention.
Embodiment
The present invention is described in further detail with reference to embodiment, but protection scope of the present invention is not limited to
This.
The present invention relates to a kind of data filtering processing module based on java, the data filtering processing module includes being used for
The data receipt unit of data, the data sequence for data receipt unit to be received after reception user transmission data and filtration treatment
The serialization unit of rowization processing, for data object after temporary serializing and support that theme is subscribed to and user grouping function
Message queue unit, subscribed to for the data after serializing is handled according to theme and the different of user grouping carry out unserializings
The unserializing unit of processing and handled for the data after handling unserializing or for direct processing data receive
The data processing unit for the data that unit receives；The data processing unit includes some filters to being equipped with and place
Manage device.
In the present invention, by for the default of data receipt unit, confirming either synchronously or asynchronously to carry out data filtering processing
Module, data are docked and transmitted from suitable unit.
In the present invention, data receipt unit is mainly used in receiving user data, confirms either synchronously or asynchronously tupe and connect
The data after filtration treatment are received, are either typically all used using tupe either synchronously or asynchronously, data receipt unit
Accept method interface users are transmitted through the data come, and the return value of Accept methods is exactly to locate under synchronous filtering tupe
Data after reason, data processing unit is received using GetData blocking method interfaces under asynchronous filtration treatment pattern and treated
Data.
In the present invention, serialization unit, message queue unit and unserializing unit are only in asynchronous filtration treatment pattern
It is lower called.Wherein, serialization unit is used to serialize processing reception data, after message queue unit is used for temporary serializing
Data object and provide that theme is subscribed to and user grouping is to support to handle multigroup asynchronous task simultaneously, unserializing unit is used for pair
Data object carries out unserializing processing.
In the present invention, the theme in message queue unit is subscribed to and user grouping function is mainly used for supporting while had more
Individual module needs asynchronous process service.
In the present invention, data processing unit is handled or for direct processing data the data after unserializing processing
Receiving unit receive data, data processing unit after receiving the data, using some filterings to being equipped with set in it
Device and processor are filtered and handled operation to data.
In the present invention, data processing unit uses DataIn method interface data, data source can be filter or
Processor.
In the present invention, data processing unit meets the data of filter condition using the output of DataPassOut methods interface, adopts
The data of filter condition are not met with the output of DataUnpassOut methods interface, using the output of DataOut methods interface at
Manage the data of operation.
In the present invention, using multiple the call definition filter and processor of AddFilterAndResolve method interfaces
Chain, because the method can be called repeatedly, more set filter blockers of addition can be integrated into link.
The present invention only can be embedded in java codes in the operation of reality with jar bags, it is not necessary to extra server
Or large-scale engine, it is absorbed in data filtering processing, lightweight, the efficiency high of data filtering processing.
The filter includes character string filter, date filter and digital filter.
The processor includes being used to delete field, delete object and replace the string processor of field, date processing
Device and digital processing unit, the string processor, date processor and digital processing unit and character string filter, date filtering
Device and digital filter fits are set.
In the present invention, for example, user need to do to character string to be handled " if wrapped in the character string for the field specified as follows
The character of the basic section containing non-unicode, i.e. mess code, then delete object ", now user can create character string filter,
And specifying will filter to which field, string processor is then created, pairing is put into engine, as user is defeated
It is " you also Ah add " to have a field in the object entered, then the object will be deleted and elect progress after treated
Delete.
In the present invention, filter and processor are generally all arranged in pairs.
The character string filter includes string length filter, string encoding filter, formulation character and judged
Filter, judge air filter and character string custom filter.
In the present invention, formulate character and judge that filter refers to whether character string includes the filter of designated character.
In the present invention, character string filter includes and is not limited to the filter of above-mentioned secretary.
The date filter includes date range filter, special time filter, date precision filter and date
Custom filter.
The date custom filter be for add and subtract the time, month, week, when, minute, second, integrate week, integrate the moon, whole
Close the filter of year or form conversion.
In the present invention, date filter includes and is not limited to the filter of above-mentioned secretary.
The digital filter includes digital scope filter, numeric type filter, numerical accuracy filter and numeral
Custom filter.
It is for taking negative, taking absolute value, evolution fortune with the digital processing unit that the digital filter coordinates in the present invention
The processor of calculation, power operation or exponent arithmetic.
In the present invention, digital filter includes and is not limited to above-mentioned filter.
In the present invention, filter and processor are to complete corresponding function setting, and its specific filtering, processing logic can be with
Demand according to those skilled in the art is voluntarily set, therefore filter and processor can also be by those skilled in the art according to reality
Border demand is set.
A kind of synchronous filtering method of the data filtering processing module based on java described in use, methods described include with
Lower step：
Step 1.1：Initialize some filters and processor to being equipped with；Initialization data receiving unit is synchronization process
Pattern；
Step 1.2：Data receipt unit is synchronously sent to data processing unit after data are received；
Step 1.3：Data processing unit calls some filters and processor to being equipped with after data are received；
Step 1.4：Filter and processor are filtered and handled to data, returned data receiving unit；
Step 1.5：Data after filtering and processing are back to user by data receipt unit as return value.
In the present invention, the method for synchrodata filtration treatment only relates to data receipt unit and data processing unit, two
The work of unit is synchronously carried out.
A kind of asynchronous filter method of the data filtering processing module based on java described in use, methods described include with
Lower step：
Step 2.1：Initialize some filters and processor to being equipped with；Initialization data receiving unit is asynchronous process
Pattern；Initialization sequence unit and unserializing unit；
Step 2.2：After data receipt unit receives data, serializing processing is carried out to data using serialization unit；
Step 2.3：Data object after serializing is put into message queue unit by data receipt unit；
Step 2.4：Data processing unit obtains data object at message queue unit, and using unserializing unit to data
Object is reduced, and gradually data are filtered and handled by some filters to being equipped with and processor；
Step 2.5：The data filtered and processing is completed are back to data receipt unit by message queue unit；
Step 2.6：Data after processing are back to user by data receipt unit as return value.
In the present invention, the method for asynchronous data filtration treatment is related to data receipt unit, serialization unit, message queue list
Member, unserializing unit and data processing unit, the asynchronous progress of work of data receipt unit and data processing unit.
The present invention by set data receipt unit be used for receive user data and confirm either synchronously or asynchronously tupe and
The data after filtration treatment are received, set serialization unit to be used to serialize reception data when handling asynchronous filtering, setting disappears
Cease the data after the temporary serializing of queue unit and theme subscription and user grouping are provided, unserializing unit is set to data pair
As carrying out unserializing processing, data processing unit is set to be handled the data after unserializing processing or for directly locating
The data that data receipt unit receives are managed, data processing unit is after receiving the data, some to being equipped with using what is set in it
Filter and processor to data carry out filtration treatment；By for the default of data receipt unit, confirming with synchronous or different
Stepping row data filtration treatment is operated, and data are docked and transmitted from suitable unit.Operation of the present invention in reality
In java codes only can be embedded in jar bags, it is not necessary to extra server or large-scale engine, be absorbed at data filtering
Reason, lightweight, the efficiency high of data filtering processing.
Claims (10)
- A kind of 1. data filtering processing module based on java, it is characterised in that：The data filtering processing module includes being used for The data receipt unit of data, the data sequence for data receipt unit to be received after reception user transmission data and filtration treatment The serialization unit of rowization processing, for data object after temporary serializing and support that theme is subscribed to and user grouping function Message queue unit, subscribed to for the data after serializing is handled according to theme and the different of user grouping carry out unserializings The unserializing unit of processing and handled for the data after handling unserializing or for direct processing data receive The data processing unit for the data that unit receives；The data processing unit includes some filters to being equipped with and place Manage device.
- A kind of 2. data filtering processing module based on java according to claim 1, it is characterised in that：The filter Including character string filter, date filter and digital filter.
- A kind of 3. data filtering processing module based on java according to claim 2, it is characterised in that：The processor It is described including string processor, date processor and the digital processing unit for deleting field, deletion object and replacement field String processor, date processor and digital processing unit and character string filter, date filter and digital filter fits Set.
- A kind of 4. data filtering processing module based on java according to claim 2, it is characterised in that：The character string Filter include string length filter, string encoding filter, formulate character judge filter, judge air filter and Character string custom filter.
- A kind of 5. data filtering processing module based on java according to claim 2, it is characterised in that：The date mistake Filter includes date range filter, special time filter, date precision filter and date custom filter.
- A kind of 6. data filtering processing module based on java according to claim 5, it is characterised in that：The date is certainly Define filter be for add and subtract the time, month, week, when, minute, second, integrate week, integrate the moon, integrate year or form conversion mistake Filter.
- A kind of 7. data filtering processing module based on java according to claim 2, it is characterised in that：The digital mistake Filter includes digital scope filter, numeric type filter, numerical accuracy filter and digital custom filter.
- A kind of 8. data filtering processing module based on java according to claim 6, it is characterised in that：With the numeral The digital processing unit of filter fits is for taking negative, taking absolute value, the processing of extracting operation, power operation or exponent arithmetic Device.
- 9. a kind of synchronous filtering method of data filtering processing module based on java using described in one of claim 1~8, It is characterized in that：It the described method comprises the following steps：Step 1.1：Initialize some filters and processor to being equipped with；Initialization data receiving unit is synchronization process Pattern；Step 1.2：Data receipt unit is synchronously sent to data processing unit after data are received；Step 1.3：Data processing unit calls some filters and processor to being equipped with after data are received；Step 1.4：Filter and processor are filtered and handled to data, returned data receiving unit；Step 1.5：Data after filtering and processing are back to user by data receipt unit as return value.
- A kind of 10. asynchronous filtering side of data filtering processing module based on java using described in one of claim 1~8 Method, it is characterised in that：It the described method comprises the following steps：Step 2.1：Initialize some filters and processor to being equipped with；Initialization data receiving unit is asynchronous process Pattern；Initialization sequence unit and unserializing unit；Step 2.2：After data receipt unit receives data, serializing processing is carried out to data using serialization unit；Step 2.3：Data object after serializing is put into message queue unit by data receipt unit；Step 2.4：Data processing unit obtains data object at message queue unit, and using unserializing unit to data Object is reduced, and gradually data are filtered and handled by some filters to being equipped with and processor；Step 2.5：The data filtered and processing is completed are back to data receipt unit by message queue unit；Step 2.6：Data after processing are back to user by data receipt unit as return value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711037833.5A CN107766538A (en) | 2017-10-28 | 2017-10-28 | Data filtering processing module and synchronous, asynchronous filter method based on java |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711037833.5A CN107766538A (en) | 2017-10-28 | 2017-10-28 | Data filtering processing module and synchronous, asynchronous filter method based on java |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107766538A true CN107766538A (en) | 2018-03-06 |
Family
ID=61271543
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711037833.5A Pending CN107766538A (en) | 2017-10-28 | 2017-10-28 | Data filtering processing module and synchronous, asynchronous filter method based on java |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107766538A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111913821A (en) * | 2020-08-17 | 2020-11-10 | 武汉众邦银行股份有限公司 | Method for realizing cross-data-source real-time data stream production consumption |
| CN114490834A (en) * | 2022-04-07 | 2022-05-13 | 梯度云科技(北京)有限公司 | Method and device for replacing big data calculation operation data source based on Kubernetes |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040059943A1 (en) * | 2002-09-23 | 2004-03-25 | Bertrand Marquet | Embedded filtering policy manager using system-on-chip |
| US7797276B1 (en) * | 2000-10-02 | 2010-09-14 | Guang Yang | Integrated database data editing system |
| CN102646228A (en) * | 2012-02-23 | 2012-08-22 | 天津市电力公司 | Multi-service real-time data integration processing system and method of intelligent power grid |
| CN103581305A (en) * | 2013-10-15 | 2014-02-12 | 医惠科技(苏州)有限公司 | Internet-of-things middleware system |
| CN105956082A (en) * | 2016-04-29 | 2016-09-21 | 深圳前海大数点科技有限公司 | Real-time data processing and storage system |
| CN107038161A (en) * | 2015-07-13 | 2017-08-11 | 阿里巴巴集团控股有限公司 | A kind of device for filtering data and method |
-
2017
- 2017-10-28 CN CN201711037833.5A patent/CN107766538A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7797276B1 (en) * | 2000-10-02 | 2010-09-14 | Guang Yang | Integrated database data editing system |
| US20040059943A1 (en) * | 2002-09-23 | 2004-03-25 | Bertrand Marquet | Embedded filtering policy manager using system-on-chip |
| CN102646228A (en) * | 2012-02-23 | 2012-08-22 | 天津市电力公司 | Multi-service real-time data integration processing system and method of intelligent power grid |
| CN103581305A (en) * | 2013-10-15 | 2014-02-12 | 医惠科技(苏州)有限公司 | Internet-of-things middleware system |
| CN107038161A (en) * | 2015-07-13 | 2017-08-11 | 阿里巴巴集团控股有限公司 | A kind of device for filtering data and method |
| CN105956082A (en) * | 2016-04-29 | 2016-09-21 | 深圳前海大数点科技有限公司 | Real-time data processing and storage system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111913821A (en) * | 2020-08-17 | 2020-11-10 | 武汉众邦银行股份有限公司 | Method for realizing cross-data-source real-time data stream production consumption |
| CN111913821B (en) * | 2020-08-17 | 2021-07-16 | 武汉众邦银行股份有限公司 | Method for realizing cross-data-source real-time data stream production consumption |
| CN114490834A (en) * | 2022-04-07 | 2022-05-13 | 梯度云科技(北京)有限公司 | Method and device for replacing big data calculation operation data source based on Kubernetes |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101976240B (en) | Form number generating method and system | |
| CN109828903A (en) | Automated testing method, device, computer installation and storage medium | |
| CN106802947B (en) | The data processing system and method for entity relationship diagram | |
| CN107766538A (en) | Data filtering processing module and synchronous, asynchronous filter method based on java | |
| CN103782285B (en) | Collection and management to accurate user preference data | |
| DK1016014T3 (en) | Order processing apparatus and method | |
| CN104317970A (en) | Data flow type processing method based on data processing center | |
| CN103353957A (en) | Realization method of human-centered cooperative workflow system and apparatus thereof | |
| CN107741950A (en) | Processing method, device, processor and the service end of data syn-chronization task | |
| CN113722055A (en) | Data processing method and device, electronic equipment and computer readable medium | |
| CN100452704C (en) | Method and method for issuing blog articles | |
| CN110532559B (en) | Rule processing method and device | |
| CN105656643A (en) | Method and equipment for performing compatible management based on network configuration protocol | |
| CN106293995B (en) | The data backup system and method for manufacturing execution system | |
| CN110601866A (en) | Flow analysis system, data acquisition device, data processing device and method | |
| CN110278220A (en) | A kind of multimedia message processing method and device | |
| CN105187607A (en) | Message processing method and system | |
| CN107329832A (en) | A kind of data receiver method and device | |
| CN106846190A (en) | A kind of intellectual property case management system | |
| CN111857901A (en) | Data processing method, method for generating session background, electronic device and medium | |
| CN101572708A (en) | Ground-to-air data link information processing method and device | |
| WO2017107438A1 (en) | Method for controlling reporting of crash data, server and client | |
| CN104219136A (en) | System and method for updating attachment in real time during circulation of E-mail | |
| CN104866463A (en) | Information arrangement method and device | |
| CN108416572B (en) | Method for integrating insurance company insurance application processes into general insurance application process |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180306 |
|
| RJ01 | Rejection of invention patent application after publication |