CN109063066A - Data query method and device, data management system - Google Patents
Data query method and device, data management system Download PDFInfo
- Publication number
- CN109063066A CN109063066A CN201810806501.7A CN201810806501A CN109063066A CN 109063066 A CN109063066 A CN 109063066A CN 201810806501 A CN201810806501 A CN 201810806501A CN 109063066 A CN109063066 A CN 109063066A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- target
- user
- list
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of data query method and devices, data management system.Wherein, this method comprises: whether the target data set in interval predetermined time detection first database changes, wherein, it include multiple target datas in target data set, target data is the data information of target user's list, and target user is the user for recording prestige degree and being lower than predeterminable level；If detecting, the target data set in first database changes, and obtains data variation set, wherein includes the set of changed data in target data set in data variation set；Second database is updated using data variation set, wherein the second database is the database inquired for the data information to target user's list；Data query is carried out in the second database.The technical issues of present invention, which solves, in the related technology inquires mass data, and speed is slow, reduces user experience.
Description
Technical field
The present invention relates to technical field of data processing, in particular to a kind of data query method and device, data pipe
Reason system.
Background technique
In the related technology, when the data stored in inquiring database, usually according to the demand of user, database is inquired
In storage total data, the data of inquiry are then back to inquiry user, this inquiry mode is inquiring a small amount of number
According to when, can quick search, but for more data, inquiring data can be very slow.And in current data query, inquiry into
Journey is only capable of being inquired for a small amount of data, and in the presence of multiple processes, it is slow to have a high concurrent inquiry velocity,
It is lower using the experience sense of database to will result in user in this way for the lower problem of search efficiency.
Mass data to be inquired in the related technology for above-mentioned, speed is slow, the technical issues of reducing user experience,
Currently no effective solution has been proposed.
Summary of the invention
The embodiment of the invention provides a kind of data query method and devices, data management system, at least to solve correlation
The technical issues of for mass data inquiry in technology, speed is slow, reduces user experience.
According to an aspect of an embodiment of the present invention, a kind of data query method is provided, comprising: the inspection of interval predetermined time
Whether the target data set surveyed in first database changes, wherein includes multiple targets in the target data set
Data, the target data are the data information of target user's list, and the target user is record prestige degree lower than default
The user of degree；If detecting, the target data set in the first database changes, and obtains data variation set,
In, it include the set of changed data in the target data set in the data variation set；Using the data
Variation set updates the second database, wherein second database is to carry out for the data information to target user's list
The database of inquiry；Data query is carried out in second database.
Further, obtaining data variation set includes: to obtain object run instruction, wherein the object run instruction
It is used to indicate and stores the target data set to second database；It is instructed according to the object run, described in determination
Data variation set.
Further, updating the second database using the data variation set includes: to read the data variation set
In all data, wherein the data variation set, the main collection are read by main cluster library in second database
Group library is the database of all data for synchronizing the target data set in second database, second number
Include at least according to library: a main cluster library, multiple spare distributed data bases, the main cluster library configure multiple scheduling processes,
The scheduling process is used for more new data；All data in the data variation set is updated to second database.
Further, it includes: to be counted by preset interface to described first that data query is carried out in second database
Prompt information is sent according to library, wherein the prompt information is completed for prompting the first database data to update.
Further, in second database after progress data query, comprising: target journaling data are obtained,
In, the target journaling data are to carry out inquiry operation to the data information of target user's list in second database
And/or carry out the daily record data of processing operation；The target journaling data are stored to first database.
Further, before obtaining target journaling data, which comprises record is in second database
The data information of target user's list carries out the inquiry log of inquiry operation, and carries out the processing log of processing operation, is looked into
Ask daily record data and processing daily record data；According to the inquiry log data and the processing daily record data, the target is determined
Daily record data, wherein the target journaling data are looked into for what the spare distributed data base of second database was recorded
Ask daily record data and processing daily record data.
According to another aspect of an embodiment of the present invention, a kind of data management system is additionally provided, comprising: first database,
For storing target data, wherein the target data is the data information of target user's list, and the target user is record
Prestige degree is lower than the user of predeterminable level；Second database, the data of target user's list for inquiring in main cluster library
Information, and/or, the data information of target user's list in main cluster library is handled, inquiry log data and processing log are obtained
Data；Wherein, first database interval preset time updates the target data once stored, the master of second database
The target data of variation is generated in first database described in the synchronized update of cluster library, for user query and/or the processing mesh
Mark the data information of user list；Wherein, the user is stored in second database to carry out inquiring target user's name
The inquiry log data that single data information obtains, and/or, store the number that the user handle target user's list
It is believed that the processing daily record data that breath obtains, to obtain target journaling data, the spare distributed data base of second database
The target journaling data are stored to the first database.
According to another aspect of an embodiment of the present invention, a kind of data query device is additionally provided, comprising: detection unit is used
Whether the target data set in interval predetermined time detection first database changes, wherein the target data set
It include multiple target datas in conjunction, the target data is the data information of target user's list, and the target user is record
Prestige degree is lower than the user of predeterminable level；Acquiring unit, if for detecting the target data set in the first database
Conjunction changes, and obtains data variation set, wherein includes occurring in the data variation set in the target data set
The set of the data of variation；Updating unit, for updating the second database using the data variation set, wherein described the
Two databases are the database inquired for the data information to target user's list；Query unit, for described the
Data query is carried out in two databases.
Further, acquiring unit includes: acquisition module, for obtaining object run instruction, wherein the object run
Instruction, which is used to indicate, stores the target data set to second database；Determining module, for according to the target
Operational order determines the data variation set.
Further, updating unit includes: read module, for reading all data in the data variation set,
Wherein, the data variation set is read by main cluster library in second database, the main cluster library is described the
In two databases, the database of all data for synchronizing the target data set, second database is included at least:
One main cluster library, multiple spare distributed data bases, the main cluster library configure multiple scheduling processes, and the scheduling process is used
In more new data；Update module, for all data in the data variation set to be updated to second database.
Further, the query unit includes: sending module, for being sent out by preset interface to the first database
Send prompt information, wherein the prompt information is completed for prompting the first database data to update.
Further, comprising: acquisition submodule obtains after carrying out data query in second database
Target journaling data, wherein the target journaling data are to believe the data of target user's list in second database
Breath carries out inquiry operation and/or carries out the daily record data of processing operation；Memory module, for storing the target journaling data
To first database.
Further, described device further include: logging modle, for recording to institute before obtaining target journaling data
The data information for stating target user's list in the second database carries out the inquiry log of inquiry operation, and carries out processing operation
Log is handled, inquiry log data and processing daily record data are obtained；Determine submodule, for according to the inquiry log data and
The processing daily record data determines the target journaling data, wherein the target journaling data are second database
The inquiry log data and processing daily record data that spare distributed data base is recorded.
According to another aspect of an embodiment of the present invention, a kind of storage medium is additionally provided, the storage medium includes storage
Program, wherein described program operation when control the storage medium where equipment execute number described in above-mentioned any one
According to querying method.
According to another aspect of an embodiment of the present invention, a kind of processor is additionally provided, the processor is used to run program,
Wherein, data query method described in above-mentioned any one is executed when described program is run.
In embodiments of the present invention, the target data set that the interval predetermined time is detected in first database, which may be implemented, is
It is no to change, it include multiple target datas in the target data set, target data is the data information of target user's list,
Information of the prestige degree lower than the user of predeterminable level is recorded, and is detecting the target data set generation in first database
In the case where variation, the set of changed data is obtained, the second database is then updated using data variation set, finally
Data query can be carried out in the second database.In this embodiment it is possible to realize the data in first database are quick
It being updated in the second database, user can inquire the data information of target user's list in the second database, so that it is determined that
Whether user to be checked can be led on target user's list in the second database with the data information of fast target user list
The target user's list crossed quickly, in second database that timely updates can allow personnel query to carry out inquiry behaviour in the second database
It is convenient when making, mass data is inquired without entering first database, it is only necessary in second data base querying target user's list
Data information, and the data information of the target user's list quickly updated make inquiry more acurrate, and then solve related skill
The technical issues of for mass data inquiry in art, speed is slow, reduces user experience.
Detailed description of the invention
The drawings described herein are used to provide a further understanding of the present invention, constitutes part of this application, this hair
Bright illustrative embodiments and their description are used to explain the present invention, and are not constituted improper limitations of the present invention.In the accompanying drawings:
Fig. 1 is the flow chart of data query method according to an embodiment of the present invention；
Fig. 2 is a kind of schematic diagram of database association relationship according to an embodiment of the present invention；
Fig. 3 is the schematic diagram of data management system according to an embodiment of the present invention；
Fig. 4 is the schematic diagram of data query device according to an embodiment of the present invention.
Specific embodiment
In order to enable those skilled in the art to better understand the solution of the present invention, below in conjunction in the embodiment of the present invention
Attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is only
The embodiment of a part of the invention, instead of all the embodiments.Based on the embodiments of the present invention, ordinary skill people
The model that the present invention protects all should belong in member's every other embodiment obtained without making creative work
It encloses.
It should be noted that description and claims of this specification and term " first " in above-mentioned attached drawing, "
Two " etc. be to be used to distinguish similar objects, without being used to describe a particular order or precedence order.It should be understood that using in this way
Data be interchangeable under appropriate circumstances, so as to the embodiment of the present invention described herein can in addition to illustrating herein or
Sequence other than those of description is implemented.In addition, term " includes " and " having " and their any deformation, it is intended that cover
Cover it is non-exclusive include, for example, the process, method, system, product or equipment for containing a series of steps or units are not necessarily limited to
Step or unit those of is clearly listed, but may include be not clearly listed or for these process, methods, product
Or other step or units that equipment is intrinsic.
To understand the present invention convenient for user, solution is made to part term or noun involved in various embodiments of the present invention below
It releases:
Oracle database, Oracle RDBMS are a relational database management system, data stable storage, side
Just.
Redis database is being write using ANSI C language, supported network, memory-based persistence an of open source
Log type data, quick search data may be implemented.
According to embodiments of the present invention, a kind of embodiment of the method for data query is provided, it should be noted that in attached drawing
The step of process illustrates can execute in a computer system such as a set of computer executable instructions, although also,
Logical order is shown in flow chart, but in some cases, it can be to be different from shown by sequence execution herein or retouch
The step of stating.
Following embodiment can be applied in various data processing circumstances, for example, in bank savings, postal saving, wherein
The system of application does not limit, the embodiment of the present invention by taking transaction swindling management system as an example, to data access and data store into
Row explanation, to improve data access speed and data storage efficiency.
It is illustrated below with reference to preferred implementation steps, Fig. 1 is data query method according to an embodiment of the present invention
Flow chart, as shown in Figure 1, this method comprises the following steps:
Whether step S102, the target data set that the interval predetermined time is detected in first database change, wherein
It include multiple target datas in target data set, target data is the data information of target user's list, and target user is note
Record the user that prestige degree is lower than predeterminable level.
Wherein, above-mentioned first database can be the database for storing mass data, including but not limited to: relationship
Type database, the present invention are illustrated using oracle database as first database, but are not limited to the Oracle data
Library.In addition, the core that first database is stored as data, can not only store the data information of target user's list, may be used also
To store other data informations, including but not limited to: public administration class data, batching data, information pre-processing data, rule
Then engine data, interface service class data.Using first database as the database of core memory, data exchange, can wrap
Include data processing class data, using data class data, regular associated class data, basic data class data, wherein data processing class
Data may include: scheduling configuration data, running log data, may include: organization information, Gui Yuanxin using data class data
Breath, risk information, customer information, report data, user list information may include: that rule is drawn for regular associated class data
Configuration data, business rule data, relation access control data are held up, may include: teller's incremental number for basic data class data
According to, mechanism full dose data, client's incremental data, list incremental data, inquiry log data, processing daily record data.In the present invention
Emphasis application data class data in the data of user list information handled, the especially number to user list information
Data according to middle prestige degree lower than predeterminable level are handled.Optionally, the present invention can define prestige journey with grey black list
Lower user list information is spent, multiple target user's information are contained in grey black list, which is understood that
For prestige degree it is low, not can be carried out the relevant information of the user of business exchange.
In addition, the target data set in the embodiment of the present invention can be above-mentioned grey black list, including multiple users'
Relevant information can be the data that user is not desired to business exchange or contact, user for the data in target data set
Can by the second data base querying target data set, at this moment just need by first database store target data set and
When synchronized update into the second database, to facilitate user query.When user inquires, looked by the second database
It askes, the data in the second database can be caused with the target data set unification stored in first database, at this moment, it is necessary to by the
The data that variation is generated in one database also timely update into the second database.Above-mentioned interval preset time, can be use
The time of the more new data of family self-setting, for example, 5 minutes, 7 minutes, 10 minutes, in the embodiment of the present invention for 5 minutes,
Every 5 minutes, whether the data detected in the target data set in all data stored in a first database became
Change, once it changes, it is necessary to data changed in the target collection in first database are timely updated to second
In database.It is illustrated in the embodiment of the present invention with two databases, wherein the second database can be immediately for for quick
The database used is inquired, the data that user wants inquiry can be temporarily stored in the second database.Although in first database
Mass data is stored, however, it is desirable to which inquiring the data used can be target data set, i.e., in inquiry target data set
Data.Target data set can be identical with the data in the second database in first database.In the related art,
Target data set in first database may be the data acquisition system for generating variation at any time, for example, it was discovered that there is new user
Prestige degree is lower than predeterminable level, the relevant information of new user can be both input in target data set, target data
Data in set may include a plurality of types of data, comprising: user name, account, identification card number.
It should be noted that can be in the data stored by the second data base querying by identification card number or use
Whether name in an account book is input in the corresponding data acquisition system of the second database, have the identification card number is corresponding to make to inquire in grey black list
User.Target data set in first database can be understood as first object data acquisition system, the target in the second database
Data acquisition system can be understood as the second target data set.When updating, updating basis is first object data acquisition system, the second number
It is updated in time according to the second target data set needs in library, in order to inquire.
Step S104, if detecting, the target data set in first database changes, and obtains data variation set,
It wherein, include the set of changed data in target data set in data variation set.
Step S106 updates the second database using data variation set, wherein the second database is for using target
The database that the data information of name in an account book list is inquired.
Step S108 carries out data query in the second database.
Optionally, can be in second database it is various inquire the databases that use, such as Redis database, thus fastly
The data that user wants are ask in quick checking, so that data query is convenient, the second database in the present invention may only be by first database
In data synchronization updating into the main cluster library of the second database, but cannot inquiry when modify data.
Through the above steps, whether the target data set that may be implemented in interval predetermined time detection first database is sent out
Changing includes multiple target datas in the target data set, and target data is the data information of target user's list, record
Prestige degree and is detecting that the target data set in first database changes lower than the information of the user of predeterminable level
In the case where, the set of changed data is obtained, the second database is then updated using data variation set, it finally can be with
Data query is carried out in the second database.The data in first database are quickly updated in this embodiment it is possible to realize
Into the second database, user can inquire the data information of target user's list in the second database, so that it is determined that be checked
User is ask whether on target user's list, it can be with the data information of fast target user list, by fast in the second database
Target user's list in speed, second database that timely updates can allow personnel query when the second database carries out inquiry operation
It is convenient, mass data is inquired without entering first database, it is only necessary in the data of second data base querying target user's list
Information, and the data information of the target user's list quickly updated make inquiry more acurrate, and then solve in the related technology
The technical issues of for mass data inquiry, speed is slow, reduces user experience.
Optionally, obtaining data variation set includes: to obtain object run instruction, wherein object run is instructed for referring to
Show and stores target data set to the second database；It is instructed according to object run, determines data variation set.
In addition, updating the second database using data variation set includes: all data read in data variation set,
Wherein, data variation set is read by the main cluster library in the second database, main cluster library is in the second database, for same
The database of all data of target data set is walked, the second database includes at least: a main cluster library, multiple spare distributions
Formula database, main cluster library configure multiple scheduling processes, and scheduling process is used for more new data；By the items in data variation set
Data are updated to the second database.
The second database in the embodiment of the present invention may include multiple databases, including a main cluster library and multiple standby
With distributed data base, wherein it can store the second target data set in main cluster library, i.e. storage grey black list data, this
Sample needs the data that can directly inquire in main cluster library in inquiry.It, can be from spare point if main cluster library is abnormal
It selects a database as main cluster library in cloth database, in selection, can be selected based on ballot selection mechanism,
So that it is determined that main cluster library out.Optionally, the main cluster library in the embodiment of the present invention can have multiple guard processes (i.e. above-mentioned
Scheduling process), the guard process can be for dispatch, coordinate to carry out data update and data query and data processing into
Journey.Spare distributed data base in second database can be also used for generating inquiry log and handle log, the inquiry log,
It can be the log of the second target collection in the second database of user query, log content may include query time, inquiry
As a result etc., processing log can be to the log of inquiry handle.
In addition, above-mentioned guard process can also monitor the state in main cluster library and spare distributed data base, in main collection
When group library occurs abnormal, new main cluster library is determined by ballot selection mechanism in time, guarantees the safety of data service.
It should be noted that in the embodiment of the present invention, when reading data variation set, its historical data can also be read
Gather, i.e. corresponding second target data set in the second database.
Optionally, it includes: to be mentioned by preset interface to first database transmission that data query is carried out in the second database
Show information, wherein prompt information is completed for prompting first database data to update.
For the above embodiment, it can be understood as in the second database with the target data set in first database
Data synchronously complete after, inform first database update complete, that is, complete grey black list data synchronization process.
Wherein, in the second database after progress data query, comprising: obtain target journaling data, wherein target day
Will data are to carry out inquiry operation to the data information of target user's list in the second database and/or carry out processing operation
Daily record data；Target journaling data are stored to first database.
I.e. user can inquire the data after the completion of update, and generate relevant target journaling data, the second number
The target journaling data of generation can finally be stored to first database with these target journaling data of blotter according to library
In, to allow first database stable storage, used for subsequent query.
Wherein, before obtaining target journaling data, above-described embodiment further include: record is to the target in the second database
The data information of user list carries out the inquiry log of inquiry operation, and carries out the processing log of processing operation, obtains inquiry day
Will data and processing daily record data；According to inquiry log data and processing daily record data, target journaling data are determined, wherein mesh
Mark daily record data is the inquiry log data that the spare distributed data base of the second database is recorded and processing daily record data.
Optionally, daily record data is being stored into the second database, a target day can be being read every preset time
Will data, preset time can be the data of user's sets itself, for example, 5 minutes, 10 minutes etc..
Can the log of emphasis record queries and processing log, obtain inquiry log data and processing daily record data, and will
The inquiry log data and processing daily record data are stored and are stored into first database.Optionally, log number in the present invention
According to that can be correspondingly arranged log key assignments, the log key assignments is for identifying stl status, for example, having newly-increased daily record data, Ke Yishe
An information log key assignments is set, identifying in log has data increase, stores in first database by data, can pass through log
Key assignments reads daily record data, finally completes the synchronous process of daily record data, the daily record data in the second database is synchronized store to
In first database.
The embodiment of the present invention has also been taken into account the requirement of data persistence, has been realized while focusing on data query speed
The perfect combination of dynamic queries and static storage can either meet operation system and inquire undelayed requirement, and can be by business
The requirement of data persistence storage.It is a good successful case for other systems in row, especially for double 11, double
The system of ten second-class service concurrence peak periods tests, and provides a reference well.
Below with reference to another kind, examples illustrate the present invention.
Wherein, Fig. 2 is a kind of schematic diagram of database association relationship according to an embodiment of the present invention, as shown in Fig. 2, the number
According to library, including core database (the first i.e. above-mentioned data), Redis database (the second i.e. above-mentioned database), Redis number
It include Redis principal and subordinate cluster and Redis distributed type assemblies according to library, the data type stored in core database may include data
Handle class data, using data class data, regular associated class data, basic data class data, wherein data processing class data can
It may include: organization information, teller's information, wind using data class data to include: scheduling configuration data, running log data
Dangerous information, customer information, report data, grey black list (i.e. above-mentioned target data set), can for regular associated class data
To include: that regulation engine configuration data, business rule data, relation access control data can wrap basic data class data
It includes: teller's incremental data, mechanism full dose data, client's incremental data, list incremental data, inquiry log data, processing log
Data.And the grey black list in Redis principal and subordinate's cluster in synchronized update core database, remember in the distributed type assemblies in Redis
Access configuration, inquiry log, processing log have been recorded, and the inquiry log of record and processing log have been stored to core database
In.
Optionally, the data in the embodiment of the present invention can be public administration class data and be stored in core database, data
Batching data is stored in core database, and information pre-processing data are stored in core database, and regulation engine data are stored in
Core database provides interface service class data, stores core database, memory database is synchronous from core database, interface
Serve log class data, are introduced into memory database, and core database timing is synchronous from memory database.
Data relationship in the embodiment of the present invention can be applied in transaction swindling management system system, for of the invention real
Apply the data framework design in example, wherein the data framework of transaction swindling management system system is on the basis for meeting IT planning
On, need to fully consider the operating mode of each subsystem.Application door subsystem is to handle public administration data, risk information number
Based on, statistical report form data；Data process subsystem is to handle data in each interconnected system source data and system and synchronize
It is main；Interface service subsystem with handle risk inquiry transaction generate pipelined data based on；Using Monitor And Control Subsystem with processing system
Based on running log data of uniting.
The data storage of transaction swindling management system is using oracle database and the distribution of Redis principal and subordinate cluster and Redis
Three kinds of mode storing datas of formula cluster.
Transaction swindling management system mainly includes following part using the data of oracle database storage: risk inquiry is handed over
Easy flowing water, public administration data, risk information data, statistical report form data, full dose grey black list data, regulation engine correlation are matched
Set data, day runs batch related configuration data eventually.
Transaction swindling management system mainly includes following part using the data of Redis principal and subordinate cluster: on-line transaction uses
Grey black list snapshot data, i.e., the data recorded at of short duration time point.
Transaction swindling management system mainly includes following part using the data of Redis distributed type assemblies storage: risk is looked into
Ask the transaction journal data (i.e. temporary storage of data) that transaction generates.
For the data retention period of transaction swindling management system, number can be determined by the analysis to data and business
According to the common period and retention periods, wherein the common period refers to the date intervals that data can be commonly used in on-line transaction,
Retention periods refer to the data longest date intervals to be saved according to operation regulation of on-line transaction.Historical data refers to be more than normal
The data of retention periods are not arrived with the period.Historical data can only inquire, and cannot change.
Under the common period initial setting of transaction swindling management system various businesses data is as shown in table 1:
Table 1
Wherein, it according to transaction swindling management system business characteristic, needs to shift the data master being saved in historical data base
It will be including several classes once: risk inquiry transaction journal and processing result data are stored in Redis；Risk inquiry is stored in Oracle
Transaction journal and processing result data；The risk information data of importing are stored in Oracle；Report form statistics number is stored in Oracle
According to.Transaction journal and processing result historical data can only inquire, and not allow to carry out online risk behavior analysis；Risk information history
Data can only inquire, and not allow to modify；Report form statistics data only allow to inquire data in 1 year.
Since the data volume of history lists is big, it is necessary to which the inquiry quantity and content of strict control historical data such as pass through authorization
Mechanism, clear querying condition, limitation background service process, limitation inquire the measures such as stroke count every time to reduce the shadow to system performance
It rings.
Data exchange between each subsystem of transaction swindling management system internal system uses the data method of synchronization.Transaction is taken advantage of
The data exchange cheated between management system and other systems is real using modes such as on-line transaction interface, data-sharing systems transmission
It is existing.On-line transaction interface concrete mode is advised referring to interconnected system interface in post-office savings bank's transaction swindling management system and row
Model.
Data Synchronization Design
Firstly, oracle database includes: to the synchronous list data of Redis memory database
11, data synchronization process is by people's row blacklist delta file parsing storage into oracle database.
12, it was polled to the increment list data being put in storage by timing routine every 5 minutes, and passes through storing process generation pair
The operational order of Redis memory bank is put into corresponding data table, hereinafter referred to as " T1 ".
13, polling routine reads T1, and executes in Redis memory database.Since interface service is to list in Redis
Data only read, do not do and update, so Redis is built using active and standby trunking mode, host configures two guard processes, respectively from
Machine respectively configures a guard process.List data is synchronous to respectively from library by master library.Guard process monitoring principal and subordinate library state, master library
Decided whether when abnormal state, between guard process by voting mechanism master library logic is offline, and elect one from qurush grade
For master library.To ensure that the safety of data service.
14, data handling procedure is completed to update the operational order state of T1 in oracle database to after Redis operation,
Complete the synchronization process of list data.
Secondly, Redis memory database includes: to oracle database synchronization log data
21, interconnected system transaction risk inquiry log is recorded in Redis database interface service.Due to interface service
It is subject to and is written to the operation of Redis log library, so cluster environment is disposed in a distributed manner in Redis log library, it is each on every host
Deployment one main one is standby, and active and standby library misplaces configuration between each host of cluster.
22, the log key assignments that data processor is once increased newly by the every 5 minutes polls of timing.
23, data processor reads log recording in Redis log library, and Oracle is written by increasing key assignments newly
Database completes the synchronous process of daily record data.
In the above-described embodiments, interconnected system inquiry is only with 10ms or so, this when client transacts business fundamental sensation without prolonging
Late, realize on operation system it is non-delay, do not influence produce reaction speed.Data query speed is being focused in the invention
Meanwhile the requirement of data persistence is also taken into account.The perfect combination of dynamic queries and static storage is realized, industry can either be met
The business undelayed requirement of system queries, and can be by the requirement of business datum persistent storage.It is for systems other in row
One good successful case tests the system of double 11, double ten second-class service concurrence peak periods, provides one very well
Reference.
Fig. 3 is the schematic diagram of data management system according to an embodiment of the present invention, as shown in figure 3, the system may include:
First database 31, for storing target data, wherein target data is the data information of target user's list, target user
It is lower than the user of predeterminable level for record prestige degree；Second database 33, target user's name for inquiring in main cluster library
Single data information, and/or, handle the data information of target user's list in main cluster library, obtain inquiry log data and
Handle daily record data；Wherein, first database interval preset time updates the target data once stored, the master of the second database
The target data of variation is generated in the synchronized update first database of cluster library, for user query and/or processing target user name
Single data information；Wherein, what the data information that storage user carries out inquiry target user's list in the second database obtained looks into
Daily record data is ask, and/or, storage user carries out the processing daily record data that the data information of processing target user list obtains, with
Target journaling data are obtained, the spare distributed data base of the second database stores target journaling data to first database.
By above-mentioned data management system, the number of targets once stored can be updated with first database interval preset time
According to, the target data of variation is generated in the main cluster library synchronized update first database of the second database, for user query and/
Or the data information of processing target user list, and the number that user carries out inquiry target user's list is stored in the second database
It is believed that the inquiry log data that breath obtains, and/or, storage user carries out the place that the data information of processing target user list obtains
Daily record data is managed, to obtain target journaling data, the spare distributed data base of the second database stores target journaling data
To first database, the data in first database are quickly updated in the second database in this embodiment it is possible to realize,
User can inquire the data information of target user's list in the second database, so that it is determined that whether user to be checked is in target
It, can be with the data information of fast target user list, by the way that quick, timely update the second number in the second database on user list
Personnel query can be allowed convenient when the second database carries out inquiry operation according to target user's list in library, without entering first
Data base querying mass data, it is only necessary in the data information of second data base querying target user's list, and quickly
The data information of target user's list of update makes inquiry more acurrate, and then solves in the related technology to inquire mass data,
The technical issues of speed is slow, reduces user experience.
Fig. 4 is the schematic diagram of data query device according to an embodiment of the present invention, as shown in figure 4, the apparatus may include:
Whether detection unit 41, the target data set for being spaced in predetermined time detection first database change, wherein mesh
Marking includes multiple target datas in data acquisition system, and target data is the data information of target user's list, and target user is record
Prestige degree is lower than the user of predeterminable level；Acquiring unit 43, if for detecting the target data set in first database
It changes, obtains data variation set, wherein include changed data in target data set in data variation set
Set；Updating unit 45, for using data variation set update the second database, wherein the second database be for pair
The database that the data information of target user's list is inquired；Query unit 47, for carrying out data in the second database
Inquiry.
Through the foregoing embodiment, it can realize that the interval predetermined time detects the mesh in first database by detection unit 41
Whether mark data acquisition system changes, and includes multiple target datas in the target data set, and target data is target user's name
Single data information, record prestige degree are lower than the information of the user of predeterminable level, and are detecting the mesh in first database
In the case that mark data acquisition system changes, the set of changed data can be obtained by acquiring unit 43, then may be used
It, finally can be by query unit 47 in the second number to update the second database using data variation set by updating unit 45
According to carrying out data query in library.The data in first database are quickly updated to the second number in this embodiment it is possible to realize
According in library, user can inquire the data information of target user's list in the second database, so that it is determined that user to be checked is
It is no on target user's list, can be with the data information of fast target user list, by quick, in time more in the second database
Target user's list in new second database can allow personnel query convenient when the second database carries out inquiry operation, be not necessarily to
Mass data is inquired into first database, it is only necessary in the data information of second data base querying target user's list,
And the data information of the target user's list quickly updated, make inquiry more acurrate, and then solve in the related technology for a large amount of
The technical issues of data query, speed is slow, reduces user experience.
Optionally, acquiring unit 43 may include: acquisition module, for obtaining object run instruction, wherein object run
Instruction, which is used to indicate, stores target data set to the second database；Determining module is determined for being instructed according to object run
Data variation set.
In addition, updating unit 45 may include: read module, for reading all data in data variation set,
In, data variation set is read by the main cluster library in the second database, main cluster library is in the second database, for synchronizing
The database of all data of target data set, the second database include at least: a main cluster library, multiple spare distributions
Database, main cluster library configure multiple scheduling processes, and scheduling process is used for more new data；Update module is used for data variation
All data in set is updated to the second database.
Query unit 47 in above-mentioned inquiry unit may include: sending module, for passing through preset interface to first
Database sends prompt information, wherein prompt information is completed for prompting first database data to update.
It should be noted that above-mentioned apparatus can also include: acquisition submodule, for carrying out data in the second database
After inquiry, target journaling data are obtained, wherein target journaling data are the number to target user's list in the second database
It is believed that breath carries out inquiry operation and/or carries out the daily record data of processing operation；Memory module, for storing target journaling data
To first database.
In addition, device can also include: logging modle, for before obtaining target journaling data, record to be to the second number
The inquiry log of inquiry operation is carried out according to the data information of target user's list in library, and carries out the processing day of processing operation
Will obtains inquiry log data and processing daily record data；Submodule is determined, for according to inquiry log data and processing log number
According to determining target journaling data, wherein target journaling data are what the spare distributed data base of the second database was recorded
Inquiry log data and processing daily record data.
According to another aspect of an embodiment of the present invention, a kind of storage medium is additionally provided, the storage medium includes storage
Program, wherein described program operation when control the storage medium where equipment execute number described in above-mentioned any one
According to querying method.
According to another aspect of an embodiment of the present invention, a kind of processor is additionally provided, the processor is used to run program,
Wherein, data query method described in above-mentioned any one is executed when described program is run.
The serial number of the above embodiments of the invention is only for description, does not represent the advantages or disadvantages of the embodiments.
In the above embodiment of the invention, it all emphasizes particularly on different fields to the description of each embodiment, does not have in some embodiment
The part of detailed description, reference can be made to the related descriptions of other embodiments.
In several embodiments provided herein, it should be understood that disclosed technology contents can pass through others
Mode is realized.Wherein, the apparatus embodiments described above are merely exemplary, such as the division of the unit, Ke Yiwei
A kind of logical function partition, there may be another division manner in actual implementation, for example, multiple units or components can combine or
Person is desirably integrated into another system, or some features can be ignored or not executed.Another point, shown or discussed is mutual
Between coupling, direct-coupling or communication connection can be through some interfaces, the INDIRECT COUPLING or communication link of unit or module
It connects, can be electrical or other forms.
The unit as illustrated by the separation member may or may not be physically separated, aobvious as unit
The component shown may or may not be physical unit, it can and it is in one place, or may be distributed over multiple
On unit.It can some or all of the units may be selected to achieve the purpose of the solution of this embodiment according to the actual needs.
It, can also be in addition, the functional units in various embodiments of the present invention may be integrated into one processing unit
It is that each unit physically exists alone, can also be integrated in one unit with two or more units.Above-mentioned integrated list
Member both can take the form of hardware realization, can also realize in the form of software functional units.
If the integrated unit is realized in the form of SFU software functional unit and sells or use as independent product
When, it can store in a computer readable storage medium.Based on this understanding, technical solution of the present invention is substantially
The all or part of the part that contributes to existing technology or the technical solution can be in the form of software products in other words
It embodies, which is stored in a storage medium, including some instructions are used so that a computer
Equipment (can for personal computer, server or network equipment etc.) execute each embodiment the method for the present invention whole or
Part steps.And storage medium above-mentioned includes: that USB flash disk, read-only memory (ROM, Read-Only Memory), arbitrary access are deposited
Reservoir (RAM, Random Access Memory), mobile hard disk, magnetic or disk etc. be various to can store program code
Medium.
The above is only a preferred embodiment of the present invention, it is noted that for the ordinary skill people of the art
For member, various improvements and modifications may be made without departing from the principle of the present invention, these improvements and modifications are also answered
It is considered as protection scope of the present invention.
Claims (10)
1. a kind of data query method characterized by comprising
Whether the target data set in interval predetermined time detection first database changes, wherein the target data
It include multiple target datas in set, the target data is the data information of target user's list, and the target user is note
Record the user that prestige degree is lower than predeterminable level；
If detecting, the target data set in the first database changes, and obtains data variation set, wherein described
It include the set of changed data in the target data set in data variation set；
Second database is updated using the data variation set, wherein second database is for target user's name
The database that single data information is inquired；
Data query is carried out in second database.
2. the method according to claim 1, wherein acquisition data variation set includes:
Obtain object run instruction, wherein the object run instruction, which is used to indicate, stores the target data set to institute
State the second database；
It is instructed according to the object run, determines the data variation set.
3. the method according to claim 1, wherein updating the second database packet using the data variation set
It includes:
Read all data in the data variation set, wherein read by the main cluster library in second database
The data variation set, the main cluster library are in second database, for synchronizing each of the target data set
The database of item data, second database include at least: a main cluster library, multiple spare distributed data bases, described
Main cluster library configures multiple scheduling processes, and the scheduling process is used for more new data；
All data in the data variation set is updated to second database.
4. the method according to claim 1, wherein progress data query includes: in second database
Prompt information is sent to the first database by preset interface, wherein the prompt information is for prompting described the
One database data, which updates, to be completed.
5. the method according to claim 1, wherein in second database carry out data query after,
Include:
Obtain target journaling data, wherein the target journaling data are to target user's list in second database
Data information carry out inquiry operation and/or carry out processing operation daily record data；
The target journaling data are stored to first database.
6. according to the method described in claim 5, it is characterized in that, before obtaining target journaling data, which comprises
Record the inquiry log that inquiry operation is carried out to the data information of target user's list in second database, and into
The processing log of row processing operation obtains inquiry log data and processing daily record data；
According to the inquiry log data and the processing daily record data, the target journaling data are determined, wherein the target
Daily record data is the inquiry log data that the spare distributed data base of second database is recorded and processing log number
According to.
7. a kind of data management system characterized by comprising
First database, for storing target data, wherein the target data is the data information of target user's list, institute
Stating target user is the user for recording prestige degree and being lower than predeterminable level；
Second database, the data information of target user's list for inquiring in main cluster library, and/or, handle main cluster library
In target user's list data information, obtain inquiry log data and processing daily record data；
Wherein, first database interval preset time updates the target data once stored, the master of second database
The target data of variation is generated in first database described in the synchronized update of cluster library, for user query and/or the processing mesh
Mark the data information of user list；
Wherein, the user is stored in second database inquire what the data information of target user's list obtained
Inquiry log data, and/or, it stores the user and carries out handling the processing day that the data information of target user's list obtains
Will data, to obtain target journaling data, the spare distributed data base of second database is by the target journaling data
It stores to the first database.
8. a kind of data query device characterized by comprising
Whether detection unit, the target data set for being spaced in predetermined time detection first database change, wherein
It include multiple target datas in the target data set, the target data is the data information of target user's list, described
Target user is the user for recording prestige degree and being lower than predeterminable level；
Acquiring unit obtains data variation if the target data set for detecting in the first database changes
Set, wherein include the set of changed data in the target data set in the data variation set；
Updating unit, for using the data variation set update the second database, wherein second database be for
The database that the data information of target user's list is inquired；
Query unit, for carrying out data query in second database.
9. device according to claim 8, which is characterized in that acquiring unit includes:
Module is obtained, for obtaining object run instruction, wherein the object run instruction is used to indicate the target data
Set is stored to second database；
Determining module determines the data variation set for instructing according to the object run.
10. device according to claim 8, which is characterized in that updating unit includes:
Read module, for reading all data in the data variation set, wherein by second database
The data variation set is read in main cluster library, and the main cluster library is in second database, for synchronizing the target
The database of all data of data acquisition system, second database include at least: a main cluster library, multiple spare distributions
Database, the main cluster library configure multiple scheduling processes, and the scheduling process is used for more new data；
Update module, for all data in the data variation set to be updated to second database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810806501.7A CN109063066B (en) | 2018-07-20 | 2018-07-20 | Data query method and device and data management system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810806501.7A CN109063066B (en) | 2018-07-20 | 2018-07-20 | Data query method and device and data management system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109063066A true CN109063066A (en) | 2018-12-21 |
| CN109063066B CN109063066B (en) | 2022-04-01 |
Family
ID=64835293
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810806501.7A Active CN109063066B (en) | 2018-07-20 | 2018-07-20 | Data query method and device and data management system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109063066B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110134648A (en) * | 2019-05-22 | 2019-08-16 | 中国联合网络通信集团有限公司 | Log processing method, device, equipment, system and computer readable storage medium |
| CN110765121A (en) * | 2019-10-31 | 2020-02-07 | 太原蓝知科技有限公司 | Intelligent big data acquisition and processing method and system |
| CN111177157A (en) * | 2020-01-02 | 2020-05-19 | 中国银行股份有限公司 | Account blacklist detection method and system |
| CN112818059A (en) * | 2021-01-27 | 2021-05-18 | 百果园技术(新加坡)有限公司 | Information real-time synchronization method and device based on container release platform |
| CN113821492A (en) * | 2020-06-19 | 2021-12-21 | 北京国电思达科技有限公司 | Redis-based fan data read-write method, device, system and medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102207937A (en) * | 2010-03-29 | 2011-10-05 | 华为技术有限公司 | Database access method and system |
| CN105205122A (en) * | 2015-09-11 | 2015-12-30 | 烽火通信科技股份有限公司 | Telecommunication service retrieval method based on data synchronization |
| CN105808643A (en) * | 2016-02-25 | 2016-07-27 | 浪潮通信信息系统有限公司 | Redis memory database refreshing method |
| CN106897303A (en) * | 2015-12-18 | 2017-06-27 | 北京国双科技有限公司 | Data query method and device |
| CN107122497A (en) * | 2017-05-25 | 2017-09-01 | 北京微影时代科技有限公司 | Data processing method, device, electronic equipment and computer-readable recording medium |
| CN107644071A (en) * | 2017-09-18 | 2018-01-30 | 广州爱九游信息技术有限公司 | Buffering updating method, device and data-storage system |
| US20180129654A1 (en) * | 2016-11-10 | 2018-05-10 | Palantir Technologies Inc. | System and methods for live data migration |
-
2018
- 2018-07-20 CN CN201810806501.7A patent/CN109063066B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102207937A (en) * | 2010-03-29 | 2011-10-05 | 华为技术有限公司 | Database access method and system |
| CN105205122A (en) * | 2015-09-11 | 2015-12-30 | 烽火通信科技股份有限公司 | Telecommunication service retrieval method based on data synchronization |
| CN106897303A (en) * | 2015-12-18 | 2017-06-27 | 北京国双科技有限公司 | Data query method and device |
| CN105808643A (en) * | 2016-02-25 | 2016-07-27 | 浪潮通信信息系统有限公司 | Redis memory database refreshing method |
| US20180129654A1 (en) * | 2016-11-10 | 2018-05-10 | Palantir Technologies Inc. | System and methods for live data migration |
| CN107122497A (en) * | 2017-05-25 | 2017-09-01 | 北京微影时代科技有限公司 | Data processing method, device, electronic equipment and computer-readable recording medium |
| CN107644071A (en) * | 2017-09-18 | 2018-01-30 | 广州爱九游信息技术有限公司 | Buffering updating method, device and data-storage system |
Non-Patent Citations (2)
| Title |
|---|
| PRAGYA KAUSHIK等: "《Use of query logs for providing cache support to the search engine》", 《IEEE》 * |
| 张旭刚等: "《基于zookeeper和强一致性复制实现MySQL分布式数据库集群》", 《微型电脑应用》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110134648A (en) * | 2019-05-22 | 2019-08-16 | 中国联合网络通信集团有限公司 | Log processing method, device, equipment, system and computer readable storage medium |
| CN110765121A (en) * | 2019-10-31 | 2020-02-07 | 太原蓝知科技有限公司 | Intelligent big data acquisition and processing method and system |
| CN110765121B (en) * | 2019-10-31 | 2022-09-20 | 上海意臣信息科技有限公司 | Intelligent big data acquisition and processing method and system |
| CN111177157A (en) * | 2020-01-02 | 2020-05-19 | 中国银行股份有限公司 | Account blacklist detection method and system |
| CN111177157B (en) * | 2020-01-02 | 2023-12-19 | 中国银行股份有限公司 | Account blacklist detection method and system |
| CN113821492A (en) * | 2020-06-19 | 2021-12-21 | 北京国电思达科技有限公司 | Redis-based fan data read-write method, device, system and medium |
| CN112818059A (en) * | 2021-01-27 | 2021-05-18 | 百果园技术(新加坡)有限公司 | Information real-time synchronization method and device based on container release platform |
| CN112818059B (en) * | 2021-01-27 | 2024-05-17 | 百果园技术(新加坡)有限公司 | Information real-time synchronization method and device based on container release platform |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109063066B (en) | 2022-04-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109063066A (en) | Data query method and device, data management system | |
| CN105843904B (en) | For the monitoring warning system of database runnability | |
| US10237295B2 (en) | Automated event ID field analysis on heterogeneous logs | |
| US6535728B1 (en) | Event manager for use in fraud detection | |
| CN112699175A (en) | Data management system and method thereof | |
| CN104750469B (en) | Source code statistical analysis technique and system | |
| CN107220142A (en) | Perform the method and device of data recovery operation | |
| CN111460023A (en) | Service data processing method, device, equipment and storage medium based on elastic search | |
| CN109189782A (en) | A kind of indexing means in block chain commodity transaction inquiry | |
| CN111897863A (en) | Multi-source heterogeneous data fusion and convergence method | |
| CN109254901B (en) | A kind of Monitoring Indexes method and system | |
| CN110457371A (en) | Data managing method, device, storage medium and system | |
| CN112199394A (en) | Alarm information pushing method and system, intelligent terminal and storage medium | |
| CN112181840A (en) | Method, device, equipment and storage medium for determining database state | |
| CN110297742A (en) | Data monitoring system, method and server | |
| CN109800133A (en) | A kind of method, one-stop monitoring alarm platform and the system of unified monitoring alarm | |
| CN109564610A (en) | Audit, which logs in, to be strengthened | |
| CN104636341B (en) | A kind of data cleansing storage method for the several monitoring of value-added tax No.1 | |
| CN112799868B (en) | Root cause determination method and device, computer equipment and storage medium | |
| CN106844497A (en) | The check device and method of a kind of database code | |
| CN115481026A (en) | Test case generation method and device, computer equipment and storage medium | |
| CN109903197A (en) | Case management system and its operating method | |
| CN109614416A (en) | A kind of invoice management method and device based on data statistic analysis | |
| CN111459737B (en) | Problem positioning method, device, computer equipment and storage medium | |
| CN113642857A (en) | Method, device and equipment for refreshing logistics data of management cockpit and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |