CN109063066B - Data query method and device and data management system - Google Patents
Data query method and device and data management system Download PDFInfo
- Publication number
- CN109063066B CN109063066B CN201810806501.7A CN201810806501A CN109063066B CN 109063066 B CN109063066 B CN 109063066B CN 201810806501 A CN201810806501 A CN 201810806501A CN 109063066 B CN109063066 B CN 109063066B
- Authority
- CN
- China
- Prior art keywords
- data
- database
- target
- log
- query
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Abstract
The invention discloses a data query method and device and a data management system. Wherein, the method comprises the following steps: detecting whether a target data set in a first database changes at preset time intervals, wherein the target data set comprises a plurality of target data, the target data is data information of a target user list, and the target user is a user with a recording credit degree lower than a preset degree; if the target data set in the first database is detected to be changed, acquiring a data change set, wherein the data change set comprises a set of changed data in the target data set; updating a second database by adopting a data change set, wherein the second database is used for inquiring the data information of the target user list; and carrying out data query in the second database. The invention solves the technical problems of low speed and reduced user experience for mass data query in the related technology.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a data query method and device and a data management system.
Background
In the related art, when querying data stored in a database, generally, all data stored in the database is queried according to the requirements of a user, and then the queried data is returned to the querying user. In the current data query, the query process can only query a small amount of data, and under the condition that a plurality of processes exist, the problems of high concurrent query speed, low query efficiency exist, and thus the experience of using the database by a user is low.
Aiming at the technical problems of low speed and reduced user experience of mass data query in the related technology, no effective solution is provided at present.
Disclosure of Invention
The embodiment of the invention provides a data query method, a data query device and a data management system, which are used for at least solving the technical problems of low speed and reduced user experience for mass data query in the related technology.
According to an aspect of an embodiment of the present invention, there is provided a data query method, including: detecting whether a target data set in a first database changes at preset time intervals, wherein the target data set comprises a plurality of target data, the target data is data information of a target user list, and the target user is a user with a recording credit degree lower than a preset degree; if the target data set in the first database is detected to be changed, acquiring a data change set, wherein the data change set comprises a set of changed data in the target data set; updating a second database by adopting the data change set, wherein the second database is used for inquiring data information of a target user list; and carrying out data query in the second database.
Further, obtaining the set of data changes comprises: acquiring a target operation instruction, wherein the target operation instruction is used for instructing the target data set to be stored in the second database; and determining the data change set according to the target operation instruction.
Further, updating the second database with the set of data changes includes: reading each item of data in the data change set, wherein the data change set is read through a master cluster library in the second database, the master cluster library is a database in the second database and used for synchronizing each item of data of the target data set, and the second database at least includes: the system comprises a main cluster library and a plurality of standby distributed databases, wherein the main cluster library is provided with a plurality of scheduling processes, and the scheduling processes are used for updating data; and updating each item of data in the data change set to the second database.
Further, the data query in the second database comprises: and sending prompt information to the first database through a preset interface, wherein the prompt information is used for prompting the completion of data updating of the first database.
Further, after the data query is performed in the second database, the method includes: acquiring target log data, wherein the target log data is log data for performing query operation and/or processing operation on data information of a target user list in the second database; and storing the target log data to a first database.
Further, prior to obtaining the target log data, the method includes: recording an inquiry log for carrying out inquiry operation on the data information of the target user list in the second database and a processing log for carrying out processing operation to obtain inquiry log data and processing log data; and determining the target log data according to the query log data and the processing log data, wherein the target log data are the query log data and the processing log data recorded by a standby distributed database of the second database.
According to another aspect of the embodiments of the present invention, there is also provided a data management system, including: the system comprises a first database, a first database and a second database, wherein the first database is used for storing target data, the target data are data information of a target user list, and the target user is a user with a recording credit degree lower than a preset degree; the second database is used for inquiring the data information of the target user list in the main cluster library and/or processing the data information of the target user list in the main cluster library to obtain inquiry log data and processing log data; the first database updates the target data stored once at intervals of preset time, and the master cluster library of the second database synchronously updates the changed target data in the first database so as to allow users to inquire and/or process data information of the target user list; the second database stores query log data obtained by querying the data information of the target user list by the user, and/or stores processing log data obtained by processing the data information of the target user list by the user to obtain target log data, and the standby distributed database of the second database stores the target log data to the first database.
According to another aspect of the embodiments of the present invention, there is also provided a data query apparatus, including: the device comprises a detection unit, a first database and a second database, wherein the detection unit is used for detecting whether a target data set in the first database changes at intervals of preset time, the target data set comprises a plurality of target data, the target data is data information of a target user list, and the target user is a user with a recording credit degree lower than a preset degree; the data change collection processing unit is used for acquiring a data change collection if the target data collection in the first database is detected to be changed, wherein the data change collection comprises a collection of changed data in the target data collection; the updating unit is used for updating a second database by adopting the data change set, wherein the second database is used for inquiring data information of a target user list; and the query unit is used for carrying out data query in the second database.
Further, the acquisition unit includes: an obtaining module, configured to obtain a target operation instruction, where the target operation instruction is used to instruct to store the target data set to the second database; and the determining module is used for determining the data change set according to the target operation instruction.
Further, the update unit includes: a reading module, configured to read each item of data in the data change set, where the data change set is read through a master cluster library in the second database, where the master cluster library is a database in the second database and used for synchronizing each item of data of the target data set, and the second database at least includes: the system comprises a main cluster library and a plurality of standby distributed databases, wherein the main cluster library is provided with a plurality of scheduling processes, and the scheduling processes are used for updating data; and the updating module is used for updating each item of data in the data change set to the second database.
Further, the query unit includes: and the sending module is used for sending prompt information to the first database through a preset interface, wherein the prompt information is used for prompting that the data updating of the first database is completed.
Further, comprising: the acquisition submodule is used for acquiring target log data after data query is carried out in the second database, wherein the target log data is log data for carrying out query operation and/or processing operation on data information of a target user list in the second database; and the storage module is used for storing the target log data to a first database.
Further, the apparatus further comprises: the recording module is used for recording an inquiry log for carrying out inquiry operation on data information of a target user list in the second database and a processing log for carrying out processing operation before target log data are obtained, so as to obtain inquiry log data and processing log data; and the determining submodule is used for determining the target log data according to the query log data and the processing log data, wherein the target log data are the query log data and the processing log data recorded by the standby distributed database of the second database.
According to another aspect of the embodiments of the present invention, there is also provided a storage medium, where the storage medium includes a stored program, and when the program runs, a device on which the storage medium is located is controlled to execute any one of the above data query methods.
According to another aspect of the embodiments of the present invention, there is also provided a processor, where the processor is configured to execute a program, where the program executes the data query method described in any one of the above.
In the embodiment of the invention, whether a target data set in a first database changes or not can be detected at preset time intervals, the target data set comprises a plurality of target data, the target data is data information of a target user list, and information of users with reputation degrees lower than a preset degree is recorded. In the embodiment, the data in the first database can be quickly updated to the second database, and the user can query the data information of the target user list in the second database, so that whether the user to be queried is on the target user list or not is determined, the data information of the target user list can be quickly updated in the second database, and the target user list in the second database can be quickly updated in time, so that a query operator can conveniently query the second database, a large amount of data does not need to enter the first database, only the data information of the target user list needs to be queried in the second database, and the quickly updated data information of the target user list enables the query to be more accurate, so that the technical problems that the query speed is low and the user experience is reduced for a large amount of data in the related technology are solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
FIG. 1 is a flow diagram of a data query method according to an embodiment of the invention;
FIG. 2 is a diagram illustrating a database association according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a data management system according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a data query device according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
To facilitate the understanding of the present invention, some terms or nouns related to the embodiments of the present invention are explained below:
an Oracle database, Oracle RDBMS, is a relational database management system, and the data storage is stable and convenient.
The Redis database is open-source log-type data which is written by using ANSI C language, supports network and can be based on memory persistence, and can realize rapid data query.
In accordance with an embodiment of the present invention, there is provided a method embodiment of data querying, it being noted that the steps illustrated in the flowchart of the drawings may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.
The following embodiments can be applied to various data processing environments, such as bank savings and postal savings, wherein the applied system is not limited, and the embodiments of the present invention take a transaction fraud management system as an example to describe data access and data storage so as to improve data access speed and data storage efficiency.
In the following, description is made in conjunction with preferred implementation steps, and fig. 1 is a flowchart of a data query method according to an embodiment of the present invention, as shown in fig. 1, the method includes the following steps:
step S102, detecting whether a target data set in a first database changes at preset time intervals, wherein the target data set comprises a plurality of target data, the target data is data information of a target user list, and the target user is a user with a recording credit degree lower than a preset degree.
The first database may be a database for storing a large amount of data, including but not limited to: the invention is exemplified by taking an Oracle database as a first database, but not limited to the Oracle database. In addition, the first database serves as a core of data storage, which may store not only data information of the target user list, but also other data information, including but not limited to: public management class data, batch processing data, information preprocessing data, rule engine data and interface service class data. The database with the first database as core storage and data exchange can comprise data processing class data, application data class data, rule related class data and basic data class data, wherein the data processing class data can comprise: scheduling configuration data, running log data, application data class data may include: organization information, teller information, risk information, customer information, report data, user list information, and for rule-related class data, the rule-related class data may include: the rule engine configuration data, the business rule data and the rule authority data can comprise the following data for basic data classes: teller incremental data, organization full data, customer incremental data, list incremental data, query log data and processing log data. The method and the device are mainly used for processing the data of the user list information in the application data class data, and particularly processing the data of which the credit degree is lower than the preset degree in the data of the user list information. Optionally, the present invention may define the user list information with a lower reputation degree by using a black and gray list, where the black and gray list includes a plurality of target user information, and the target user information may be understood as related information of a user with a low reputation degree and unable to perform service communication.
In addition, the target data set in the embodiment of the present invention may be the black and gray list, which includes related information of multiple users, and for data in the target data set, the data may be data that the user does not want to have service communication or come and go, the user may query the target data set through the second database, and at this time, the target data set stored in the first database needs to be updated to the second database in time, so as to facilitate the user query. When a user inquires, the second database is used for inquiring, the data in the second database can be consistent with the target data set stored in the first database, and at the moment, the changed data in the first database needs to be updated into the second database in time. The preset interval time may be time for updating data set by a user, for example, 5 minutes, 7 minutes, and 10 minutes, in the embodiment of the present invention, taking 5 minutes as an example, every 5 minutes, whether data in a target data set in each item of data stored in the first database changes is detected, and once the data changes, the changed data in the target set in the first database needs to be updated to the second database in time. In the embodiment of the present invention, two databases are used for illustration, where the second database may be a database used for fast query immediately, and the second database may temporarily store data that a user wants to query. Although a large amount of data is stored in the first database, the data that needs to be used by the query may be the target data set, i.e., the data in the query target data set. The target data set in the first database and the data in the second database may be the same. In the related art, the target data set in the first database may be a data set that changes at any time, for example, if the reputation level of a new user is found to be lower than a preset level, the related information of the new user may be input into the target data set, and the data in the target data set may include various types of data, including: user name, account number, identity card number.
It should be noted that, when the stored data is queried through the second database, the identification number or the user name may be input into the data set corresponding to the second database to query whether there is a user corresponding to the identification number in the black and gray list. The target data set in the first database may be understood as a first target data set, and the target data set in the second database may be understood as a second target data set. When updating, the updating base is the first target data set, and the second target data set in the second database needs to be updated in time so as to be convenient for query.
Step S104, if the target data set in the first database is detected to be changed, acquiring a data change set, wherein the data change set comprises a set of changed data in the target data set.
And step S106, updating a second database by adopting the data change set, wherein the second database is used for inquiring the data information of the target user list.
And step S108, inquiring data in the second database.
Optionally, the second database may be a database used for various queries, such as a Redis database, so as to quickly query data desired by a user, so that data query is convenient.
Through the steps, whether a target data set in the first database changes or not can be detected at preset time intervals, the target data set comprises a plurality of target data, the target data are data information of a target user list, information of users with reputation degrees lower than a preset degree is recorded, the changed data set is obtained under the condition that the target data set in the first database is detected to change, then the second database is updated by adopting the data change set, and finally data query can be carried out in the second database. In the embodiment, the data in the first database can be quickly updated to the second database, and the user can query the data information of the target user list in the second database, so that whether the user to be queried is on the target user list or not is determined, the data information of the target user list can be quickly updated in the second database, and the target user list in the second database can be quickly updated in time, so that a query operator can conveniently query the second database, a large amount of data does not need to enter the first database, only the data information of the target user list needs to be queried in the second database, and the quickly updated data information of the target user list enables the query to be more accurate, so that the technical problems that the query speed is low and the user experience is reduced for a large amount of data in the related technology are solved.
Optionally, obtaining the data change set includes: acquiring a target operation instruction, wherein the target operation instruction is used for indicating that a target data set is stored in a second database; and determining a data change set according to the target operation instruction.
Additionally, updating the second database with the set of data changes includes: reading each item of data in the data change set, wherein the data change set is read through a primary cluster library in a second database, the primary cluster library is a database used for synchronizing each item of data of a target data set in the second database, and the second database at least comprises: the system comprises a main cluster library and a plurality of standby distributed databases, wherein the main cluster library is provided with a plurality of scheduling processes, and the scheduling processes are used for updating data; and updating each item of data in the data change set to a second database.
The second database in the embodiment of the present invention may include a plurality of databases, including a main cluster database and a plurality of standby distributed databases, where the main cluster database may store a second target data set, that is, store black and gray list data, so that when querying, data in the main cluster database may be directly queried. If the main cluster library is abnormal, one database can be selected from the standby distributed databases to serve as the main cluster library, and selection can be performed based on a voting selection mechanism during selection, so that the main cluster library is determined. Optionally, the master cluster library in the embodiment of the present invention may have a plurality of guard processes (i.e., the scheduling process described above), and the guard processes may be processes for scheduling and coordinating data update and data query and data processing. The backup distributed database in the second database may also be used to generate a query log, which may be a log of a user querying the second target set in the second database, and a processing log, which may be a log of processing the query.
In addition, the guard process can also monitor the states of the main cluster library and the standby distributed database, and when the main cluster library is abnormal, a new main cluster library is determined in time through a voting selection mechanism, so that the safety of data service is ensured.
It should be noted that, in the embodiment of the present invention, when reading the data change set, the historical data set of the data change set, that is, the corresponding second target data set in the second database, may also be read.
Optionally, the querying the data in the second database includes: and sending prompt information to the first database through a preset interface, wherein the prompt information is used for prompting the completion of data updating of the first database.
For the above embodiment, it may be understood that after the synchronization between the second database and the target data set in the first database is completed, the first database is informed that the update is completed, that is, the synchronization process of the black and gray list data is completed.
After the data query is performed in the second database, the method includes: acquiring target log data, wherein the target log data is log data for performing query operation and/or processing operation on data information of a target user list in a second database; storing the target log data to a first database.
That is, the user can query the updated data and generate related target log data, the second database can temporarily record the target log data, and finally the generated target log data can be stored in the first database, so that the first database can be stably stored for subsequent query.
Before obtaining the target log data, the above embodiment further includes: recording an inquiry log for carrying out inquiry operation on data information of a target user list in a second database and a processing log for carrying out processing operation to obtain inquiry log data and processing log data; and determining target log data according to the query log data and the processing log data, wherein the target log data are the query log data and the processing log data recorded by a standby distributed database of the second database.
Optionally, the target log data may be read once every preset time when the log data is stored in the second database, where the preset time may be data set by a user, for example, 5 minutes, 10 minutes, and the like.
The query log and the processing log can be recorded in a highlight mode, query log data and processing log data are obtained, and the query log data and the processing log data are stored in the first database for storage. Optionally, in the present invention, the log data may be correspondingly provided with a log key value, where the log key value is used to identify a log state, for example, there is newly added log data, an information log key value may be provided, and data is added in the identification log, and when the data is stored in the first database, the log data may be read through the log key value, and finally, a log data synchronization process is completed, and the log data in the second database is synchronously stored in the first database.
The embodiment of the invention pays attention to the data query speed, considers the requirement of data persistence, realizes the perfect combination of dynamic query and static storage, and can meet the requirement of service system query without delay and the requirement of service data persistence storage. The method is a good success case for other systems in the row, and particularly provides a good reference for system examination of the business concurrence peak period such as twenty-one and twenty-two.
The invention will now be described with reference to another embodiment.
Fig. 2 is a schematic diagram of a database association relationship according to an embodiment of the present invention, and as shown in fig. 2, the database includes a core database (i.e., the first data described above), a Redis database (i.e., the second database described above), the Redis database includes a Redis master-slave cluster and a Redis distributed cluster, types of data stored in the core database may include data in a data processing class, data in an application data class, data in a rule related class, and data in a basic data class, where the data in the data processing class may include: scheduling configuration data, running log data, application data class data may include: organization information, teller information, risk information, customer information, report data, black and gray lists (i.e., the target data set described above), for rule-related class data, may include: the rule engine configuration data, the business rule data and the rule authority data can comprise the following data for basic data classes: teller incremental data, organization full data, customer incremental data, list incremental data, query log data and processing log data. And synchronously updating a black and grey list in a core database in a Redis master-slave cluster, recording admission configuration, query logs and processing logs in a distributed cluster in the Redis, and storing the recorded query logs and processing logs in the core database.
Optionally, in the embodiment of the present invention, the data may be public management data stored in the core database, data batch processed data stored in the core database, information preprocessing data stored in the core database, rule engine data stored in the core database, interface service data provided, and stored in the core database, the memory database is synchronized from the core database, the interface service log data is first entered into the memory database, and the core database is synchronized from the memory database at regular time.
The data relationship in the embodiment of the invention can be applied to a transaction fraud management system, and for the data architecture design in the embodiment of the invention, the data architecture of the transaction fraud management system needs to fully consider the operation mode of each subsystem on the basis of meeting IT planning. The application portal subsystem is mainly used for processing public management data, risk information data and statistical report data; the data processing subsystem mainly processes source data of each associated system and data synchronization in the system; the interface service subsystem mainly processes running data generated by risk inquiry transaction; the application monitoring subsystem mainly processes system operation log data.
The data storage of the transaction fraud management system adopts an Oracle database, a Redis master-slave cluster and a Redis distributed cluster to store data.
The transaction fraud management system adopts data stored in an Oracle database and mainly comprises the following parts: risk inquiry transaction flow, public management data, risk information data, statistical report data, full black and grey list data, rule engine related configuration data and terminal-of-day batch related configuration data.
The transaction fraud management system mainly comprises the following parts by adopting data of a Redis master-slave cluster: black-gray list snapshot data used for online transactions, i.e., data recorded at a brief point in time.
The data stored by the transaction fraud management system by adopting the Redis distributed cluster mainly comprises the following parts: risk queries deal-generated transaction flow data (i.e., temporary deposit data).
For the data storage life of the transaction fraud management system, the common period and the retention period of the data can be determined through the analysis of the data and the service, wherein the common period refers to the date interval in which the data can be frequently used in the online transaction, and the retention period refers to the longest date interval in which the data of the online transaction is stored according to the service specification. The history data refers to data exceeding a common period and not reaching a retention period. Historical data can only be queried and cannot be changed.
The typical period of various service data of the transaction fraud management system is preliminarily set as shown in table 1 below:
TABLE 1
According to the service characteristics of the transaction fraud management system, the data which needs to be transferred and stored into the historical database mainly comprises the following types: risk inquiry transaction flow and processing result data are stored in Redis; risk inquiry transaction flow and processing result data are stored in Oracle; the imported risk information data are stored in the Oracle; and the Oracle stores report statistical data. Historical data of transaction flow and processing results can only be inquired, and online risk behavior analysis is not allowed; risk information historical data can only be inquired and is not allowed to be modified; report statistics only allow querying data within one year.
Because the data volume of the history table is large, the query quantity and content of the history data must be strictly controlled, for example, the influence on the system performance is reduced by measures such as an authorization mechanism, explicit query conditions, limitation of background service processes, limitation of the number of queries each time and the like.
The data exchange among subsystems in the transaction fraud management system adopts a data synchronization mode. The data exchange between the transaction fraud management system and other systems is realized by adopting online transaction interfaces, data sharing system transmission and other modes. The online transaction interface is described in the specification of the postal savings bank transaction fraud management system and inline correlation system interface.
Data synchronization design
Firstly, the data of the synchronous list from the Oracle database to the Redis memory database comprises the following steps:
11. and the data synchronization process analyzes and stores the person blacklist incremental file into an Oracle database.
12. Polling to the warehoused incremental list data every 5 minutes through a timing program, generating an operation command for the Redis memory bank through a storage process, and putting the operation command into a corresponding data table, hereinafter referred to as 'T1'.
13. The polling program reads T1 and executes in the Redis memory database. Because the interface service only reads and does not update the list data in the Redis, the Redis is built in a main and standby cluster mode, the host is configured with two guard processes, and each slave is configured with one guard process. The list data is synchronized to each slave library through the master library. The guard process monitors the state of the master library and the slave library, when the state of the master library is abnormal, the guard process determines whether the master library is logically offline through a voting mechanism, and elects one slave library to be upgraded as the master library. Thereby ensuring the security of the data service.
14. And after the data processing process finishes Redis operation, updating the operation command state of T1 in the Oracle database, and finishing the synchronization flow of the list data.
Secondly, the synchronization of the log data from the Redis memory database to the Oracle database comprises the following steps:
21. the interface service records the associated system transaction risk query log into a Redis database. Because the interface service operates the Redis log library on the basis of writing, the Redis log library is deployed in a distributed cluster environment, a master host and a slave host are respectively deployed on each host, and the master host and the slave host are configured between the hosts of the cluster in a staggered mode.
22. The data handler polls the newly added log key value every 5 minutes by timing.
23. And the data processing program reads the log records in the Redis log library by adding a key value, and writes the log records into an Oracle database to finish the log data synchronization process.
In the embodiment, the query of the correlation system only takes about 10ms, so that the client basically feels no delay when doing transactions, and no delay and no influence on the production reaction speed of the business system are realized. The invention pays attention to the data query speed and also gives consideration to the requirement of data persistence. The perfect combination of dynamic query and static storage is realized, the requirement of no delay of service system query can be met, and the requirement of service data persistent storage can be met. The method is a good success case for other systems in the row, and provides a good reference for system examination of the business concurrence peak period of the double eleven and the double twelve.
Fig. 3 is a schematic diagram of a data management system according to an embodiment of the present invention, and as shown in fig. 3, the system may include: the first database 31 is used for storing target data, wherein the target data is data information of a target user list, and the target user is a user with a recording credit degree lower than a preset degree; the second database 33 is used for inquiring data information of a target user list in the main cluster library and/or processing the data information of the target user list in the main cluster library to obtain inquiry log data and processing log data; the method comprises the steps that a first database updates target data stored once at intervals of preset time, and a main cluster base of a second database synchronously updates the target data which changes in the first database so that a user can inquire and/or process data information of a target user list; the second database stores query log data obtained by querying the data information of the target user list by the user and/or stores processing log data obtained by processing the data information of the target user list by the user to obtain target log data, and the standby distributed database of the second database stores the target log data to the first database.
Through the data management system, the target data stored once can be updated in the first database at preset time intervals, the main cluster library of the second database synchronously updates the target data changed in the first database so as to allow a user to inquire and/or process the data information of the target user list, the second database stores inquiry log data obtained by inquiring the data information of the target user list by the user, and/or stores the processing log data obtained by processing the data information of the target user list by the user so as to obtain the target log data, the standby distributed database of the second database stores the target log data into the first database, in the embodiment, the data in the first database can be quickly updated into the second database, and the user can inquire the data information of the target user list in the second database, therefore, whether a user to be inquired is on the target user list or not is determined, the data information of the target user list can be quickly obtained in the second database, inquiring personnel can conveniently inquire the second database by quickly and timely updating the target user list in the second database without entering the first database to inquire a large amount of data, only the data information of the target user list needs to be inquired in the second database, and the quickly updated data information of the target user list enables the inquiry to be more accurate, so that the technical problems of low speed and reduced user experience of large-amount data inquiry in the related technology are solved.
Fig. 4 is a schematic diagram of a data query apparatus according to an embodiment of the present invention, and as shown in fig. 4, the apparatus may include: the detecting unit 41 is configured to detect whether a target data set in the first database changes at preset time intervals, where the target data set includes a plurality of target data, the target data is data information of a target user list, and the target user is a user whose record reputation degree is lower than a preset degree; an obtaining unit 43, configured to obtain a data change set if it is detected that a target data set in the first database changes, where the data change set includes a set of changed data in the target data set; an updating unit 45, configured to update a second database with the data change set, where the second database is a database used for querying data information of a target user list; and the query unit 47 is used for performing data query in the second database.
Through the above embodiment, it may be realized that whether a target data set in the first database changes at predetermined time intervals is detected by the detecting unit 41, where the target data set includes a plurality of target data, the target data is data information of a target user list, and records information of users with reputation degrees lower than a preset degree, and in a case that the target data set in the first database is detected to change, the changed data set may be acquired by the acquiring unit 43, then the second database may be updated by the updating unit 45 using the data change set, and finally, data query may be performed in the second database by the querying unit 47. In the embodiment, the data in the first database can be quickly updated to the second database, and the user can query the data information of the target user list in the second database, so that whether the user to be queried is on the target user list or not is determined, the data information of the target user list can be quickly updated in the second database, and the target user list in the second database can be quickly updated in time, so that a query operator can conveniently query the second database, a large amount of data does not need to enter the first database, only the data information of the target user list needs to be queried in the second database, and the quickly updated data information of the target user list enables the query to be more accurate, so that the technical problems that the query speed is low and the user experience is reduced for a large amount of data in the related technology are solved.
Optionally, the obtaining unit 43 may include: the acquisition module is used for acquiring a target operation instruction, wherein the target operation instruction is used for indicating that a target data set is stored in a second database; and the determining module is used for determining the data change set according to the target operation instruction.
In addition, the updating unit 45 may include: a reading module, configured to read each item of data in a data change set, where the data change set is read through a master cluster database in a second database, where the master cluster database is a database in the second database and used for synchronizing each item of data of a target data set, and the second database at least includes: the system comprises a main cluster library and a plurality of standby distributed databases, wherein the main cluster library is provided with a plurality of scheduling processes, and the scheduling processes are used for updating data; and the updating module is used for updating each item of data in the data change set to the second database.
The inquiring unit 47 in the inquiring apparatus described above may include: and the sending module is used for sending prompt information to the first database through a preset interface, wherein the prompt information is used for prompting the completion of data updating of the first database.
It should be noted that the above apparatus may further include: the acquisition submodule is used for acquiring target log data after data query is carried out in the second database, wherein the target log data is log data for carrying out query operation and/or processing operation on data information of a target user list in the second database; and the storage module is used for storing the target log data to the first database.
In addition, the apparatus may further include: the recording module is used for recording an inquiry log for carrying out inquiry operation on data information of a target user list in the second database and a processing log for carrying out processing operation before acquiring target log data to obtain inquiry log data and processing log data; and the determining submodule is used for determining target log data according to the query log data and the processing log data, wherein the target log data are the query log data and the processing log data recorded by the standby distributed database of the second database.
According to another aspect of the embodiments of the present invention, there is also provided a storage medium, where the storage medium includes a stored program, and when the program runs, a device on which the storage medium is located is controlled to execute any one of the above data query methods.
According to another aspect of the embodiments of the present invention, there is also provided a processor, where the processor is configured to execute a program, where the program executes the data query method described in any one of the above.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (8)
1. A method for querying data, comprising:
detecting whether a target data set in a first database changes at preset time intervals, wherein the target data set comprises a plurality of target data, the target data is data information of a target user list, and the target user is a user with a recording credit degree lower than a preset degree;
if the target data set in the first database is detected to be changed, acquiring a data change set, wherein the data change set comprises a set of changed data in the target data set;
updating a second database by adopting the data change set, wherein the second database is used for inquiring data information of a target user list;
performing a data query in the second database,
updating the second database with the set of data changes comprises:
reading each item of data in the data change set, wherein the data change set is read through a master cluster library in the second database, the master cluster library is a database in the second database and used for synchronizing each item of data of the target data set, and the second database at least includes: the system comprises a main cluster library and a plurality of standby distributed databases, wherein the main cluster library is provided with a plurality of scheduling processes, and the scheduling processes are used for updating data;
and updating each item of data in the data change set to the second database.
2. The method of claim 1, wherein obtaining a set of data changes comprises:
acquiring a target operation instruction, wherein the target operation instruction is used for instructing the target data set to be stored in the second database;
and determining the data change set according to the target operation instruction.
3. The method of claim 1, wherein querying data in the second database comprises:
and sending prompt information to the first database through a preset interface, wherein the prompt information is used for prompting the completion of data updating of the first database.
4. The method of claim 1, wherein after querying the second database, comprising:
acquiring target log data, wherein the target log data is log data for performing query operation and/or processing operation on data information of a target user list in the second database;
and storing the target log data to a first database.
5. The method of claim 4, wherein prior to obtaining the target log data, the method comprises:
recording an inquiry log for carrying out inquiry operation on the data information of the target user list in the second database and a processing log for carrying out processing operation to obtain inquiry log data and processing log data;
and determining the target log data according to the query log data and the processing log data, wherein the target log data are the query log data and the processing log data recorded by a standby distributed database of the second database.
6. A data management system, comprising:
the system comprises a first database, a first database and a second database, wherein the first database is used for storing target data, the target data are data information of a target user list, and the target user is a user with a recording credit degree lower than a preset degree;
the second database is used for inquiring the data information of the target user list in the main cluster library and/or processing the data information of the target user list in the main cluster library to obtain inquiry log data and processing log data;
the first database updates the target data stored once at intervals of preset time, and the master cluster library of the second database synchronously updates the changed target data in the first database so as to allow users to inquire and/or process data information of the target user list;
the second database stores query log data obtained by querying the data information of the target user list by the user, and/or stores processing log data obtained by processing the data information of the target user list by the user to obtain target log data, and the standby distributed database of the second database stores the target log data to the first database.
7. A data query apparatus, comprising:
the device comprises a detection unit, a first database and a second database, wherein the detection unit is used for detecting whether a target data set in the first database changes at intervals of preset time, the target data set comprises a plurality of target data, the target data is data information of a target user list, and the target user is a user with a recording credit degree lower than a preset degree;
the data change collection processing unit is used for acquiring a data change collection if the target data collection in the first database is detected to be changed, wherein the data change collection comprises a collection of changed data in the target data collection;
the updating unit is used for updating a second database by adopting the data change set, wherein the second database is used for inquiring data information of a target user list;
a query unit for performing data query in the second database,
the update unit includes:
a reading module, configured to read each item of data in the data change set, where the data change set is read through a master cluster library in the second database, where the master cluster library is a database in the second database and used for synchronizing each item of data of the target data set, and the second database at least includes: the system comprises a main cluster library and a plurality of standby distributed databases, wherein the main cluster library is provided with a plurality of scheduling processes, and the scheduling processes are used for updating data;
and the updating module is used for updating each item of data in the data change set to the second database.
8. The apparatus of claim 7, wherein the obtaining unit comprises:
an obtaining module, configured to obtain a target operation instruction, where the target operation instruction is used to instruct to store the target data set to the second database;
and the determining module is used for determining the data change set according to the target operation instruction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810806501.7A CN109063066B (en) | 2018-07-20 | 2018-07-20 | Data query method and device and data management system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810806501.7A CN109063066B (en) | 2018-07-20 | 2018-07-20 | Data query method and device and data management system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109063066A CN109063066A (en) | 2018-12-21 |
| CN109063066B true CN109063066B (en) | 2022-04-01 |
Family
ID=64835293
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810806501.7A Active CN109063066B (en) | 2018-07-20 | 2018-07-20 | Data query method and device and data management system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109063066B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110134648A (en) * | 2019-05-22 | 2019-08-16 | 中国联合网络通信集团有限公司 | Log processing method, device, equipment, system and computer readable storage medium |
| CN110765121B (en) * | 2019-10-31 | 2022-09-20 | 上海意臣信息科技有限公司 | Intelligent big data acquisition and processing method and system |
| CN111177157B (en) * | 2020-01-02 | 2023-12-19 | 中国银行股份有限公司 | Account blacklist detection method and system |
| CN113821492A (en) * | 2020-06-19 | 2021-12-21 | 北京国电思达科技有限公司 | Redis-based fan data read-write method, device, system and medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102207937A (en) * | 2010-03-29 | 2011-10-05 | 华为技术有限公司 | Database access method and system |
| CN105205122A (en) * | 2015-09-11 | 2015-12-30 | 烽火通信科技股份有限公司 | Telecommunication service retrieval method based on data synchronization |
| CN105808643A (en) * | 2016-02-25 | 2016-07-27 | 浪潮通信信息系统有限公司 | Redis memory database refreshing method |
| CN106897303A (en) * | 2015-12-18 | 2017-06-27 | 北京国双科技有限公司 | Data query method and device |
| CN107122497A (en) * | 2017-05-25 | 2017-09-01 | 北京微影时代科技有限公司 | Data processing method, device, electronic equipment and computer-readable recording medium |
| CN107644071A (en) * | 2017-09-18 | 2018-01-30 | 广州爱九游信息技术有限公司 | Buffering updating method, device and data-storage system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9805071B1 (en) * | 2016-11-10 | 2017-10-31 | Palantir Technologies Inc. | System and methods for live data migration |
-
2018
- 2018-07-20 CN CN201810806501.7A patent/CN109063066B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102207937A (en) * | 2010-03-29 | 2011-10-05 | 华为技术有限公司 | Database access method and system |
| CN105205122A (en) * | 2015-09-11 | 2015-12-30 | 烽火通信科技股份有限公司 | Telecommunication service retrieval method based on data synchronization |
| CN106897303A (en) * | 2015-12-18 | 2017-06-27 | 北京国双科技有限公司 | Data query method and device |
| CN105808643A (en) * | 2016-02-25 | 2016-07-27 | 浪潮通信信息系统有限公司 | Redis memory database refreshing method |
| CN107122497A (en) * | 2017-05-25 | 2017-09-01 | 北京微影时代科技有限公司 | Data processing method, device, electronic equipment and computer-readable recording medium |
| CN107644071A (en) * | 2017-09-18 | 2018-01-30 | 广州爱九游信息技术有限公司 | Buffering updating method, device and data-storage system |
Non-Patent Citations (2)
| Title |
|---|
| 《Use of query logs for providing cache support to the search engine》;Pragya Kaushik等;《IEEE》;20140630;第1-6页 * |
| 《基于zookeeper和强一致性复制实现MySQL分布式数据库集群》;张旭刚等;《微型电脑应用》;20160131;第77-80页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109063066A (en) | 2018-12-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109063066B (en) | Data query method and device and data management system | |
| US5469576A (en) | Front end for file access controller | |
| US7099897B2 (en) | System and method for discriminatory replaying of log files during tablespace recovery in a database management system | |
| US8280907B2 (en) | System and method for managing access to data in a database | |
| US8631042B2 (en) | Data warehouse system | |
| CN1459061A (en) | Automatic data update | |
| CN102346744B (en) | Device for processing materialized table in multi-tenancy (MT) application system | |
| CN101719149A (en) | Data synchronization method and device | |
| CN108647357B (en) | Data query method and device | |
| EP2534621A1 (en) | Method and system for e-commerce transaction data accounting | |
| CN111506559A (en) | Data storage method and device, electronic equipment and storage medium | |
| KR20150009798A (en) | System for online monitering individual information and method of online monitering the same | |
| CN106570163A (en) | Unreliable environment-oriented audit log read-write managing method and system | |
| CN111831755B (en) | Cross-database data synchronization method, system, medium and device | |
| CN114116762A (en) | Offline data fuzzy search method, device, equipment and medium | |
| CN110297846A (en) | A kind of log feature processing system, method, electronic equipment and storage medium | |
| CN111753015B (en) | Data query method and device of payment clearing system | |
| CN103778064A (en) | Data management method and device | |
| CN110334064A (en) | A kind of processing method and relevant apparatus of journal file | |
| CN111414382A (en) | Slow SQ L polymerization display method and system based on MongoDB | |
| CN115481026A (en) | Test case generation method and device, computer equipment and storage medium | |
| CN114398334A (en) | Prometheus remote storage method and system based on ZNBase cluster | |
| CN112035458B (en) | Data isolation method, device and equipment | |
| US8583500B2 (en) | Systems and methods for providing computing device counts | |
| CN108363617B (en) | Asynchronous importing method for offline list on SSR (simple sequence repeat) |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |