CN113961580A - Data query method, service system and electronic equipment - Google Patents
Data query method, service system and electronic equipment Download PDFInfo
- Publication number
- CN113961580A CN113961580A CN202111575535.8A CN202111575535A CN113961580A CN 113961580 A CN113961580 A CN 113961580A CN 202111575535 A CN202111575535 A CN 202111575535A CN 113961580 A CN113961580 A CN 113961580A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- changed
- query
- relational database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 64
- 230000008859 change Effects 0.000 claims abstract description 19
- 238000012544 monitoring process Methods 0.000 claims abstract description 12
- 238000012545 processing Methods 0.000 claims description 56
- 238000004590 computer program Methods 0.000 claims description 20
- 238000004140 cleaning Methods 0.000 claims description 17
- 238000001914 filtration Methods 0.000 claims description 7
- 230000006870 function Effects 0.000 description 16
- 238000010586 diagram Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 7
- 230000004048 modification Effects 0.000 description 7
- 238000012986 modification Methods 0.000 description 7
- 230000001360 synchronised effect Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 6
- 230000005012 migration Effects 0.000 description 4
- 238000013508 migration Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000000638 solvent extraction Methods 0.000 description 4
- 230000026676 system process Effects 0.000 description 4
- 238000013481 data capture Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 235000019800 disodium phosphate Nutrition 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000009545 invasion Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Abstract
The embodiment of the application provides a data query method, a service system and electronic equipment, wherein the service system comprises a relational database and a distributed database, and the method comprises the following steps: monitoring whether the data in the relational database are changed or not; if the data in the relational database are changed, capturing the changed data in the relational database to obtain changed data; synchronizing the change data into the distributed database; and acquiring input query information, and acquiring a query result from the distributed database according to the query information.
Description
Technical Field
The present application relates to the field of data query technologies, and in particular, to a data query method, a service system, and an electronic device.
Background
A Relational database (Relational database) is a database created on the basis of a Relational model (such as a two-dimensional table, etc.), and is generally used for storing business data in a business system, and a Structured Query Language (SQL) is used to perform data Query on the database to meet user requirements. However, with the increasing of the traffic volume, the processing pressure of the relational database in the business system increases day by day, and the problem of capacity expansion or reconstruction is faced.
In the prior art, a relational database is subjected to database partitioning and table partitioning, for example, data (or records and the like) is partitioned according to the dimensions of regions, time and the like, so that the data of an original table is dispersed in a plurality of tables according to the dimensions, and the problems are solved. However, the prior art has the disadvantages that after database splitting, the backend service involves multiple table processing, when data fields are changed (such as order data modification), the processing of historical data is complex, and for some hot spot problems, the user access amount is large, the processing pressure on a relational database is large, and the problem that large-flow access cannot be handled exists, along with further increase of the service amount, further database splitting is needed, so that the subsequent processing pressure is further increased, the system performance is poor, and the user experience is poor.
Disclosure of Invention
The application provides a data query method, a service system and electronic equipment, which can combine a relational database and a distributed database to provide a data query function so as to improve the processing capacity of a system, improve the performance of the system and increase the user experience.
In a first aspect, the present application provides a data query method, applied to a business system, where the business system includes a relational database and a distributed database, and the method includes:
monitoring whether the data in the relational database are changed or not;
if the data in the relational database are changed, capturing the changed data in the relational database to obtain changed data;
synchronizing the change data into the distributed database;
and acquiring input query information, and acquiring a query result from the distributed database according to the query information.
In one possible implementation manner, the monitoring whether data in the relational database is changed includes:
acquiring a received first service;
and processing the first service, and writing data generated by processing the first service into the relational database, so that the data in the relational database is changed.
In one possible implementation manner, the method further includes:
acquiring the received second service;
and processing the second service, and writing data generated by processing the second service into the distributed database.
In one possible implementation manner, the synchronizing the change data to the distributed database includes:
performing data cleaning on the changed data to obtain first data, wherein the data cleaning comprises data filtering and/or data merging;
and storing the first data into the first database.
In one possible implementation manner, the distributed database further includes a second database, and the method further includes:
generating second data according to the first data, wherein the second data is used for reversely indexing the first data;
and storing the second data into the second database.
In one possible implementation manner, the service system further includes a first buffer queue, and the method further includes:
after the changed data in the relational database are captured and changed data are obtained, the changed data are stored in the first cache queue;
and/or, the service system further includes a second buffer queue, and the method further includes:
and after second data is generated according to the first data, storing the second data into the second buffer queue.
In one possible implementation manner, the obtaining a query result from the distributed database according to the query information includes:
inquiring a first result from the second database according to the inquiry information;
if the first result meets the query condition, outputting the first result as a query result;
and if the first result does not meet the query condition, querying a second result from the first database according to the first result, and outputting the second result as a query result.
In a second aspect, the present application provides a business system, where the business system includes a relational database and a distributed database, and the business system further includes:
the data capturing module is used for monitoring whether the data in the relational database are changed or not, and capturing the changed data in the relational database if the data in the relational database are changed to obtain changed data;
a data synchronization module for synchronizing the change data to the distributed database;
and the query module is used for acquiring input query information and acquiring a query result from the distributed database according to the query information.
In a third aspect, the present application provides an electronic device, comprising:
one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs comprising instructions which, when executed by the apparatus, cause the apparatus to perform the method of the first aspect.
In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program which, when run on a computer, causes the computer to perform the method according to the first aspect.
In a fifth aspect, the present application provides a computer program for performing the method of the first aspect when the computer program is executed by a computer.
In a possible design, the program of the fifth aspect may be stored in whole or in part on a storage medium packaged with the processor, or in part or in whole on a memory not packaged with the processor.
Drawings
FIG. 1 is a schematic diagram of an embodiment of a data query method according to the present application;
FIG. 2 is a schematic structural diagram of query data according to an embodiment of the data query method of the present application;
FIG. 3 is a schematic diagram illustrating the association between a relational database and a distributed database according to an embodiment of the data query method of the present application;
FIG. 4 is a schematic structural diagram of an embodiment of a business system of the present application;
fig. 5 is a schematic structural diagram of an embodiment of an electronic device according to the present application.
Detailed Description
The terminology used in the description of the embodiments section of the present application is for the purpose of describing particular embodiments of the present application only and is not intended to be limiting of the present application.
With the increasing of the traffic, the processing pressure of the relational database in the business system is increasing day by day, and the problems of capacity expansion or transformation and the like are faced.
In the prior art, a relational database is subjected to database partitioning and table partitioning, for example, data (or records and the like) is partitioned according to the dimensions of regions, time and the like, so that the data of an original table is dispersed in a plurality of tables according to the dimensions, and the problems are solved. However, the prior art has the disadvantages that after database splitting, the backend service involves multiple table processing, when data fields are changed (such as order data modification), the processing of historical data is complex, and for some hot spot problems, the user access amount is large, the processing pressure on a relational database is large, and the problem that large-flow access cannot be handled exists, along with further increase of the service amount, further database splitting is needed, so that the subsequent processing pressure is further increased, the system performance is poor, and the user experience is poor.
Especially for a service system which has been operated for many years, the service relationship, the software architecture and the like are complex, the system performance meets a great bottleneck, new functions cannot be developed, and the system is difficult to upgrade or expand.
The applicant considers that the distributed database at least has the advantages of being suitable for single-point read-write operation of a large amount of data, relatively low in cost, suitable for storing massive obvious data, high in single-point read-write updating speed, high in efficiency and the like, for example, the Hbase, Cassandra and other distributed databases organized and built based on map or hash and other modes. Therefore, if the distributed database is adopted to store the service data and provide a data query function, the processing capacity of the system can be improved, the performance of the system can be improved, and the user experience can be improved.
However, the applicant also considers that if the relational database in the service system is directly replaced by the distributed database, developers need to redesign the table structure, write the interface, redevelop related functions and the like according to the characteristics of the distributed database, and old data stored in the relational database of the service system needs to be migrated into the distributed database, so that the workload is large, and some functions such as transactional characteristics, complex table association, instantaneity and the like cannot be met. Obviously, how to improve the processing capability of the system, improve the performance of the system, and increase the user experience degree under the condition of being compatible with the original relational database in the business system is one of the problems which need to be solved urgently at present.
Therefore, the data query method, the service system and the electronic device are provided, and can provide a data query function by combining the relational database and the distributed database, so as to improve the system processing capacity, improve the system performance and increase the user experience.
In this embodiment, the data query method shown in fig. 1 may be applied to a service system, and the service system may be used to process an order, such as creating an order, modifying an order, and the like, and provide a data query function. The business system can comprise a relational database and a distributed database, wherein the relational database comprises large databases such as oracle, sqlserver, db2, infomix, Sybase and the like, open databases such as MySQL, Postpresql and the like, and file type databases such as Assized, SQL Anywhere, sqlite, interbase and the like. The distributed database can comprise a distributed database organized based on map or hash and the like, such as Hbase, Cassandra and the like, or a distributed database organized by inverted index, such as elastic search, solr and the like.
The data query method provided by the application can be compatible with the original relational database in the service system, synchronizes the data in the relational database to the distributed database, and then queries the data from the distributed database, so that the processing capacity of the system is improved, the performance of the system is improved, and the user experience is improved.
Fig. 1 is a flowchart of an embodiment of a data query method of the present application, and as shown in fig. 1 to fig. 3, the data query method may include:
s101, monitoring whether the data in the relational database are changed or not.
In this embodiment, whether data in the relational database is changed or not is used to indicate whether new data is written in the relational database or not, whether old data in the relational database is changed or deleted, or the like. That is, if it is monitored that new data generated when the business system processes the business is written or recorded in the relational database, or old data in the relational database is changed or deleted, it indicates that the data in the relational database is changed. The relational database may include one or more tables, and when the service system processes a service (e.g., a newly created order service, etc.), the generated new data may be written or recorded in a corresponding row or column, etc. in a corresponding table in the relational database, or old data in a corresponding row or column, etc. in a corresponding table in the relational database may be modified or deleted, etc.
In step S101, a Change Data Capture technology (CDC) is used to monitor the Data in the relational database to determine whether the Data in the relational database is changed. The change data capturing technology is used for monitoring and capturing change data of a table in a relational database, such as a change data capturing technology based on a log, and the like.
In one possible implementation manner, step S101 may include:
s201, acquiring a received first service;
s202, processing the first service, and writing data generated by processing the first service into the relational database, so that the data in the relational database is changed.
In this embodiment, the service system may establish a communication connection with a user terminal, such as a mobile phone, a computer, and the like, and the service system may further include an application end, configured to receive a service (such as a first service or a second service, and the like) input by a user at the terminal, where the first service may include a new order service and the like. Therefore, when the service system processes a newly created order service, the generated data can be written or added into a corresponding row or column in a corresponding table of the relational database according to the set dimension (that is, the original dimension or rule in the relational database can be kept unchanged), that is, the data in the relational database is represented to be changed.
That is, for some relational databases that are native to the business system, their functionality may be preserved for writing data that is generated while processing the first business.
S102, if the data in the relational database are changed, capturing the changed data in the relational database to obtain changed data.
That is, by using the above-mentioned modified data capturing technique, if it is monitored that data in the relational database is modified, the corresponding modified data is captured. The changed data may be used to indicate that data generated during processing of the first service is written or added to a corresponding table of the relational database, and the like, which is changed (e.g., newly added).
S103, synchronizing the changed data to the distributed database.
That is, data generated when a new order service is processed can be directly written or added into the relational database, changed data is captured by using a changed data capturing technology (such as CDC) and the captured changed data is synchronized (or stored) into the distributed database in real time, so that the original relational database in the compatible service system is realized, no invasion to the service is achieved, and data migration is completed.
In one possible implementation manner, the method may further include:
s301, acquiring the received second service;
s302, the second service is processed, and data generated by processing the second service is written into the distributed database.
In this embodiment, the second service may include an order modification service, that is, when the service system processes the order modification service, the generated data is written or added to the distributed database, and is not required to be written or added to the relational database, so that the processing pressure of the relational database is reduced, the problem that the processing of the historical data is complicated when the order modification service (such as changing order data) is modified is avoided, and the system performance is improved.
It should be noted that, for data migration of old data or historical data stored in the relational database, the old data or the historical data may be directly backed up from the relational database to the distributed database to complete the data migration, or data or fields and the like may be inserted into a table of the relational database, and then the changed data may be captured by using the changed data capturing technique and synchronized to the distributed database to complete the data migration. In other words, for some old service systems including a relational database, a distributed database can be introduced into the old service system, and by adopting the method provided by the application, the data in the relational database is synchronized into the distributed database in real time, so that the real-time association between the relational database and the distributed database is realized, the original relational database is compatible, the loss on the function is avoided, the data query function is provided, the read-write separation is formed, and the user requirements are met.
That is, during the process of writing data, at least two write paths may be included, where the first write path is: and directly writing data generated when a service (such as a second service) is processed into a first database of the distributed database, then synchronizing the data from the first database into a second database, and completing the creation of the index data. The second write path is: data generated when a service (such as a first service) is processed is written into a relational database, and the data is synchronized into the first database and then synchronized into a second database from the first database by a CDC technique, and creation of index data is completed.
In the process of reading data, reading data from a second database in the distributed databases, if the read data meet the query conditions, outputting the data as a query result, and if not, reading detailed data from the first database and outputting the data as the query result.
And S104, acquiring input query information, and acquiring a query result from the distributed database according to the query information.
In this embodiment, the service system may include an inquiry module, which is connected to the application end, and is configured to acquire inquiry information input by a user at the application end and provide an inquiry service. That is to say, after the data in the relational database is synchronized or migrated to the distributed database, the corresponding query result can be read from the distributed database according to the query information, so that the problem that large-flow access cannot be performed due to large user access amount for some hot spot problems is avoided, the processing capability of the system is improved, the performance of the system is improved, and the user experience is improved.
In this embodiment, the distributed database may include a first database for storing first data, such as detail data generated during processing of the business, and a second database for storing second data, such as index data or a conditional query field for reversely indexing the first data.
The first database is suitable for single-point read-write operation of mass data, is relatively cheap, is suitable for storing mass detailed data, is high in single-point read-write updating speed and high in efficiency, and can comprise a distributed database and the like organized and established based on maps or hashes and the like, such as Hbase, Cassandra and the like. The second database is suitable for filtering, grouping of a large amount of data, and an analysis short board or the like which complements the first database, and may include a distributed database organized by inverted indexes such as an elastic search, solr or the like.
In one possible implementation manner, step S103 may include:
s401, performing data cleaning on the changed data to obtain first data, wherein the data cleaning includes but is not limited to data filtering and/or data merging and the like;
s402, storing the first data into the first database.
In this embodiment, the service system may include a data cleaning module, configured to perform data cleaning on the changed data, so as to filter invalid data or merge the same data by using a service layer as a core, thereby simplifying service data and achieving system lightweight. The data cleaning module may include a data cleaning channel, such as pipeline, which inputs the changed data and outputs the first cleaned data.
In one possible implementation manner, the method may further include:
s403, generating second data according to the first data, wherein the second data is used for reversely indexing the first data;
s404, storing the second data into the second database.
That is, the first database and the second database have the same primary key association, the first data in the first database and the second data in the second database are in one-to-one correspondence, and the corresponding first data can be queried through the second data. For example, the second data in the second database includes _ id, the key field (or data) in the relational database includes name, area, childId, etc., the _ id may be set to { area } _{ childId } _ name } in the second database, and the first data in the first database may set the primary key associated with the second database to ((area, childId), name).
In one possible implementation manner, the service system may further include a first buffer queue and a second buffer queue, and the method may further include:
s501, in the step S102, after the changed data in the relational database are captured and the changed data are obtained, storing the changed data into the first cache queue;
and/or, in step S502, after generating second data according to the first data in step S403, storing the second data into the second buffer queue.
In this embodiment, a first cache queue is provided between the relational database and the first database, and a second cache queue is provided between the first database and the second database, the cache queues being, for example, kafka or the like. For example, the changed data is extracted into a first cache queue in real time, the data partition is specified based on database and table hash, the correct time sequence of adding, deleting and modifying of the same table is guaranteed, due to the existence of the cache queue, support can be provided for subsequent multiple consumption, complex processing and the like, concurrence conditions can also be provided, and the extraction processing flow and the like are decoupled.
Further, the business system may include a plurality of relational databases, and capture change data in each relational database by using a change data capture technology to obtain a plurality of change data, where the plurality of change data are stored in the first cache queue, and then the plurality of change data are cleaned through the data cleaning channel (e.g., pipeline) to obtain first data, and the first data are stored in the first database. The data cleaning channel can perform cleaning, filtering, merging and the like on data.
In one possible implementation manner, step S104 may include:
s601, inquiring a first result from the second database according to the inquiry information;
s602, if the first result meets the query condition, outputting the first result as a query result;
s603, if the first result does not meet the query condition, querying a second result from the first database according to the first result, and outputting the second result as a query result.
For example, the query condition may include, but is not limited to, a field target length or a target number, and the first result may include target second data, which may be queried according to the query information. And if the length or the number of the field of the target second data in the first result is greater than or equal to the target length or the target number of the field, the first result meets the query condition, and the first result is directly output as a query result. And if the length or the number of the field of the target second data in the first result is less than the length or the number of the field target, the first result does not meet the query condition, querying a second result from the first database according to the first result, and outputting the second result as a query result. Specifically, the second result may include the target first data, so that the target first data corresponding to the target second data is obtained from the first database by indexing according to the target second data in the first result, and the target first data is output as a query result, thereby satisfying the user data query requirement.
It is to be understood that some or all of the steps or operations in the above-described embodiments are merely examples, and other operations or variations of various operations may be performed by the embodiments of the present application. Further, the various steps may be performed in a different order presented in the above-described embodiments, and it is possible that not all of the operations in the above-described embodiments are performed.
Fig. 4 is a schematic structural diagram of an embodiment of the service system 100 of the present application. As shown in fig. 4, the business system 100 includes a relational database 10 and a distributed database 20, and the business system 100 may further include:
the data capturing module 30 is configured to monitor whether data in the relational database is changed, and if the data in the relational database is changed, capture the changed data in the relational database to obtain changed data;
a data synchronization module 40 for synchronizing the change data to the distributed database;
and the query module 50 is configured to obtain input query information, and obtain a query result from the distributed database according to the query information.
In one possible implementation manner, the data capturing module 30 is further configured to:
acquiring a received first service;
and processing the first service, and writing data generated by processing the first service into the relational database, so that the data in the relational database is changed.
In one possible implementation manner, the service system 100 is further configured to:
acquiring the received second service;
and processing the second service, and writing data generated by processing the second service into the distributed database.
In one possible implementation manner, the distributed database 20 includes a first database 21, and the data synchronization module 40 is further configured to:
performing data cleaning on the changed data to obtain first data, wherein the data cleaning comprises data filtering and/or data merging;
and storing the first data into the first database.
In one possible implementation manner, the distributed database 20 further includes a second database 22, and the service system 100 is further configured to:
generating second data according to the first data, wherein the second data is used for indexing the first data;
and storing the second data into the second database.
In one possible implementation manner, the service system 100 further includes a first buffer queue 61, and the service system 100 is further configured to:
after the changed data in the relational database are captured and changed data are obtained, the changed data are stored in the first cache queue;
and/or, the service system 100 further includes a second buffer queue 62, and the service system 100 is further configured to:
and after second data is generated according to the first data, storing the second data into the second buffer queue.
In one possible implementation manner, the query module 50 is further configured to:
inquiring a first result from the second database according to the inquiry information;
if the first result meets the query condition, outputting the first result as a query result;
and if the first result does not meet the query condition, querying a second result from the first database according to the first result, and outputting the second result as a query result.
The service system 100 shown in fig. 4 may be configured to execute the data query method provided in the method embodiment shown in fig. 1, and specific functions or principles may refer to the method embodiment described above, which is not described herein again.
It should be noted that, in the service system 100 provided in this embodiment, data in the relational database is synchronized to the distributed database in real time, a secondary index is formed by introducing the first database and the second database of the distributed database, a data query function is provided, a new function can be added based on the relational database and the distributed database by using a modified data capture technology (CDC), integration of service data is completed, a service processing flow is simplified, a flexible read-write separation scheme is provided, and the architecture is flexible and efficient. Furthermore, the service system can introduce a real-time warehouse, provide real-time processing capacity of data in each dimension, provide a multi-dimension real-time report, arrange the data to form a wide table, provide a query interface, lighten the system, read the operation log successfully operated by a user into a cache queue at a client through the operable CDC of the self-defined distributed database, and solve the performance problem of native CDC-log.
It should be understood that the division of the modules of the business system 100 shown in fig. 4 is only a logical division, and the actual implementation may be wholly or partially integrated into one physical entity or may be physically separated. And these modules can be realized in the form of software called by processing element; or may be implemented entirely in hardware; and part of the modules can be realized in the form of calling by the processing element in software, and part of the modules can be realized in the form of hardware. For example, the query module may be a separate processing element, or may be integrated into a chip of the electronic device. Other modules are implemented similarly. In addition, all or part of the modules can be integrated together or can be independently realized. In implementation, each step of the above method or each module above may be implemented by an integrated logic circuit of hardware in a processor element or an instruction in the form of software.
For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more Application Specific Integrated Circuits (ASICs), one or more microprocessors (DSPs), one or more Field Programmable Gate Arrays (FPGAs), etc. For another example, these modules may be integrated together and implemented in the form of a System-On-a-Chip (SOC).
Fig. 5 is a schematic structural diagram of an embodiment of an electronic device according to the present application, and as shown in fig. 5, the electronic device may include: one or more processors; a memory; and one or more computer programs.
The electronic device may be a computer, a server, a data query device, or the like.
Wherein the one or more computer programs are stored in the memory, the one or more computer programs comprising instructions which, when executed by the apparatus, cause the apparatus to perform the steps of:
monitoring whether the data in the relational database are changed or not;
if the data in the relational database are changed, capturing the changed data in the relational database to obtain changed data;
synchronizing the change data into the distributed database;
and acquiring input query information, and acquiring a query result from the distributed database according to the query information.
In one possible implementation manner, when the instruction is executed by the device, the device may perform the monitoring whether the data in the relational database is changed, where the monitoring includes:
acquiring a received first service;
and processing the first service, and writing data generated by processing the first service into the relational database, so that the data in the relational database is changed.
In one possible implementation manner, when the instruction is executed by the apparatus, the apparatus is further caused to perform:
acquiring the received second service;
and processing the second service, and writing data generated by processing the second service into the distributed database.
In one possible implementation manner, the distributed database includes a first database, and when the instruction is executed by the apparatus, the apparatus executes the synchronization of the change data to the distributed database, including:
performing data cleaning on the changed data to obtain first data, wherein the data cleaning comprises data filtering and/or data merging;
and storing the first data into the first database.
In one possible implementation manner, the distributed database further includes a second database, and when the instruction is executed by the device, the device further performs:
generating second data according to the first data, wherein the second data is used for indexing the first data;
and storing the second data into the second database.
In one possible implementation manner, the service system further includes a first buffer queue, and when the instruction is executed by the device, the device further executes:
after the changed data in the relational database are captured and changed data are obtained, the changed data are stored in the first cache queue;
and/or the service system further comprises a second buffer queue, and when the instruction is executed by the device, the device further executes:
and after second data is generated according to the first data, storing the second data into the second buffer queue.
In one possible implementation manner, when the instruction is executed by the device, the device may execute the obtaining of the query result from the distributed database according to the query information, where the obtaining of the query result includes:
inquiring a first result from the second database according to the inquiry information;
if the first result meets the query condition, outputting the first result as a query result;
and if the first result does not meet the query condition, querying a second result from the first database according to the first result, and outputting the second result as a query result.
The electronic device shown in fig. 5 may be a server or a circuit device built in the server. The apparatus may be used to perform the functions/steps of the method provided by the embodiment of fig. 1 of the present application.
As shown in fig. 5, the electronic device 900 includes a processor 910 and a memory 920. Wherein, the processor 910 and the memory 920 can communicate with each other through the internal connection path to transmit control and/or data signals, the memory 920 is used for storing computer programs, and the processor 910 is used for calling and running the computer programs from the memory 920.
The memory 920 may be a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a Random Access Memory (RAM), or other types of dynamic storage devices that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, blu-ray disc, etc.), magnetic disc storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, etc.
The processor 910 and the memory 920 may be combined into a processing device, and more generally, independent components, and the processor 910 is configured to execute the program codes stored in the memory 920 to realize the functions. In particular implementations, the memory 920 may be integrated with the processor 910 or may be separate from the processor 910.
It should be understood that the electronic device 900 shown in fig. 5 is capable of implementing the processes of the methods provided by the embodiments shown in fig. 1 of the present application. The operations and/or functions of the respective modules in the electronic device 900 are respectively for implementing the corresponding flows in the above-described method embodiments. Reference may be made specifically to the description of the embodiment of the method illustrated in fig. 1 of the present application, and a detailed description is appropriately omitted herein to avoid redundancy.
It should be understood that the processor 910 in the electronic device 900 shown in fig. 5 may be a system on chip SOC, and the processor 910 may include a Central Processing Unit (CPU), and may further include other types of processors, such as: an image Processing Unit (hereinafter, referred to as GPU), and the like.
In summary, various portions of the processors or processing units within the processor 910 may cooperate to implement the foregoing method flows, and corresponding software programs for the various portions of the processors or processing units may be stored in the memory 930.
The application also provides an electronic device, the device includes a storage medium and a central processing unit, the storage medium may be a non-volatile storage medium, a computer executable program is stored in the storage medium, and the central processing unit is connected with the non-volatile storage medium and executes the computer executable program to implement the method provided by the embodiment shown in fig. 1 of the application.
In the above embodiments, the processors may include, for example, a CPU, a DSP, a microcontroller, or a digital Signal processor, and may further include a GPU, an embedded Neural Network Processor (NPU), and an Image Signal Processing (ISP), and the processors may further include necessary hardware accelerators or logic Processing hardware circuits, such as an ASIC, or one or more integrated circuits for controlling the execution of the program according to the technical solution of the present application. Further, the processor may have the functionality to operate one or more software programs, which may be stored in the storage medium.
Embodiments of the present application further provide a computer-readable storage medium, in which a computer program is stored, and when the computer program runs on a computer, the computer is enabled to execute the method provided by the embodiment shown in fig. 1 of the present application.
Embodiments of the present application also provide a computer program product, which includes a computer program, when the computer program runs on a computer, causing the computer to execute the method provided by the embodiment shown in fig. 1 of the present application.
In the embodiments of the present application, "at least one" means one or more, "a plurality" means two or more. "and/or" describes the association relationship of the associated objects, and means that there may be three relationships, for example, a and/or B, and may mean that a exists alone, a and B exist simultaneously, and B exists alone. Wherein A and B can be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. "at least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c may represent: a, b, c, a and b, a and c, b and c or a and b and c, wherein a, b and c can be single or multiple.
Those of ordinary skill in the art will appreciate that the various elements and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, any function, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only for the specific embodiments of the present application, and any person skilled in the art can easily conceive of modifications or substitutions within the technical scope of the present disclosure, and all such modifications or alterations should be covered by the protection scope of the present application. The protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A data query method is applied to a business system, and is characterized in that the business system comprises a relational database and a distributed database, and the method comprises the following steps:
monitoring whether the data in the relational database are changed or not;
if the data in the relational database are changed, capturing the changed data in the relational database to obtain changed data;
synchronizing the change data into the distributed database;
and acquiring input query information, and acquiring a query result from the distributed database according to the query information.
2. The method of claim 1, wherein the monitoring whether the data in the relational database is changed comprises:
acquiring a received first service;
and processing the first service, and writing data generated by processing the first service into the relational database, so that the data in the relational database is changed.
3. The method of claim 1, further comprising:
acquiring the received second service;
and processing the second service, and writing data generated by processing the second service into the distributed database.
4. The method of any of claims 1 to 3, wherein the distributed database comprises a first database, and wherein synchronizing the change data into the distributed database comprises:
performing data cleaning on the changed data to obtain first data, wherein the data cleaning comprises data filtering and/or data merging;
and storing the first data into the first database.
5. The method of claim 4, wherein the distributed database further comprises a second database, the method further comprising:
generating second data according to the first data, wherein the second data is used for reversely indexing the first data;
and storing the second data into the second database.
6. The method of claim 5, wherein the business system further comprises a first buffer queue, and wherein the method further comprises:
after the changed data in the relational database are captured and changed data are obtained, the changed data are stored in the first cache queue;
and/or, the service system further includes a second buffer queue, and the method further includes:
and after second data is generated according to the first data, storing the second data into the second buffer queue.
7. The method of claim 5, wherein obtaining query results from the distributed database according to the query information comprises:
inquiring a first result from the second database according to the inquiry information;
if the first result meets the query condition, outputting the first result as a query result;
and if the first result does not meet the query condition, querying a second result from the first database according to the first result, and outputting the second result as a query result.
8. A business system, comprising a relational database and a distributed database, the business system further comprising:
the data capturing module is used for monitoring whether the data in the relational database are changed or not, and capturing the changed data in the relational database if the data in the relational database are changed to obtain changed data;
a data synchronization module for synchronizing the change data to the distributed database;
and the query module is used for acquiring input query information and acquiring a query result from the distributed database according to the query information.
9. An electronic device, comprising:
one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs comprising instructions which, when executed by the apparatus, cause the apparatus to perform the method of any of claims 1-7.
10. A computer-readable storage medium, in which a computer program is stored which, when run on a computer, causes the computer to carry out the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111575535.8A CN113961580A (en) | 2021-12-22 | 2021-12-22 | Data query method, service system and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111575535.8A CN113961580A (en) | 2021-12-22 | 2021-12-22 | Data query method, service system and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113961580A true CN113961580A (en) | 2022-01-21 |
Family
ID=79473547
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111575535.8A Pending CN113961580A (en) | 2021-12-22 | 2021-12-22 | Data query method, service system and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113961580A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115062028A (en) * | 2022-07-27 | 2022-09-16 | 中建电子商务有限责任公司 | Method for multi-table join query in OLTP field |
| CN115809248A (en) * | 2022-12-30 | 2023-03-17 | 联通智网科技股份有限公司 | Data query method and device and storage medium |
| CN116069774A (en) * | 2023-04-03 | 2023-05-05 | 北京全路通信信号研究设计院集团有限公司 | Data cleaning method, device and medium based on wireless timeout intelligent analysis |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107679158A (en) * | 2017-09-28 | 2018-02-09 | 泰康保险集团股份有限公司 | Data managing method, device, computer-readable medium and electronic equipment |
| CN110019310A (en) * | 2017-12-29 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Data processing method and system, computer system, computer readable storage medium |
| CN110837520A (en) * | 2019-10-10 | 2020-02-25 | 苏宁云计算有限公司 | Data processing method, platform and system |
| CN111930817A (en) * | 2020-07-28 | 2020-11-13 | 银盛支付服务股份有限公司 | Big data-based distributed unstructured database correlation query method |
| US10963434B1 (en) * | 2018-09-07 | 2021-03-30 | Experian Information Solutions, Inc. | Data architecture for supporting multiple search models |
| CN113761053A (en) * | 2021-01-18 | 2021-12-07 | 北京京东拓先科技有限公司 | Data query method and device, electronic equipment and storage medium |
-
2021
- 2021-12-22 CN CN202111575535.8A patent/CN113961580A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107679158A (en) * | 2017-09-28 | 2018-02-09 | 泰康保险集团股份有限公司 | Data managing method, device, computer-readable medium and electronic equipment |
| CN110019310A (en) * | 2017-12-29 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Data processing method and system, computer system, computer readable storage medium |
| US10963434B1 (en) * | 2018-09-07 | 2021-03-30 | Experian Information Solutions, Inc. | Data architecture for supporting multiple search models |
| CN110837520A (en) * | 2019-10-10 | 2020-02-25 | 苏宁云计算有限公司 | Data processing method, platform and system |
| CN111930817A (en) * | 2020-07-28 | 2020-11-13 | 银盛支付服务股份有限公司 | Big data-based distributed unstructured database correlation query method |
| CN113761053A (en) * | 2021-01-18 | 2021-12-07 | 北京京东拓先科技有限公司 | Data query method and device, electronic equipment and storage medium |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115062028A (en) * | 2022-07-27 | 2022-09-16 | 中建电子商务有限责任公司 | Method for multi-table join query in OLTP field |
| CN115062028B (en) * | 2022-07-27 | 2023-01-06 | 中建电子商务有限责任公司 | Method for multi-table join query in OLTP field |
| CN115809248A (en) * | 2022-12-30 | 2023-03-17 | 联通智网科技股份有限公司 | Data query method and device and storage medium |
| CN115809248B (en) * | 2022-12-30 | 2024-03-22 | 联通智网科技股份有限公司 | Data query method and device and storage medium |
| CN116069774A (en) * | 2023-04-03 | 2023-05-05 | 北京全路通信信号研究设计院集团有限公司 | Data cleaning method, device and medium based on wireless timeout intelligent analysis |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109408551B (en) | Data query method and system, consensus method and system, device and storage medium | |
| CN113961580A (en) | Data query method, service system and electronic equipment | |
| US9767149B2 (en) | Joining data across a parallel database and a distributed processing system | |
| US10585913B2 (en) | Apparatus and method for distributed query processing utilizing dynamically generated in-memory term maps | |
| JP6553822B2 (en) | Dividing and moving ranges in distributed systems | |
| EP2743845B1 (en) | Graph traversal operator inside a column store | |
| US9195657B2 (en) | Columnar storage of a database index | |
| US11030196B2 (en) | Method and apparatus for processing join query | |
| US11194762B2 (en) | Spatial indexing using resilient distributed datasets | |
| WO2016192496A1 (en) | Data migration processing method and device | |
| US11782924B2 (en) | Distributed join index for shared-nothing and log-structured databases | |
| WO2014052202A2 (en) | Method and system for memory efficient, update optimized, transactional full-text index view maintenance | |
| US20230161765A1 (en) | System and method for disjunctive joins using a lookup table | |
| CN115552390A (en) | Server-free data lake indexing subsystem and application programming interface | |
| WO2016206100A1 (en) | Partitioned management method and apparatus for data table | |
| US20230205769A1 (en) | System and method for disjunctive joins | |
| CN115129738B (en) | Cross-database data writing method, device and equipment | |
| CN114416884A (en) | Method and device for connecting partition table | |
| Nidzwetzki et al. | BBoxDB streams: scalable processing of multi-dimensional data streams | |
| CN110297842B (en) | Data comparison method, device, terminal and storage medium | |
| US9183255B1 (en) | Spool management and checkpointing in a multi-database system | |
| CN113051274B (en) | Mass tag storage system and method | |
| US20240078221A1 (en) | Systems and methods of modeling and querying dynamic temporal graph on massive parallel graph processing and storage engine | |
| CN111753017B (en) | Method and device for processing dimension table based on Kylin system, electronic equipment and storage medium | |
| CN115422162A (en) | Industrial data storage system, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220121 |