CN109165257A - Data query method and related system, equipment and storage medium - Google Patents
Data query method and related system, equipment and storage medium Download PDFInfo
- Publication number
- CN109165257A CN109165257A CN201810890125.4A CN201810890125A CN109165257A CN 109165257 A CN109165257 A CN 109165257A CN 201810890125 A CN201810890125 A CN 201810890125A CN 109165257 A CN109165257 A CN 109165257A
- Authority
- CN
- China
- Prior art keywords
- data source
- data
- query
- inquiry request
- query result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of data query method and related systems, equipment and storage medium, wherein, the data query method selects a data source as primary data source in multiple data sources, specific method includes: the inquiry request for receiving client and sending, and determines data source according to the inquiry request；It is inquired in the data source, obtains query result；The query result is integrated in application service, and integrated results are back to the client.Method of the invention realizes the function that primary data source is selected according to different business scene by determining data source according to inquiry request, meanwhile, by integrating query result in application service, reduces the resource consumption of primary data source, improve access efficiency.
Description
Technical field
The present invention relates to information technology fields, more particularly, it is related to a kind of data query method and related system, sets
Standby and storage medium.
Background technique
In the operation system of some large sizes, such as banking establishments, e-commerce venture, support the backstage system of enterprise operation
In store a large amount of data record in system database.Data query service is the basic service that enterprise is supplied to client, and is mentioned
An important factor for rising customer experience, can effectively promote the powerful guarantee of horizontal competition power.But with the enterprise operation time
Explosive growth is presented in passage, information data, and the data volume in database can be very big, data query service facing big data,
The problems of multi-data source.The data inquiry request or even heap file read-write operation of high frequency will occupy more database
Resource and application server resource.If the business to take place frequently and data query service sharing application server resource and database provide
Source, high frequency inquiry will certainly tie up a large amount of system resources, slack-off so as to cause the data trade response time, increase system wind
Danger, reduces the experience of user.
When the record in system database is too many, common solution is periodically to migrate the data of the database
Into other databases.Wherein, the database of initial storage record is known as Production database, abbreviation storage facility located at processing plant；Storage is moved
The database for moving data is known as historical data base, abbreviation history library.After a part of Data Migration to history library in storage facility located at processing plant,
Data volume in storage facility located at processing plant is reduced, to improve the efficiency of access storage facility located at processing plant.
But in data query business, system not only needs often to access storage facility located at processing plant, sometimes according to different business demands,
The data being also required in access history library.It, will be multiple when data query is related to multiple databases in existing processing method
Database query result is deposited in primary database by the interim table of creation and is summarized, and then unifies to inquire obtaining from primary database
Required multi-data source data.But this method selects a master by database data amount size from multiple databases
Database is unsatisfactory for selecting the requirement of primary data source according to different business scene.Meanwhile the technical method needs to count other
It is deposited in the interim table of primary database according to the query result in library, for the big data quantity of high frequency inquires scene, primary database
Resource loss it is larger.
Summary of the invention
In order to solve the above-mentioned problems in the prior art, the present invention provides a kind of data query method and phase relations
System, equipment and storage medium, have the advantages that can flexibly switch data source, reduction primary data source resource consumption.
Embodiment according to the present invention, provides a kind of data query method, and the method is selected in multiple data sources
A data source is selected as primary data source, specific method includes: the inquiry request for receiving client and sending, and is asked according to the inquiry
Seek determining data source；It is inquired in the data source, obtains query result；The inquiry knot is integrated in application service
Integrated results are back to the client by fruit.
In certain embodiments of the present invention, described to determine that data source includes: according to according to the inquiry request
Inquiry request query configuration mapping table；If inquiring map record, saving scene is recorded and is switched to according to the map record
Corresponding data source.
In certain embodiments of the present invention, described to determine that data source includes: described in interception according to the inquiry request
Inquiry request is analyzed；Saving scene records and switches to corresponding data source based on the analysis results.
In certain embodiments of the present invention, the method also includes: if the data volume of the query result be greater than it is pre-
If threshold value, the query result is synchronized in the primary data source using data synchronous mode；The client query
Primary data source obtains the query result.
In certain embodiments of the present invention, the synchronous mode of the data includes Kafka, CDC or Dblink.
Embodiment according to the present invention, provides a kind of data query system, and the system is selected in multiple data sources
A data source is selected as primary data source, is specifically included: communication module, for receiving the inquiry request of client transmission；Data
Source determining module, for determining data source according to the inquiry request；Enquiry module, for being looked into the data source
It askes, obtains query result；Module is integrated, for integrating the query result in application service；Wherein, the communication module is also
For integrated results to be back to the client.
In certain embodiments of the present invention, the data source determining module comprises determining that unit, for according to
Inquiry request query configuration mapping table, if inquiring map record, saving scene is recorded and is switched to according to the map record
Corresponding data source.
In certain embodiments of the present invention, the system also includes data simultaneous modules, in inquiry knot
When the data volume of fruit is greater than preset threshold value, the query result is synchronized to the primary data source using data synchronous mode
In；The client query primary data source obtains the query result.
In certain embodiments of the present invention, the synchronous mode of the data includes Kafka, CDC or Dblink.
Meanwhile the present invention provides a kind of data query equipment, including memory and processor, the memory is for depositing
Store up one or more computer instruction；The processor is for calling one or more computer instruction thereby executing aforementioned
The data query method of any one.
The present invention also provides a kind of computer storage medium, it is stored with one or more computer program, described one
Or a plurality of computer program realizes any one of aforementioned data query method when calling.
Embodiments of the present invention are realized and are selected according to different business scene by determining data source according to inquiry request
The function of primary data source, meanwhile, by integrating query result in application service, reduces the resource consumption of primary data source, mention
High access efficiency.
Detailed description of the invention
To facilitate the understanding of the present invention, the present invention is carried out specifically below by way of specific embodiment and in conjunction with attached drawing
It is bright.
Fig. 1 is a kind of flow diagram of data query method of embodiment according to the present invention；
Fig. 2 is a kind of block diagram of data query system of embodiment according to the present invention.
Specific embodiment
It is described in detail to various aspects of the present invention below in conjunction with the drawings and specific embodiments.Wherein, many institute's weeks
Module, unit and its mutual connection, link, communication or the operation known are not shown or do not elaborate.Also, institute
Feature, framework or the function of description can combine in any way in one or more embodiments.Those skilled in the art
Member is it should be appreciated that following various embodiments are served only for the protection scope for example, and is not intended to limit the present invention.May be used also
To be readily appreciated that, module or unit or processing mode in each embodiment described herein and shown in the drawings can by it is various not
It is combined and designs with configuration.
Firstly, being explained to noun of the present invention:
Account transaction data base: the information such as contract information, real-time deal flowing water, real-time account balance of storage high frequency change
Database；
Information service primary database: store the data volume separated it is huge and do not rechange transaction journal, history close
The about database of the information such as record, historical account remaining sum；
Kafka: message system is subscribed to by the distributed post of Scala and a kind of high-throughput of written in Java, it can be with
Handle the everything flow data in the website of consumer's scale.This movement (web page browsing, the row of search and other users
It is dynamic) be many social functions on modern network a key factor.These data are often as the requirement of handling capacity
And it is solved by processing log and log aggregation.For the daily record data and off-line analysis system as Hadoop, but
The limitation handled when realistic again, this is a feasible solution.The purpose of Kafka is the loaded in parallel by Hadoop
Mechanism unifies Message Processing on line and offline, also for providing real-time message by cluster；
CDC:Changed Data Capture, data pick-up, extraction processing need emphasis consider increment extraction, also by
Referred to as delta data captures；
Dblink:Database Link, database link, as the term suggests it is exactly the link of database, just as telephone wire one
Sample is a channel, when we will be across local data base, when accessing the data in another database table, local data base
In just have to creation remote data base Dblink, can be as access local database by Dblink local data base
Access the data in remote data base table.
Fig. 1 is a kind of flow diagram of data query method of embodiment according to the present invention, in implementation of the invention
In mode, referring to Fig.1, this method selects a data source as primary data source in multiple data sources, specifically includes:
100: receiving the inquiry request that client is sent, data source is determined according to the inquiry request；
101: being inquired in the data source, obtain query result；
102: integrating the query result in application service, integrated results are back to the client.
In embodiments of the present invention, when application service receives the inquiry request of client initiation, application service system
System provides data query service according to request message.In application service logic, the conduct of On-line Information Service primary database is defaulted
Primary data source.According to specific inquiry request, determine which data source is inquiry data be located at.
Specifically, in embodiments of the present invention, before specific execution inquiry, application service intercepts the inquiry request,
And according to inquiry request query configuration mapping table.The query configuration mapping table is pre-configured in application service, has recorded data
Mapping relations between data source.If inquiring map record, saving scene is recorded and is switched to according to the map record
Corresponding data source is inquired.
Preferably, inquiry request can be Sql sentence, and application service is reflected by executing the SqlID query configuration of Sql sentence
Firing table finds out the automatic database that restores after the Sql needs the data source that accesses, inquiry to complete and connects default setting, citing and
Speech, specific logic are as follows:
By this way, developer only needs to safeguard the mapping table of database connection selector, and the selection of data source is cut
It changes, resetting movement is transparent operation.Whole system is easy to set up, maintenance is simple, and substantially increases search efficiency.
In embodiments of the present invention, while another method for determining data source is provided.Specifically, specifically holding
Before row inquiry, inquiry request is artificially intercepted, is analyzed it.Saving scene is recorded and is switched to corresponding based on the analysis results
Data source, inquiry restore database after completing automatically and connect default setting, and for example, specific logic is as follows:
In embodiments of the present invention, after the data source where inquiring data is confirmed by above two method, application
Service is switched to corresponding data source connection, carries out data query.If the data inquired are related to multiple data sources, by dividing
Query result is not obtained in corresponding data source, then the mode integrated in application service, by the query result of integration
It is back to client.In this way, reduce the resource consumption of primary data source, while improving access efficiency.
Optionally, in embodiments of the present invention, it is previously provided with the threshold value of data volume size.When being obtained in data source
When the data volume of the query result taken is greater than the threshold value, query result is synchronized to the master data using data synchronous mode
In the backup table in source, client passes through inquiry primary data source again and obtains query result, avoids occupying biggish application service memory
Resource optimizes customer experience.
Specifically, the mode that data synchronize can be realized using technologies such as Kafka, CDC, Dblink.
Embodiments of the present invention are realized and are selected according to different business scene by determining data source according to inquiry request
The function of primary data source, meanwhile, by integrating query result in application service, reduces the resource consumption of primary data source, mention
High access efficiency.
Fig. 2 is a kind of block diagram of data query system 1 of embodiment according to the present invention, referring to Fig. 2, the data query system
System 1 includes: communication module 11, for receiving the inquiry request of client transmission；Data source determining module 12, for according to
Inquiry request determines data source；Enquiry module 13 obtains query result for being inquired in data source；Module 14 is integrated,
For integrating the query result in application service；Wherein, the communication module 11 is also used to integrated results being back to visitor
Family end.
In embodiments of the present invention, data source determining module 12 includes determination unit 121, for according to inquiry request
Query configuration mapping table, if inquiring map record, saving scene records and switches to corresponding data source according to map record.
In embodiments of the present invention, which further includes data simultaneous module 15, for tying in inquiry
When the data volume of fruit is greater than preset threshold value, query result is synchronized in the primary data source using data synchronous mode；
Client query primary data source obtains the query result.The synchronous mode of data includes Kafka, CDC or Dblink.
Optionally, embodiment of the present invention provides a kind of data query equipment, which includes memory,
For storing one or more computer instruction；Processor, for call one or more computer instruction thereby executing
Data query method provided by aforementioned embodiments of the present invention or implementation.Optionally, the one of embodiment of the present invention
In kind implementation, the data query equipment can also include the input/output interface for carrying out data communication.For example, institute
Stating processing unit can be intelligent terminal, server etc..
Embodiment of the present invention additionally provides a kind of computer storage medium, is stored with one or more computer instruction,
The data query method provided for realizing aforementioned embodiments of the present invention or implementation when calling.For example, the storage
Medium may include hard disk, floppy disk, CD etc..
Although illustrating some embodiments herein, it, can be right under the premise of not departing from essence of the present invention
These embodiments carry out various modifications, and all these deformations still fall within design of the invention, and fall into right of the present invention and want
Protection scope defined by asking.
Specific embodiment disclosed herein is only used for illustrating the present invention, to those skilled in the art,
Obviously can carry out various modifications according to the teaching of this article, can using it is various it is equivalent by the way of implement the present invention, therefore, this hair
Bright specific embodiment disclosed above is only exemplary, and protection scope is not by construction or design disclosed herein
Details is limited, unless being otherwise noted in the claims.Therefore, specific illustrative embodiment disclosed above can carry out
Various replacements, combination or modification, all deformations are both fallen in range disclosed herein.It is not specifically disclosed herein lacking
Any element or in the case where lacking optional component disclosed herein, disclosed data query method and phase exemplified here
Relationship system, equipment and storage medium can be still appropriately carried out.All numerical value and range disclosed above can also centainly be become
Change.Whenever disclosing the numberical range with lower and upper limit, fall into any numerical value within the scope of this and it is any by comprising model
It encloses and is all specifically disclosed.Specifically, any range of numerical value disclosed herein can be regarded as listing be included in compared with
Any value and range in wide numberical range.Equally, unless applicant is explicitly and clearly defined otherwise, in claim
Term has their clear, common meaning.
Through the above description of the embodiments, those skilled in the art can be understood that the present invention can be by
The mode of software combination hardware platform is realized.Based on this understanding, technical solution of the present invention makes tribute to background technique
That offers can be embodied in the form of software products in whole or in part, which can store is situated between in storage
In matter, such as ROM/RAM, magnetic disk, CD, including some instructions use is so that a computer equipment (can be individual calculus
Machine, server or network equipment etc.) execute side described in certain parts of each embodiment of the present invention or embodiment
Method.
In addition, the quantity of the component in claims includes one or at least one, unless otherwise indicated.If this hair
Word or term in bright in other documents usage or meaning there are inconsistent, then should be to be with defined in the present invention
It is quasi-.
Claims (11)
1. a kind of data query method, the method selects a data source as primary data source in multiple data sources, special
Sign is, which comprises
The inquiry request that client is sent is received, data source is determined according to the inquiry request；
It is inquired in the data source, obtains query result；
The query result is integrated in application service, and integrated results are back to the client.
2. the method as described in claim 1, which is characterized in that described to determine that data source includes: according to the inquiry request
According to the inquiry request query configuration mapping table；
If inquiring map record, saving scene records and switches to corresponding data source according to the map record.
3. the method as described in claim 1, which is characterized in that described to determine that data source includes: according to the inquiry request
The inquiry request is intercepted to be analyzed；
Saving scene records and switches to corresponding data source based on the analysis results.
4. the method as described in claim 1, which is characterized in that the method also includes:
It is using data synchronous mode that the query result is synchronous if the data volume of the query result is greater than preset threshold value
Into the primary data source；
The client query primary data source obtains the query result.
5. method as claimed in claim 4, which is characterized in that
The synchronous mode of the data includes Kafka, CDC or Dblink.
6. a kind of data query system, the system selects a data source as primary data source in multiple data sources, special
Sign is, the system comprises:
Communication module, for receiving the inquiry request of client transmission；
Data source determining module, for determining data source according to the inquiry request；
Enquiry module obtains query result for being inquired in the data source；
Module is integrated, for integrating the query result in application service；
Wherein, the communication module is also used to for integrated results to be back to the client.
7. system as claimed in claim 6, which is characterized in that the data source determining module includes:
Determination unit, for according to the inquiry request query configuration mapping table, if inquiring map record, saving scene to be recorded
And corresponding data source is switched to according to the map record.
8. system as claimed in claim 6, which is characterized in that the system also includes:
Data simultaneous module when being greater than preset threshold value for the data volume in the query result, uses the synchronous side of data
The query result is synchronized in the primary data source by formula；
The client query primary data source obtains the query result.
9. system as claimed in claim 8, which is characterized in that
The synchronous mode of the data includes Kafka, CDC or Dblink.
10. a kind of data query equipment, including memory and processor, which is characterized in that
The memory is for storing one or more computer instruction；
The processor is for calling one or more computer instruction thereby executing any one of such as claim 1-5 institute
The data query method stated.
11. a kind of computer storage medium is stored with one or more computer program, which is characterized in that described one or more
Computer program executes data query method according to any one of claims 1 to 5 when calling.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810890125.4A CN109165257A (en) | 2018-08-07 | 2018-08-07 | Data query method and related system, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810890125.4A CN109165257A (en) | 2018-08-07 | 2018-08-07 | Data query method and related system, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109165257A true CN109165257A (en) | 2019-01-08 |
Family
ID=64895253
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810890125.4A Pending CN109165257A (en) | 2018-08-07 | 2018-08-07 | Data query method and related system, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109165257A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112732664A (en) * | 2020-12-31 | 2021-04-30 | 五八有限公司 | Evidence extraction method and device, electronic equipment and storage medium |
| CN113190578A (en) * | 2021-03-26 | 2021-07-30 | 有半岛(北京)信息科技有限公司 | Multi-source data query system, method, device, equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101908075A (en) * | 2010-08-17 | 2010-12-08 | 上海云数信息科技有限公司 | SQL-based parallel computing system and method |
| CN102637206A (en) * | 2012-03-21 | 2012-08-15 | 浪潮集团山东通用软件有限公司 | Method for inquiring mass data |
| CN105045877A (en) * | 2015-07-20 | 2015-11-11 | 深圳市深信服电子科技有限公司 | Database data fragmentation storage method and apparatus and data query method and apparatus |
| US20150324373A1 (en) * | 2014-05-09 | 2015-11-12 | Edward-Robert Tyercha | Querying Spatial Data in Column Stores Using Grid-Order Scans |
| CN106844405A (en) * | 2015-12-07 | 2017-06-13 | 杭州海康威视数字技术股份有限公司 | Data query method and apparatus |
-
2018
- 2018-08-07 CN CN201810890125.4A patent/CN109165257A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101908075A (en) * | 2010-08-17 | 2010-12-08 | 上海云数信息科技有限公司 | SQL-based parallel computing system and method |
| CN102637206A (en) * | 2012-03-21 | 2012-08-15 | 浪潮集团山东通用软件有限公司 | Method for inquiring mass data |
| US20150324373A1 (en) * | 2014-05-09 | 2015-11-12 | Edward-Robert Tyercha | Querying Spatial Data in Column Stores Using Grid-Order Scans |
| CN105045877A (en) * | 2015-07-20 | 2015-11-11 | 深圳市深信服电子科技有限公司 | Database data fragmentation storage method and apparatus and data query method and apparatus |
| CN106844405A (en) * | 2015-12-07 | 2017-06-13 | 杭州海康威视数字技术股份有限公司 | Data query method and apparatus |
Non-Patent Citations (1)
| Title |
|---|
| 李秀云 等: "《计算机在邮电管理中的应用》", 30 September 1999, 北京邮电大学出版社 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112732664A (en) * | 2020-12-31 | 2021-04-30 | 五八有限公司 | Evidence extraction method and device, electronic equipment and storage medium |
| CN112732664B (en) * | 2020-12-31 | 2024-04-05 | 五八有限公司 | Evidence extraction method, device, electronic equipment and storage medium |
| CN113190578A (en) * | 2021-03-26 | 2021-07-30 | 有半岛(北京)信息科技有限公司 | Multi-source data query system, method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108629029B (en) | Data processing method and device applied to data warehouse | |
| US10637990B1 (en) | Call center load balancing and routing management | |
| CN110008018A (en) | A kind of batch tasks processing method, device and equipment | |
| CN101937474A (en) | Mass data query method and device | |
| CN109558421A (en) | Data processing method, system, device and storage medium based on caching | |
| CN108650334A (en) | A kind of setting method and device of session failed | |
| CN107301214A (en) | Data migration method, device and terminal device in HIVE | |
| CN111429241A (en) | Accounting processing method and device | |
| CN111258978A (en) | Data storage method | |
| CN107480205A (en) | A kind of method and apparatus for carrying out data partition | |
| CN106022874A (en) | Order data processing method, order system, and flow charging system | |
| CN109087199A (en) | Account inquiries system, method and storage medium for bank | |
| CN107491549A (en) | A kind of data processing method and system | |
| CN109165257A (en) | Data query method and related system, equipment and storage medium | |
| CN114020444B (en) | Calling system and method for resource service application in enterprise digital middle station | |
| CN110321364B (en) | Transaction data query method, device and terminal of credit card management system | |
| CN106250501A (en) | Report processing method and reporting system | |
| EP2348676A1 (en) | Method for accessing magnanimity data of intelligent network service database and system and device thereof | |
| CN113297188A (en) | Data processing method and device | |
| Cruz et al. | Assessing nosql databases for telecom applications | |
| Tseng et al. | A successful application of big data storage techniques implemented to criminal investigation for telecom | |
| US11870706B2 (en) | Method and system for allocating and managing cloud resources | |
| CN114116908A (en) | Data management method and device and electronic equipment | |
| CN110750608A (en) | Spatial data analysis automatic scheduling method based on rule base | |
| CN117708219B (en) | Processing method, processing device and storage medium for data of Internet of things |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190108 |
|
| RJ01 | Rejection of invention patent application after publication |