CN106649828A - Data query method and system - Google Patents
Data query method and system Download PDFInfo
- Publication number
- CN106649828A CN106649828A CN201611248518.2A CN201611248518A CN106649828A CN 106649828 A CN106649828 A CN 106649828A CN 201611248518 A CN201611248518 A CN 201611248518A CN 106649828 A CN106649828 A CN 106649828A
- Authority
- CN
- China
- Prior art keywords
- data
- record
- engine
- query information
- query
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Embodiments of the invention disclose a data query method and system. The method comprises the steps of receiving a query request message sent by a business system; determining a target data engine corresponding to a to-be-queried data record according to query information of the to-be-queried data record and a corresponding relationship between the query information and the data engine; querying the data record in the target data engine; and obtaining the to-be-queried data record. The query information of the data record comprises a query concurrent quantity and a query condition quantity of the data record, and the query information and the data engine have the corresponding relationship, so that data records with different query concurrent quantities and query condition quantities can correspond to different data engines. The target data engine of the to-be-queried data record is determined according to the corresponding relationship between the query information and the data engine, and the data record in the target data engine is queried through the determined target data engine, so that the advantages of the different data engines can be fully utilized and the data query efficiency is effectively improved.
Description
Technical field
The present invention relates to technical field of data processing, more particularly to a kind of data query method and system.
Background technology
With the arrival in big data epoch, the data volume in each operation system is increasingly huge, and the inquiry application of big data becomes
Obtain more and more universal.Because search efficiency directly affects the response time of inquiry system, in the face of growing mass data, such as
What realizes that efficient, accurate, real-time data query has become industry major issue urgently to be resolved hurrily.
At present, in each operation system generally using relevant database come storage service data, but relevant database
Ability extending transversely, the high cost of dilatation, and be difficult to do distributed extension.When mass data is stored in data base, meeting
Occur because data base takes resource excessively, and when allowing to carry out data query, response relatively slow, the read and write access performance of data base
It is poor.
In order to solve this problem, in prior art, data can be sliced into different by carrying out subregion to data base
In storehouse and different tables, with the data of storing excess in the single table for avoiding data base.Even if however, having carried out to data base point
Area is processed, due to data are write and read when, need to process complicated point storehouse point table logic, when depositing in data base
During storage mass data, the quantity of table is excessive, and the access performance that still can cause data base is deteriorated, data query it is less efficient,
And make the management of data base and O＆M become very complicated.
To sum up, a kind of data query method is needed badly at present, the efficiency of the data query to improve.
The content of the invention
The present invention provides a kind of data query method and system, for solving prior art in data base access performance compared with
Difference, the relatively low technical problem of efficiency data query.
A kind of data query method provided in an embodiment of the present invention, including：
The inquiry request message that operation system sends is received, the inquiry request message includes looking into for data record to be checked
Inquiry information；The Query Information of the data record to be checked includes the corresponding inquiry concurrency of the data record to be checked and looks into
Inquiry condition quantity；
According to the inquiry concurrency and querying condition quantity and the Query Information sum of the data record to be checked
According to the corresponding relation of engine, it is determined that target data engine corresponding with the Query Information of the data record to be checked；
The data record inquired about in the target data engine, obtains the data record to be checked.
Alternatively, the data record in the target data engine is imported in the following manner：
The N datas record that on-line system sends is received, and by the N data record storages in data buffering
Area；
The Query Information of the N datas record and N datas record is obtained from the data buffer zone；
According to the Query Information of N datas record, and the corresponding relation of the Query Information and data engine,
By in the N datas record storage to the target data engine.
Alternatively, according to the Query Information of N datas record, and the Query Information and data engine is right
Should be related to, by the N datas record storage to the target data engine, including：
If the Query Information of the N datas record is inquiry concurrency more than or equal to default inquiry concurrency, and is looked into
Inquiry condition quantity is less than or equal to preset query condition threshold value, then according to the Query Information and the corresponding relation of data engine, will
Hbase data engines are defined as the target data engine, and by the N datas record storage to Hbase data engines
In；
If the Query Information of the N datas record is less than default inquiry concurrency, or inquiry bar for inquiry concurrency
Number of packages amount is more than or equal to preset query condition threshold value, then according to the Query Information and the corresponding relation of data engine, will
Impala data engines are defined as the target data engine, and the N datas record storage is drawn to Impala data
In holding up.
Alternatively, according to the Query Information of N datas record, and the Query Information and data engine is right
Should be related to, after in the N datas record storage to the target data engine, also include：
Receive the batch data record that the on-line system sends；
According to the Query Information of batch data record, determine that the batch data records corresponding target data and draws
Hold up；
To store in the target data engine with the batch data record corresponding data record replace with it is described
Batch data is recorded.
Alternatively, the target data engine includes M clustered node, and M is the integer more than or equal to 1.
Based on same inventive concept, the embodiment of the present invention further provides for a kind of data query system, including：
Receiver module, for receiving the inquiry request message of operation system transmission, the inquiry request message includes to be checked
Ask the Query Information of data record；The Query Information of the data record to be checked includes that the data record to be checked is corresponding
Inquiry concurrency and querying condition quantity；
Determining module, for according to the inquiry concurrency and querying condition quantity of the data record to be checked and described
The corresponding relation of Query Information and data engine, it is determined that target data corresponding with the Query Information of the data record to be checked
Engine；
Processing module, for inquiring about the target data engine in data record, obtain the data record to be checked.
Alternatively, the receiver module is additionally operable to：
The N datas record that on-line system sends is received, and by the N data record storages in data buffering
Area；
The processing module is additionally operable to：
The Query Information of the N datas record and N datas record is obtained from the data buffer zone；
And,
According to the Query Information of N datas record, and the corresponding relation of the Query Information and data engine,
By in the N datas record storage to the target data engine.
Alternatively, the processing module specifically for：
If the Query Information of the N datas record is inquiry concurrency more than or equal to default inquiry concurrency, and is looked into
Inquiry condition quantity is less than or equal to preset query condition threshold value, then according to the Query Information and the corresponding relation of data engine, will
Hbase data engines are defined as the target data engine, and by the N datas record storage to Hbase data engines
In；
If the Query Information of the N datas record is less than default inquiry concurrency, or inquiry bar for inquiry concurrency
Number of packages amount is more than or equal to preset query condition threshold value, then according to the Query Information and the corresponding relation of data engine, will
Impala data engines are defined as the target data engine, and the N datas record storage is drawn to Impala data
In holding up.
Alternatively, the receiver module is additionally operable to：
Receive the batch data record that the on-line system sends；
The processing module is additionally operable to：
According to the Query Information of batch data record, determine that the batch data records corresponding target data and draws
Hold up；
To store in the target data engine with the batch data record corresponding data record replace with it is described
Batch data is recorded.
Alternatively, the target data engine includes M clustered node, and M is the integer more than or equal to 1.
The embodiment of the present invention, by receiving the inquiry request message that operation system sends, wraps according in inquiry request message
The Query Information of the data record to be checked for containing, and the corresponding relation of Query Information and data engine, determine with it is to be checked
The corresponding target data engine of Query Information of data record, and then the data record in inquiry target data engine can be passed through,
Obtain data record to be checked.Because the Query Information of data record includes the inquiry concurrency and querying condition number of data record
Amount, and Query Information has corresponding relation with data engine, it is seen then that the inquiry concurrency data note different with querying condition quantity
Record can correspond to different data engines.Thus, according to inquiry concurrency and querying condition quantity and between data engine
Corresponding relation determines the target data engine of data record to be checked, then the target data engine by determining, inquires about mesh
Data record in mark data engine, can make full use of the advantage of different pieces of information engine, so as to effectively improve the effect of data query
Rate.
Description of the drawings
Technical scheme in order to be illustrated more clearly that the embodiment of the present invention, below will be to making needed for embodiment description
Accompanying drawing is briefly introduced, it should be apparent that, drawings in the following description are only some embodiments of the present invention, for this
For the those of ordinary skill in field, without having to pay creative labor, can be obtaining it according to these accompanying drawings
His accompanying drawing.
Fig. 1 is the schematic flow sheet corresponding to a kind of data query method in the embodiment of the present invention；
Fig. 2 is the schematic flow sheet corresponding to the real-time guiding flow of the data record in the embodiment of the present invention；
Fig. 3 is the schematic flow sheet corresponding to the batch data record guiding flow in the embodiment of the present invention；
Fig. 4 is a kind of structural representation of data query system in the embodiment of the present invention.
Specific embodiment
In order that the object, technical solutions and advantages of the present invention are clearer, below in conjunction with accompanying drawing the present invention is made into
One step ground is described in detail, it is clear that described embodiment, only a part of embodiment of the invention, rather than the enforcement of whole
Example.Based on the embodiment in the present invention, what those of ordinary skill in the art were obtained under the premise of creative work is not made
All other embodiment, belongs to the scope of protection of the invention.
Data query method in the embodiment of the present invention can be applicable to big data system, and the big data system includes one
Individual or multiple data engines, the data engine is recorded specifically for data storage, and the data record to storing is looked into
Inquiry etc. processes operation.
Specifically, the data engine can be polytype data engine, such as non-relational NoSQL data bases
Hbase data engines or MPP data base's Impala data engines, or other kinds of data engine, do not limit herein
System.
In the embodiment of the present invention, the big data system can be connected with one or more on-line systems.Wherein, it is described
Machine system can be polytype on-line system, and the content of its Business Processing for carrying out can be by those skilled in the art according to reality
Border needs to arrange, and is not limited herein.And, the on-line system can in real time be produced or more while Business Processing is carried out
Data record in new its source database.
Therefore, the process operation such as data query for carrying out in order to avoid big data system is affected at the business of on-line system
Reason, the data record in large database concept can be with the data record separate storage in the source database of on-line system.The present invention is implemented
In example, big data system can obtain the copy of data record from the source database of on-line system, to realize that data record is inquired about.
In the embodiment of the present invention, the big data system can be connected with one or more operation systems.Wherein, arbitrary business
System can be by way of corresponding interface sends inquiry request message in the big data system, in inquiry big data system
The data record for having stored.
The embodiment of the present invention is described in further detail with reference to Figure of description.
Fig. 1 is the schematic flow sheet corresponding to a kind of data query method provided in an embodiment of the present invention, as shown in figure 1,
Comprise the following steps 101 to step 103：
Step 101：The inquiry request message that operation system sends is received, the inquiry request message includes data to be checked
The Query Information of record；The Query Information of the data record to be checked includes the corresponding inquiry of the data record to be checked simultaneously
Send out amount and querying condition quantity；
Step 102：Inquiry concurrency and querying condition quantity and the inquiry according to the data record to be checked
The corresponding relation of information and data engine, it is determined that target data corresponding with the Query Information of the data record to be checked is drawn
Hold up；
Step 103：The data record inquired about in the target data engine, obtains the data record to be checked.
Because the Query Information of data record includes the inquiry concurrency and querying condition quantity of data record, and inquire about letter
Breath has corresponding relation with data engine, it is seen then that the inquiry concurrency data record different with querying condition quantity can be corresponded to not
Same data engine.Thus, the corresponding relation according to inquiry concurrency and querying condition quantity and between data engine is true
The target data engine of data record to be checked, then the target data engine by determining are made, target data engine is inquired about
In data record, the advantage of different pieces of information engine can be made full use of, so as to effectively improve the efficiency of data query.
Specifically, in step 201, big data system includes data query client, the data query client
With one or more interfaces.Thus, operation system can pass through connecing accordingly for the data query client of big data system
Mouth to data query service sends inquiry request message.
Wherein, the inquiry request message includes the Query Information of data record to be checked, i.e., described data to be checked
Corresponding inquiry concurrency and querying condition are recorded, to the service application scene for representing the data record to be checked.It is described
Operation system can be one or more, the interface of the corresponding data query client of different operation systems can with identical,
Can differ, not be limited herein.
Data query service is may also include in big data system, the data query service possesses data engine adaptation work(
Energy.Further, in step 202 and 203, data query service is received can be according to data record to be checked after inquiry request message
Query Information, and the corresponding relation of Query Information and data engine determines the number of targets that data record to be checked is located
According to engine.
Further, in step 103, big data system can pass through the corresponding access of the target data engine determined and connect
Mouthful, inquire about and obtain data record to be checked.
Because above-mentioned data record to be checked can be before operation system sends inquiry request message, according to data to be checked
Record the quantity that corresponding service application scene inquires about concurrency and querying condition, and service application scene and data engine
Corresponding relation, by data record to be checked store in corresponding target data engine, thus, can give full play to during inquiry
The advantage of different pieces of information engine, improves the efficiency of data query.
By taking the inquiry of transaction details data in Unionpay's system 1 year as an example, if be traded detail by card number and time looking into
Ask, 90% or so data inquiry request can be completed in 10ms, and 97% or so data inquiry request can be complete in 1s
Into.
In the embodiment of the present invention, big data system, can be with while providing data query for each operation system and servicing
The mode for importing in real time obtains data record from the source database of on-line system, and stores in target data engine.
After the big data system is connected with on-line system, big data system can be from the source database of on-line system
In import data record in real time.Due to data record it is generally all more, below with on-line system send N datas
As a example by record, the real-time guiding flow of data record is described in detail.
Fig. 2 is the schematic flow sheet in the embodiment of the present invention corresponding to the real-time guiding flow of data record, such as Fig. 2 institutes
Show, comprise the steps 201 to 203：
Step 201：The N datas record that on-line system sends is received, and the N datas record storage is being counted
According to relief area；
Step 202：Record from the data buffer zone acquisition N datas record and the N datas
Query Information；
Step 203：According to the Query Information of N datas record, and the Query Information and data engine
Corresponding relation, by the N datas record storage to the target data engine.
Specifically, big data system can receive the N datas record that on-line system sends by data access layer, and
N datas record is converted into into default data form.Because different on-line systems produces and sends real time data
The mode of record is different, and for the ease of the data record that big data system docking is received subsequent treatment is carried out, and big data system can
The data record received from different on-line systems is converted into into preset data form by corresponding adapter.The present invention is implemented
In example, the preset data form can according to actual needs be arranged by those skilled in the art, be not limited herein.
Because big data system receives possible with the speed of processing data record different, thus, the big data system will
The N datas record for receiving is converted into after preset data form, can be by N data record storages in data buffer zone
In, to improve the reliability of data record reception, and the reception and follow-up data record process of data record can be prevented effectively from
Between influence each other.
In the embodiment of the present invention, the data buffer zone can adopt the storage medium of persistence.Therefore, even if big data
There is exception in the processing procedure of data record in system, and data record will not lose, still can read from data buffer zone
Data record is processed again.
Further, since the speed of the generation data record in the on-line system being connected with the big data system may be compared with
Hurry up, and data volume is larger, thus, the data buffering layer should possess larger data throughout, such as, adopting can be horizontal
The data buffering layer of extension, by increasing node data-handling capacity is improved.
In step 202., the stream data process layer in the big data system can obtain described from data buffer zone
N datas are recorded, and the business need for recording correlation according to the N datas carries out corresponding data processing.
According to the complexity and the difference of time-consuming situation of data processing, the stream data process layer can adopt various
The processing structure of type.Such as, the stream data process layer can be concurrent using one process or process internal multi-thread
Processing structure, it would however also be possible to employ distributed, the real-time processing that cooperated using the multi-process of Stream Processing framework framework, is not done herein
Limit.
Further, in step 203, be able to will be located according to the Query Information of N datas record and the corresponding relation of data engine
In the target data engine in N datas record storage to data storage layer after reason.
Wherein, the target data engine is built by distributed computing technology, and with ability extending transversely, that is to say, that
The target data engine may include M clustered node, and M is the integer more than or equal to 1.When target data engine needs storage sea
In the case of the data record of amount, the quantity for increasing clustered node can be passed through to improve what whole big data system data was processed
Ability.For example, a small-sized Hbase cluster for including 60 nodes, can be used to managing the Unionpay 1 year comprising 36 fields
Transaction details data, and the inquiry service of excellent performance is externally provided.More data amount ask condition under, can further increase
The node of Hbase clusters, with the performance for supporting to inquire about service.
In the embodiment of the present invention, data storage layer includes multiple different types of data engines, due to different types of
Data engine stores different with the mode of inquiry data record, thus, the data-handling capacity of different types of data engine
Difference, the data record of different business application scenarios is respectively stored in different types of data engine, can make full use of number
According to the characteristic of engine, the efficiency of Import data records and inquiry is improved.
Specifically, the Query Information that big data system is recorded according to N datas, and the Query Information and data draw
The corresponding relation held up, can be by N datas record storage to corresponding, in target data engine.Wherein, the data record
Query Information includes inquiry concurrency and querying condition quantity, for representing the corresponding service application scene of the data record.
If the Query Information of the N datas record is inquiry concurrency more than or equal to default inquiry concurrency, and is looked into
Inquiry condition quantity is less than or equal to preset query condition threshold value, then according to the Query Information and the corresponding relation of data engine, will
Hbase data engines are defined as target data engine, and by the N datas record storage to Hbase data engines；
If the Query Information of the N datas record is less than default inquiry concurrency, or inquiry bar for inquiry concurrency
Number of packages amount is more than or equal to preset query condition threshold value, then according to the Query Information and the corresponding relation of data engine, will
Impala data engines are defined as target data engine, and by the N datas record storage to Impala data engines.
Wherein, the Query Information of the data record can be received for big data system from the source database of on-line system
, or big data system voluntarily determines according to the data record for receiving, is not limited herein；And, it is described pre-
If inquiry concurrency and the preset query condition threshold value can according to actual needs be arranged by those skilled in the art, herein not
It is limited.
It is pointed out that may also include query performance requirement in the Query Information.If the N datas record right
The query performance answered requires threshold value more than default capabilities, then can be according to query performance requirement and the corresponding relation of data engine, will
Hbase data engines are defined as target data engine, and by the N datas record storage to Hbase data engines.
For example, the high concurrent such as facing external holder, trade company, the inquiry data of high performance requirements in Unionpay's system,
As Unionpay CUPS clearance details are storable in being processed in Hbase data engines, and towards Internal Management System, querying condition
Abundant data are storable in being processed in Impala data engines, to give full play to the strong point of each data engine and excellent
Point.
In order to further improve the reliability that data record is imported in real time, the big data system is from the source of on-line system
Import the flow process of data record in data base in real time, may also include the process of distributed transaction, in the write of real-time data record
In the case of there is exception, transaction rollback can be carried out, and re-write data.
Data due to also there are some off lines processed by batch backstage upload in the source database of on-line system
Record, the data record of these off lines can not be imported in data engine by real-time guiding flow.Therefore, in order to realize number
According in each data engine of accumulation layer institute's data storage record not weigh do not leak, the big data system can also using day terminal hour from
The mode of batch data record is imported in on-line system, is filled or is wiped and lose because of network or system exception in real-time guiding flow
Lose or repeat the data record write.
Fig. 3 is the schematic flow sheet corresponding to the batch data record guiding flow in the embodiment of the present invention, such as Fig. 3 institutes
Show, batch data guiding flow comprises the steps 301 to 303：
Step 301：Receive the batch data record that the on-line system sends；
Step 302：According to the Query Information of batch data record, determine that the batch data records corresponding target
Data engine；
Step 303：Record what is stored in the target data engine corresponding data record with the batch data and replace
It is changed to the batch data record.
Specifically, in step 301, big data system can using ETL (Extract-Transform-Load) instruments from
Batch data record is extracted in the source database of on-line system, and by the batch data record storage to intermediate data storage area
In.Wherein, the ETL instruments can be polytype ETL instruments, for example, Informatica, Datastage etc., ability
Field technique personnel can according to actual needs select suitable ETL instruments, not be limited herein.
Correspondingly, the intermediate data storage area can also be polytype data storage area.In the embodiment of the present invention,
As a result of two kinds of data engines of Hbase and Impala, thus, intermediate data storage area can adopt and the data engine phase
The Hadoop distributed file systems of matching.Certainly, in the case where big data system adopts other data engines, intermediate data
Memory block may also be employed other kinds of data storage format, not be limited herein.
Subsequently, in step 302, the batch data for getting record can be processed conversion processing by big data system
Afterwards, concurrency and querying condition quantity, and the corresponding relation of Query Information and data engine are inquired about according to Query Information, from
Determine that the batch data records corresponding target data engine in multiple data engines in data storage, and by described batch
Amount data record store in corresponding target data engine, to replace the target data engine in original data record.
Specifically, for each data record in batch data record, the big data system can will be described each
Data record is processed, and according to its Query Information, that is, the quantity of concurrency and querying condition is inquired about, by described each
Data record is stored in corresponding target data engine.Wherein, the process change process of the batch data record includes right
The format analysis processing of batch data log file, and corresponding Business Processing is recorded to each data in batch data record.
It should be noted that the process performance in order to improve batch data guiding flow, big data system can be using simultaneously
Extraction and the processed process of formal layout batch data are sent out, thus, it is parallel using MapReduce in the embodiment of the present invention
Data processing shelf carries out the extraction of data and processed, effectively shortens the time that batch data is imported.
Based on same inventive concept, the embodiment of the present invention further provides a kind of data query system, such as Fig. 4 institutes
Show, the system 400 includes：
Receiver module 401, for receiving the inquiry request message of operation system transmission, the inquiry request message includes treating
The Query Information of inquiry data record；The Query Information of the data record to be checked includes the data record correspondence to be checked
Inquiry concurrency and querying condition quantity；
Determining module 402, for according to the inquiry concurrency and querying condition quantity of the data record to be checked and
The corresponding relation of the Query Information and data engine, it is determined that target corresponding with the Query Information of the data record to be checked
Data engine；
Processing module 403, for inquiring about the target data engine in data record, obtain the data to be checked note
Record.
Alternatively, the receiver module 401 is additionally operable to：
The N datas record that on-line system sends is received, and by the N data record storages in data buffering
Area；
The processing module 403 is additionally operable to：
The Query Information of the N datas record and N datas record is obtained from the data buffer zone；
And,
According to the Query Information of N datas record, and the corresponding relation of the Query Information and data engine,
By in the N datas record storage to the target data engine.
Alternatively, the processing module 403 specifically for：
If the Query Information of the N datas record is inquiry concurrency more than or equal to default inquiry concurrency, and is looked into
Inquiry condition quantity is less than or equal to preset query condition threshold value, then according to the Query Information and the corresponding relation of data engine, will
Hbase data engines are defined as the target data engine, and by the N datas record storage to Hbase data engines
In；
If the Query Information of the N datas record is less than default inquiry concurrency, or inquiry bar for inquiry concurrency
Number of packages amount is more than or equal to preset query condition threshold value, then according to the Query Information and the corresponding relation of data engine, will
Impala data engines are defined as the target data engine, and the N datas record storage is drawn to Impala data
In holding up.
Alternatively, the receiver module 401 is additionally operable to：
Receive the batch data record that the on-line system sends；
The processing module 403 is additionally operable to：
According to the Query Information of batch data record, determine that the batch data records corresponding target data and draws
Hold up；
To store in the target data engine with the batch data record corresponding data record replace with it is described
Batch data is recorded.
Alternatively, the target data engine includes M clustered node, and M is the integer more than or equal to 1.
By the above it can be seen that：
The embodiment of the present invention, by receiving the inquiry request message that operation system sends, wraps according in inquiry request message
The Query Information of the data record to be checked for containing, and the corresponding relation of Query Information and data engine, determine with it is to be checked
The corresponding target data engine of Query Information of data record, and then the data record in inquiry target data engine can be passed through,
Obtain data record to be checked.Because the Query Information of data record includes the inquiry concurrency and querying condition number of data record
Amount, and Query Information has corresponding relation with data engine, it is seen then that the inquiry concurrency data note different with querying condition quantity
Record can correspond to different data engines.Thus, according to inquiry concurrency and querying condition quantity and between data engine
Corresponding relation determines the target data engine of data record to be checked, then the target data engine by determining, inquires about mesh
Data record in mark data engine, can make full use of the advantage of different pieces of information engine, so as to effectively improve the effect of data query
Rate.
Those skilled in the art are it should be appreciated that embodiments of the invention can be provided as method, system or computer program
Product.Therefore, the present invention can be using complete hardware embodiment, complete software embodiment or with reference to the reality in terms of software and hardware
Apply the form of example.And, the present invention can be adopted and wherein include the meter of computer usable program code at one or more
The computer journey implemented in calculation machine usable storage medium (including but not limited to disk memory, CD-ROM, optical memory etc.)
The form of sequence product.
The present invention is the flow process with reference to method according to embodiments of the present invention, equipment (system) and computer program
Figure and/or block diagram are describing.It should be understood that can be by computer program instructions flowchart and/or each stream in block diagram
The combination of journey and/or square frame and flow chart and/or the flow process in block diagram and/or square frame.These computer programs can be provided
The processor of general purpose computer, special-purpose computer, Embedded Processor or other programmable data processing devices is instructed to produce
A raw machine so that produced for reality by the instruction of computer or the computing device of other programmable data processing devices
The function of specifying in present one flow process of flow chart or one square frame of two or more flow process and/or block diagram or two or more square frame
Device.
These computer program instructions may be alternatively stored in can guide computer or other programmable data processing devices with spy
In determining the computer-readable memory that mode works so that the instruction being stored in the computer-readable memory is produced to be included referring to
The manufacture of device is made, the command device is realized in one side of one flow process of flow chart or two or more flow process and/or block diagram
The function of specifying in frame or two or more square frame.
These computer program instructions also can be loaded in computer or other programmable data processing devices so that in meter
Series of operation steps is performed on calculation machine or other programmable devices to produce computer implemented process, so as in computer or
The instruction performed on other programmable devices is provided for realizing in one flow process of flow chart or two or more flow process and/or square frame
The step of function of specifying in one square frame of figure or two or more square frame.
, but those skilled in the art once know basic creation although preferred embodiments of the present invention have been described
Property concept, then can make other change and modification to these embodiments.So, claims are intended to be construed to include excellent
Select embodiment and fall into having altered and changing for the scope of the invention.
Obviously, those skilled in the art can carry out the essence of various changes and modification without deviating from the present invention to the present invention
God and scope.So, if these modifications of the present invention and modification belong to the scope of the claims in the present invention and its equivalent technologies
Within, then the present invention is also intended to comprising these changes and modification.
Claims (10)
1. a kind of data query method, it is characterised in that methods described includes：
The inquiry request message that operation system sends is received, the inquiry request message includes that the inquiry of data record to be checked is believed
Breath；The Query Information of the data record to be checked includes the corresponding inquiry concurrency of the data record to be checked and inquiry bar
Number of packages amount；
Drawn according to the inquiry concurrency and querying condition quantity and the Query Information and data of the data record to be checked
The corresponding relation held up, it is determined that target data engine corresponding with the Query Information of the data record to be checked；
The data record inquired about in the target data engine, obtains the data record to be checked.
2. method according to claim 1, it is characterised in that the data record in the target data engine be by with
What under type was imported：
The N datas record that on-line system sends is received, and by the N data record storages in data buffer zone；
The Query Information of the N datas record and N datas record is obtained from the data buffer zone；
According to the Query Information of N datas record, and the corresponding relation of the Query Information and data engine, by institute
N datas record storage is stated in the target data engine.
3. method according to claim 2, it is characterised in that according to the Query Information of N datas record, and
The corresponding relation of the Query Information and data engine, by the N datas record storage to the target data engine,
Including：
If the Query Information of the N datas record is inquiry concurrency more than or equal to default inquiry concurrency, and inquires about bar
Number of packages amount is less than or equal to preset query condition threshold value, then according to the Query Information and the corresponding relation of data engine, by Hbase
Data engine is defined as the target data engine, and by the N datas record storage to Hbase data engines；
If the Query Information of the N datas record is less than default inquiry concurrency, or querying condition number for inquiry concurrency
Amount is more than or equal to preset query condition threshold value, then according to the Query Information and the corresponding relation of data engine, by Impala numbers
It is defined as the target data engine according to engine, and by the N datas record storage to Impala data engines.
4. method according to claim 2, it is characterised in that according to the Query Information of N datas record, and
The corresponding relation of the Query Information and data engine, by the N datas record storage to the target data engine
Afterwards, also include：
Receive the batch data record that the on-line system sends；
According to the Query Information of batch data record, determine that the batch data records corresponding target data engine；
Record what is stored in the target data engine corresponding data record with the batch data and replace with the batch
Data record.
5. method according to any one of claim 1 to 4, it is characterised in that the target data engine includes M collection
Group node, M is the integer more than or equal to 1.
6. a kind of data query system, it is characterised in that the system includes：
Receiver module, for receiving the inquiry request message of operation system transmission, the inquiry request message includes number to be checked
According to the Query Information of record；The Query Information of the data record to be checked includes the corresponding inquiry of the data record to be checked
Concurrency and querying condition quantity；
Determining module, for according to the inquiry concurrency and querying condition quantity of the data record to be checked and the inquiry
The corresponding relation of information and data engine, it is determined that target data corresponding with the Query Information of the data record to be checked is drawn
Hold up；
Processing module, for inquiring about the target data engine in data record, obtain the data record to be checked.
7. system according to claim 6, it is characterised in that the receiver module is additionally operable to：
The N datas record that on-line system sends is received, and by the N data record storages in data buffer zone；
The processing module is additionally operable to：
The Query Information of the N datas record and N datas record is obtained from the data buffer zone；With
And,
According to the Query Information of N datas record, and the corresponding relation of the Query Information and data engine, by institute
N datas record storage is stated in the target data engine.
8. system according to claim 7, it is characterised in that the processing module specifically for：
If the Query Information of the N datas record is inquiry concurrency more than or equal to default inquiry concurrency, and inquires about bar
Number of packages amount is less than or equal to preset query condition threshold value, then according to the Query Information and the corresponding relation of data engine, by Hbase
Data engine is defined as the target data engine, and by the N datas record storage to Hbase data engines；
If the Query Information of the N datas record is less than default inquiry concurrency, or querying condition number for inquiry concurrency
Amount is more than or equal to preset query condition threshold value, then according to the Query Information and the corresponding relation of data engine, by Impala numbers
It is defined as the target data engine according to engine, and by the N datas record storage to Impala data engines.
9. system according to claim 7, it is characterised in that the receiver module is additionally operable to：
Receive the batch data record that the on-line system sends；
The processing module is additionally operable to：
According to the Query Information of batch data record, determine that the batch data records corresponding target data engine；
Record what is stored in the target data engine corresponding data record with the batch data and replace with the batch
Data record.
10. the system according to any one of claim 6 to 9, it is characterised in that the target data engine includes M
Clustered node, M is the integer more than or equal to 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611248518.2A CN106649828B (en) | 2016-12-29 | 2016-12-29 | Data query method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611248518.2A CN106649828B (en) | 2016-12-29 | 2016-12-29 | Data query method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106649828A true CN106649828A (en) | 2017-05-10 |
| CN106649828B CN106649828B (en) | 2019-12-24 |
Family
ID=58836655
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611248518.2A Active CN106649828B (en) | 2016-12-29 | 2016-12-29 | Data query method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106649828B (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107480268A (en) * | 2017-08-17 | 2017-12-15 | 北京奇虎科技有限公司 | Data query method and device |
| CN107993151A (en) * | 2018-01-17 | 2018-05-04 | 平安科技（深圳）有限公司 | Fund exchange settlement method, apparatus, equipment and computer-readable recording medium |
| CN108763300A (en) * | 2018-04-19 | 2018-11-06 | 北京奇艺世纪科技有限公司 | A kind of data query method and device |
| CN109033123A (en) * | 2018-05-31 | 2018-12-18 | 康键信息技术（深圳）有限公司 | Querying method, device, computer equipment and storage medium based on big data |
| CN109977140A (en) * | 2019-03-25 | 2019-07-05 | 中国农业银行股份有限公司 | A kind of transaction data querying method, apparatus and system |
| CN110866033A (en) * | 2018-08-28 | 2020-03-06 | 北京国双科技有限公司 | Feature determination method and device for predicting query resource occupancy |
| CN111159219A (en) * | 2019-12-31 | 2020-05-15 | 湖南亚信软件有限公司 | Data management method, device, server and storage medium |
| CN111190901A (en) * | 2019-12-12 | 2020-05-22 | 平安医疗健康管理股份有限公司 | Business data storage method and device, computer equipment and storage medium |
| CN111639078A (en) * | 2020-05-25 | 2020-09-08 | 北京百度网讯科技有限公司 | Data query method and device, electronic equipment and readable storage medium |
| CN112835717A (en) * | 2021-02-05 | 2021-05-25 | 远光软件股份有限公司 | Integrated application processing method and device for cluster |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070143273A1 (en) * | 2005-12-08 | 2007-06-21 | Knaus William A | Search engine with increased performance and specificity |
| CN102262662A (en) * | 2011-07-22 | 2011-11-30 | 浪潮(北京)电子信息产业有限公司 | System, device and method for realizing database data migration in heterogeneous platform |
| CN102831245A (en) * | 2012-09-17 | 2012-12-19 | 洛阳翔霏机电科技有限责任公司 | Real-time data storage and reading method of relational database |
| EP2693349A1 (en) * | 2012-08-03 | 2014-02-05 | Tata Consultancy Services Limited | A system and method for massive call data storage and retrieval |
| CN104102702A (en) * | 2014-07-07 | 2014-10-15 | 浪潮(北京)电子信息产业有限公司 | Software and hardware combined application-oriented big data system and method |
| CN104102710A (en) * | 2014-07-15 | 2014-10-15 | 浪潮(北京)电子信息产业有限公司 | Massive data query method |
| CN104361091A (en) * | 2014-11-18 | 2015-02-18 | 浪潮(北京)电子信息产业有限公司 | Big data system |
| CN105224651A (en) * | 2015-09-30 | 2016-01-06 | 国网天津市电力公司 | A kind of infosystem intranet and extranet database optimizing method based on read and write abruption |
| CN105574052A (en) * | 2014-11-06 | 2016-05-11 | 中兴通讯股份有限公司 | Database query method and apparatus |
-
2016
- 2016-12-29 CN CN201611248518.2A patent/CN106649828B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070143273A1 (en) * | 2005-12-08 | 2007-06-21 | Knaus William A | Search engine with increased performance and specificity |
| CN102262662A (en) * | 2011-07-22 | 2011-11-30 | 浪潮(北京)电子信息产业有限公司 | System, device and method for realizing database data migration in heterogeneous platform |
| EP2693349A1 (en) * | 2012-08-03 | 2014-02-05 | Tata Consultancy Services Limited | A system and method for massive call data storage and retrieval |
| US20140040292A1 (en) * | 2012-08-03 | 2014-02-06 | Tata Consultancy Services Limited | System and method for massive call data storage and retrieval |
| CN102831245A (en) * | 2012-09-17 | 2012-12-19 | 洛阳翔霏机电科技有限责任公司 | Real-time data storage and reading method of relational database |
| CN104102702A (en) * | 2014-07-07 | 2014-10-15 | 浪潮(北京)电子信息产业有限公司 | Software and hardware combined application-oriented big data system and method |
| CN104102710A (en) * | 2014-07-15 | 2014-10-15 | 浪潮(北京)电子信息产业有限公司 | Massive data query method |
| CN105574052A (en) * | 2014-11-06 | 2016-05-11 | 中兴通讯股份有限公司 | Database query method and apparatus |
| CN104361091A (en) * | 2014-11-18 | 2015-02-18 | 浪潮(北京)电子信息产业有限公司 | Big data system |
| CN105224651A (en) * | 2015-09-30 | 2016-01-06 | 国网天津市电力公司 | A kind of infosystem intranet and extranet database optimizing method based on read and write abruption |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107480268A (en) * | 2017-08-17 | 2017-12-15 | 北京奇虎科技有限公司 | Data query method and device |
| CN107993151B (en) * | 2018-01-17 | 2020-12-29 | 平安科技（深圳）有限公司 | Fund transaction clearing method, device, equipment and computer readable storage medium |
| CN107993151A (en) * | 2018-01-17 | 2018-05-04 | 平安科技（深圳）有限公司 | Fund exchange settlement method, apparatus, equipment and computer-readable recording medium |
| CN108763300A (en) * | 2018-04-19 | 2018-11-06 | 北京奇艺世纪科技有限公司 | A kind of data query method and device |
| CN108763300B (en) * | 2018-04-19 | 2020-07-31 | 北京奇艺世纪科技有限公司 | Data query method and device |
| CN109033123A (en) * | 2018-05-31 | 2018-12-18 | 康键信息技术（深圳）有限公司 | Querying method, device, computer equipment and storage medium based on big data |
| CN109033123B (en) * | 2018-05-31 | 2023-09-22 | 康键信息技术（深圳）有限公司 | Big data-based query method and device, computer equipment and storage medium |
| CN110866033A (en) * | 2018-08-28 | 2020-03-06 | 北京国双科技有限公司 | Feature determination method and device for predicting query resource occupancy |
| CN110866033B (en) * | 2018-08-28 | 2022-06-21 | 北京国双科技有限公司 | Feature determination method and device for predicting query resource occupancy |
| CN109977140A (en) * | 2019-03-25 | 2019-07-05 | 中国农业银行股份有限公司 | A kind of transaction data querying method, apparatus and system |
| CN109977140B (en) * | 2019-03-25 | 2022-04-05 | 中国农业银行股份有限公司 | Transaction data query method, device and system |
| CN111190901A (en) * | 2019-12-12 | 2020-05-22 | 平安医疗健康管理股份有限公司 | Business data storage method and device, computer equipment and storage medium |
| CN111190901B (en) * | 2019-12-12 | 2023-02-07 | 深圳平安医疗健康科技服务有限公司 | Business data storage method and device, computer equipment and storage medium |
| CN111159219A (en) * | 2019-12-31 | 2020-05-15 | 湖南亚信软件有限公司 | Data management method, device, server and storage medium |
| CN111639078A (en) * | 2020-05-25 | 2020-09-08 | 北京百度网讯科技有限公司 | Data query method and device, electronic equipment and readable storage medium |
| CN112835717A (en) * | 2021-02-05 | 2021-05-25 | 远光软件股份有限公司 | Integrated application processing method and device for cluster |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106649828B (en) | 2019-12-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106649828A (en) | Data query method and system | |
| CN103577440B (en) | A kind of data processing method and device in non-relational database | |
| CN102725753B (en) | Method and apparatus for optimizing data access, method and apparatus for optimizing data storage | |
| CN105989129B (en) | Real time data statistical method and device | |
| CN104794123B (en) | A kind of method and device building NoSQL database indexes for semi-structured data | |
| CN104794162B (en) | Real-time data memory and querying method | |
| CN106407207B (en) | Real-time newly-added data updating method and device | |
| CN107329983B (en) | Machine data distributed storage and reading method and system | |
| CN103793493B (en) | A kind of method and system for handling car-mounted terminal mass data | |
| CN103853714B (en) | A kind of data processing method and device | |
| CN110175154A (en) | A kind of processing method of log recording, server and storage medium | |
| CN104407879B (en) | A kind of power network sequential big data loaded in parallel method | |
| CN106874320A (en) | The method and apparatus of distributive type data processing | |
| Gupta et al. | Faster as well as early measurements from big data predictive analytics model | |
| CN102779138B (en) | The hard disk access method of real time data | |
| CN104584524A (en) | Aggregating data in a mediation system | |
| CN111258978A (en) | Data storage method | |
| CN106294826A (en) | A kind of company-data Query method in real time and system | |
| CN110083600A (en) | A kind of method, apparatus, calculating equipment and the storage medium of log collection processing | |
| CN104021088B (en) | log storing method and device | |
| CN109947729A (en) | A kind of real-time data analysis method and device | |
| CN110209714A (en) | Report form generation method, device, computer equipment and computer readable storage medium | |
| CN102968456A (en) | Method and device for reading and processing raster data | |
| CN107402982A (en) | Data write-in, data matching method, device and computing device | |
| CN110381136A (en) | A kind of method for reading data, terminal, server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |