CN110222072A - Data Query Platform, method, equipment and storage medium - Google Patents
Data Query Platform, method, equipment and storage medium Download PDFInfo
- Publication number
- CN110222072A CN110222072A CN201910492537.7A CN201910492537A CN110222072A CN 110222072 A CN110222072 A CN 110222072A CN 201910492537 A CN201910492537 A CN 201910492537A CN 110222072 A CN110222072 A CN 110222072A
- Authority
- CN
- China
- Prior art keywords
- inquiry
- data
- executive plan
- query
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24532—Query optimisation of parallel queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a kind of Data Query Platform, method, equipment and storage mediums, wherein, the Data Query Platform includes: that executive plan obtains module, for analyzing the corresponding query statement of inquiry request, determine the executive plan of the inquiry request, wherein, the executive plan from the pre- occupancy situation of resource of the different dimensions to inquiry request for being described；Query engine selecting module, for the attribute information according to the executive plan and at least two enforcement engines, selection target enforcement engine is inquired from least two enforcement engine；Query result obtains module, for obtaining the query result of the target enforcement engine.The embodiment of the present invention determines the executive plan of inquiry request by the analysis to query statement, and different inquiry requests is routed to different enforcement engines according to executive plan, ensures the stability of search efficiency and each engine cluster.
Description
Technical field
The present embodiments relate to big data technical field more particularly to a kind of Data Query Platform, method, equipment and deposit
Storage media.
Background technique
Currently, the business side of the offline cluster access of big data is increasing, there are offline search in the cluster in these business sides
The demand of data, moreover, hardware resource and human input cost are at growing trend.With the rapid growth of data volume and to inquiry
The increase of demand, sql that offline Data Query Platform calculates daily inquire search efficiency more than 10,000 times, to platform and surely
It is qualitative etc. to be proposed challenge.
In the prior art, searching platform mainly uses Hue (the Apache Hadoop UI system of an open source), should
Platform can be used in common big data and inquire scene, but it is not still high to the utilization rate of platform resource, influences looking into for data
Ask efficiency.
Summary of the invention
The embodiment of the present invention provides a kind of Data Query Platform, method, equipment and storage medium, to improve data query effect
Rate.
In a first aspect, the embodiment of the invention provides a kind of Data Query Platforms, comprising:
Executive plan obtains module and determines that the inquiry is asked for analyzing the corresponding query statement of inquiry request
The executive plan asked, wherein the executive plan from the pre- occupancy situation of resource of the different dimensions to inquiry request for retouching
It states；
Query engine selecting module, for the attribute information according to the executive plan and at least two enforcement engines, from
Selection target enforcement engine is inquired at least two enforcement engine；
Query result obtains module, for obtaining the query result of the target enforcement engine.
Second aspect, the embodiment of the invention also provides a kind of data query methods, comprising:
The corresponding query statement of inquiry request is analyzed, determines the executive plan of the inquiry request, wherein described
Executive plan from estimate occupation condition of the different dimensions to inquiry request for being described；
According to the attribute information of the executive plan and at least two enforcement engines, from least two enforcement engine
Selection target enforcement engine is inquired；
It indicates that the target enforcement engine handles the inquiry request, and obtains the inquiry knot of the target enforcement engine
Fruit.
The third aspect, the embodiment of the invention also provides a kind of computer equipments, comprising:
One or more processors；
Storage device, for storing one or more programs,
When one or more of programs are executed by one or more of processors, so that one or more of processing
Device realizes data query method described in any embodiment of the present invention.
Fourth aspect, the embodiment of the invention also provides a kind of computer readable storage mediums, are stored thereon with computer
Program realizes data query method described in any embodiment of the present invention when the program is executed by processor.
The embodiment of the present invention determines the execution meter of inquiry request by the analysis to the corresponding query statement of inquiry request
It draws, the pre- occupancy situation of the resource of inquiry request is estimated, to therefrom be selected in conjunction with the attribute information of different enforcement engines
Target enforcement engine handles the inquiry request, and realization offsets the different inquiry request in cost source and handled respectively, and inquiry is provided
Source is routed to different enforcement engines, and then ensures the stability of search efficiency and each engine cluster.
Detailed description of the invention
Fig. 1 is the structural schematic diagram of the Data Query Platform in the embodiment of the present invention one；
Fig. 2 is the structural schematic diagram of the Data Query Platform in the embodiment of the present invention two；
Fig. 3 is the structural schematic diagram of the Data Query Platform in the embodiment of the present invention three；
Fig. 4 is the flow chart of the data query method in the embodiment of the present invention four；
Fig. 5 is the structural schematic diagram of the computer equipment in the embodiment of the present invention five.
Specific embodiment
The present invention is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched
The specific embodiment stated is used only for explaining the present invention rather than limiting the invention.It also should be noted that in order to just
Only the parts related to the present invention are shown in description, attached drawing rather than entire infrastructure.
Embodiment one
Fig. 1 is the structural schematic diagram for the Data Query Platform that the embodiment of the present invention one provides, and the present embodiment is applicable to benefit
The case where inquiring off-line data with Data Query Platform.As shown in Figure 1, the platform 1 specifically include executive plan obtain module 10,
Query engine selecting module 11 and query result obtain module 12, in which:
Executive plan obtains module 10 and determines the inquiry for analyzing the corresponding query statement of inquiry request
The executive plan of request, wherein the executive plan is used to carry out from the pre- occupancy situation of resource of the different dimensions to inquiry request
Description.
Business side issues inquiry request to Data Query Platform according to demand, and inputs corresponding query statement, such as SQL
Query statement, query statement is naturally different according to different requirements,.It would generally be related to inquiring target, Ye Jiyao in query statement
What the data item of inquiry is, and a data item may be can be obtained by a tables of data, it is also possible to pass through multiple tables
It is obtained by different phase, wherein the tables of data is to be established in the database according to demand by technical staff based on big number
According to multiple tables of data, the data to be inquired of business side be namely based on these tables of data acquisition.Further, according to tables of data
The data volume of difference, inquiry is also different, also just different to the occupation condition for the enforcement engine for implementing inquiry.Therefore, pass through
Query statement is analyzed, determines the executive plan of inquiry request, namely preempt from resource of the different dimensions to inquiry request
With the description information of situation, realization may occupy that how many resource are estimated to inquiry request, thus next to ask to inquiry
Data preparation is done in the routing asked.
Preferably, executive plan obtains module 10 and is specifically used for:
The field in query statement is analyzed according to syntax rule, obtains the tables of data that inquiry is related to；
According to the size and incidence relation for inquiring tables of data in the tables of data being related to and database, looked into described in determination
Ask the executive plan of request；Wherein, the executive plan includes at least between the data volume of inquiry, the tables of data that inquiry is related to
The degree of parallelism of dependence and data operation.
Specifically, its syntax rule of different query statements is different, analyzed in query statement according to corresponding syntax rule
Field, the tables of data that is related to of inquiry can be obtained.The size of each tables of data and the incidence relation between them are in database
It is known, it therefore, may further be according to the size and incidence relation of tables of data in the tables of data and database that inquiry is related to, really
The executive plan for making inquiry request, including at least between the data volume of inquiry, the tables of data that is related to of inquiry dependence and
The degree of parallelism of data operation.
Wherein, the degree of parallelism of data operation can be estimated in conjunction with the data volume and some default rules to be inquired, together
When be also required to do between parallel expense and execution efficiency and weigh.For example, the request big for inquiry data volume, if opened
Multiple thread parallels execute the speed that may be inquired can be fast, it is contemplated that opening multiple threads needs certain overhead,
If expense is excessive, final inquiry velocity also will affect, therefore, suitable degree of parallelism can accelerate inquiry velocity, while again
It is avoided that excessive overhead, is than more preferably.
Query engine selecting module 11, for the attribute information according to the executive plan and at least two enforcement engines,
Selection target enforcement engine is inquired from least two enforcement engine.
Wherein, enforcement engine for example may include the data queries tool such as impala, presto, spark or hive, different
Its attribute of tool feature it is different, the efficiency of execution is also different, therefore, it is necessary to combine executive plan and respective attribute feature,
Most suitable target enforcement engine is selected, to improve search efficiency.For example, inquiry few to data volume, that resource consumption is few, it can
To be routed to impala quick search, to timely respond to user, the inquiry big for data volume, resource consumption is excessive can drop
Grade is inquired to hive, to ensure the search efficiency of data and the stability of system.It, can be in advance to executive plan when specific implementation
In include each dimension setting threshold value, and corresponding rule is set, to hold based on executive plan and the regular selection target
Row engine.
Preferably, query engine selecting module 11 is also used to:
According to the attribute information of the executive plan and at least two enforcement engines, at least two enforcement engine
Priority is ranked up；
According to the sequence of priority from high to low, the target enforcement engine is selected, and in target enforcement engine
When executing failure, the enforcement engine for selecting priority to take second place is continued with.
Specifically, can be handled in time when an error occurs to improve query success rate, enforcement engine can be taken
Degrade strategy.That is, first according to the attribute information of executive plan and at least two enforcement engines, at least two enforcement engines
Priority is ranked up, and selects the enforcement engine of the first priority as target enforcement engine, when event occurs for target enforcement engine
When barrier can not be inquired, then it can choose and the enforcement engine for indicating that priority is taken second place continue with, so that it is guaranteed that the success of inquiry
Rate.
Query result obtains module 12, for obtaining the query result of the target enforcement engine.
When specific implementation, enforcement engine can feed back the storage address of its query result or the write-in position of database to platform
It sets, module 12 is obtained according to the position acquisition query result by query result.User or third party can also be showed flat later
Platform.
In addition, as a preference, the Data Query Platform 1 can also be including query statement detection module (in figure not
Show), for giving a mark to the quality of the query statement, according to scoring according to syntax rule and preset standards of grading
As a result, can evaluate the business side for initiating inquiry, or improvement idea etc. is provided to business side.
The embodiment of the present invention determines the execution meter of inquiry request by the analysis to the corresponding query statement of inquiry request
It draws, the pre- occupancy situation of the resource of inquiry request is estimated, to therefrom be selected in conjunction with the attribute information of different enforcement engines
Target enforcement engine handles the inquiry request, and realization offsets the different inquiry request in cost source and handled respectively, and inquiry is provided
Source is routed to different enforcement engines, and then ensures the stability of search efficiency and each engine cluster.
Embodiment two
Fig. 2 is the structural schematic diagram of Data Query Platform provided by Embodiment 2 of the present invention, and the present embodiment two is in embodiment
Further optimized on the basis of one, as shown in Fig. 2, the present embodiment two is in each module basis that embodiment one describes,
Query statement correction verification module 13, library table parsing module 14, library exterior heat degree analysis module 15 and Authority Verification module 16 are increased,
In:
Query statement correction verification module 13, it is right for being obtained before module 10 analyzes query statement in executive plan
The legitimacy of the query statement is verified.
Specifically, can be by grammers such as the predicates of inspection query statement, to verify the legitimacy of query statement, for closing
The inquiry of method continues with, and illegal inquiry is directly returned.
Library table parsing module 14, for being obtained before module 10 analyzes query statement in executive plan, to described
Query statement carries out the parsing of library table, obtains the library and/or table that the inquiry request is related to.
Library exterior heat degree analysis module 15, for the parsing result based on library table parsing module 14, records each library and/or table
Inquiry times, and analyze according to the inquiry times inquiry temperature of library and/or table.
Whether Authority Verification module 16 judges the inquiry request for the parsing result based on library table parsing module 14
There is the search access right in the library and/or table that are related to it, and when judging to have permission, turns to obtain module by the executive plan
10 processing.
Specifically, which type of data business side will inquire, wherefrom inquire, which library and/or table be related to, can all lead to
It crosses query statement to embody, therefore library table parsing module 14 can be according to the syntax rules of different query languages from query statement
In parse the library and/or table that inquiry request is related to.For the parsing of library table mainly there are two purpose, one is to carry out library exterior heat degree point
Analysis, the other is carrying out Authority Verification.
Wherein, parsing result of the library exterior heat degree analysis module 15 based on library table parsing module 14, records each library and/or table
Inquiry times, and analyze according to the inquiry times inquiry temperature of library and/or table.Within equally long a period of time, look into
Library table its temperature ask often is higher, conversely, few library table its temperature of inquiry times is with regard to low.It is carried out by the temperature to library table
Analysis, can in time be adjusted library or table, to save memory space, increase operation rate.
Further, after parsing outbound table, the parsing result can also be based on by Authority Verification module 16, verification user is
The no permission for having inquiry related libraries table just turns to be handled by executive plan acquisition module 10, otherwise directly when judging to have permission
It returns, so that it is guaranteed that the safety of data, prevents leaking data.
The embodiment of the present invention is ensured on the basis of realizing to the routing of different inquiry requests by the verification to query statement
The success rate of inquiry may be implemented the analysis of library exterior heat degree and Authority Verification by the parsing of library table, provide data base to improve library table
Plinth, it may also be ensured that the safety of data, prevents leaking data.
Embodiment three
Fig. 3 is the structural schematic diagram for the Data Query Platform that the embodiment of the present invention three provides, and the present embodiment three is at above-mentioned
Further optimized on the basis of one embodiment, as shown in figure 3, the present embodiment three described in any of the above-described embodiment it is each
In module basis, inquiry control module 17, log analysis module 18 and result display module 19 are increased, in which:
Inquiry control module 17, for obtaining the execution state information of enforcement engine, according to the execution state information to looking into
Request is ask to be managed；Wherein, the execution state information includes at least and executes time threshold, executes occupation condition and hold
Row queue resource residue situation；The control, which includes at least, stops inquiry or sending alarm.
It is well known that the inquiry of big data quantity is the common challenge that big data searching platform encounters, then, by time with
The resource consumption situation of track, monitoring and inquiry, the inquiry excessive to resource consumption carry out automatic degrade and even cancel, then can ensure
Entirely cluster is stable.Therefore, in embodiments of the present invention, module 17 is managed by inquiry, obtains holding for enforcement engine
Row status information realizes the control to inquiry request, for example, the huge inquiry of resource occupation can be stopped in peak period, or
The inquiry of the type is adjusted to the progress of other periods by person.
Log analysis module 18, for obtaining the inquiry log of enforcement engine, and based on configuration from the inquiry log
Multiple fields are extracted, structuring log is formed.
As a result display module 19 show the inquiry knot for obtaining query result and the structuring log of enforcement engine
Fruit and structuring log.
Specifically, data resource is the core asset of enterprise-essential, leaking data happens occasionally in enterprise, some are to be not intended to
Some are that malice is revealed, by the statistical analysis to inquiry log, in order to which the tracking in later period is audited.And enforcement engine is daily all
A large amount of inquiry log can be generated, these logs are not easy to check, therefore, it is necessary to carry out the analysis of structuring and processing, with life
At the structuring log convenient for checking.For example, presetting the multiple fields for needing to extract from log, it is then based on these words
Corresponding data form of Duan Shengcheng etc..
Further, it is also possible to the resource consumption situation of cluster resource Expenditure Levels and each query task to real time monitoring into
Row summarizes displaying.Paging displaying can also be carried out to the inquiry that history executes, queue is distinguished and be presented to the user, such as show unit
Column monitoring resource service condition, the concise message for executing duration and user and details etc..
The Data Query Platform of the embodiment of the present invention provides the unified entrance to big data inquiry, flat by the inquiry
Platform can verify, parse query statement and carry out Authority Verification, and different inquiry requests is routed to different enforcement engines
It is inquired, then by way of readjustment, obtains query result and inquiry log.Further, it is also possible to monitor inquiry in real time
The operating status of resource consumption and cluster, the inquiry excessive to resource consumption, can support manually or automatically to cancel in peak period or
Stop, while guaranteeing search efficiency, ensures the monolithic stability of cluster.
Example IV
Fig. 4 is the flow chart for the data query method that the embodiment of the present invention four provides.The present embodiment is applicable to utilize number
The case where inquiring off-line data according to searching platform, this method can be executed by Data Query Platform, which can use soft
The mode of part and/or hardware is realized, and can be integrated on a computing device, or is deployed in the cluster being made of computer equipment
In.As shown in figure 4, this method comprises:
The corresponding query statement of inquiry request is analyzed, determines the executive plan of the inquiry request, wherein described
Executive plan from estimate occupation condition of the different dimensions to inquiry request for being described；
According to the attribute information of the executive plan and at least two enforcement engines, from least two enforcement engine
Selection target enforcement engine is inquired；
It indicates that the target enforcement engine handles the inquiry request, and obtains the inquiry knot of the target enforcement engine
Fruit.
Optionally, described that the corresponding query statement of inquiry request is analyzed, determine the execution meter of the inquiry request
It draws, comprising:
The field in query statement is analyzed according to syntax rule, obtains the tables of data that inquiry is related to；
According to the size and incidence relation for inquiring tables of data in the tables of data being related to and database, looked into described in determination
Ask the executive plan of request；Wherein, the executive plan includes at least between the data volume of inquiry, the tables of data that inquiry is related to
The degree of parallelism of dependence and data operation.
Optionally, the attribute information according to the executive plan and at least two enforcement engines, from described at least two
Selection target enforcement engine is inquired in a enforcement engine, comprising:
According to the attribute information of the executive plan and at least two enforcement engines, at least two enforcement engine
Priority is ranked up；
According to the sequence of priority from high to low, the target enforcement engine is selected, and in target enforcement engine
When executing failure, the enforcement engine for selecting priority to take second place is continued with.
Optionally, it is described the corresponding query statement of inquiry request is analyzed before, the method also includes:
The legitimacy of the query statement is verified.
Optionally, it is described the corresponding query statement of inquiry request is analyzed before, the method also includes:
Table parsing in library is carried out to the query statement, obtains the library and/or table that the inquiry request is related to.
Optionally, the method also includes:
Based on the parsing result of library table parsing, the inquiry times of each library and/or table are recorded, and according to the inquiry
Number analyzes the inquiry temperature of library and/or table.
Optionally, the method also includes:
Based on the library table parsing parsing result, judge the inquiry request whether have the library that it is related to and/or
The search access right of table, and when judging to have permission, execute the behaviour analyzed the corresponding query statement of inquiry request
Make.
Optionally, the method also includes:
The execution state information for obtaining enforcement engine, manages inquiry request according to the execution state information；Wherein,
The execution state information includes at least and executes time threshold, executes occupation condition and execute queue resource residue situation；
The control, which includes at least, stops inquiry or sending alarm.
Optionally, the method also includes:
The inquiry log of the enforcement engine is obtained, and multiple fields, shape are extracted from the inquiry log based on configuration
At structuring log；
Query result and the structuring log for obtaining the enforcement engine, show the query result and structuring log.
Optionally, the method also includes:
According to syntax rule and preset standards of grading, give a mark to the quality of the query statement.
The embodiment of the present invention determines the execution meter of inquiry request by the analysis to the corresponding query statement of inquiry request
It draws, the pre- occupancy situation of the resource of inquiry request is estimated, to therefrom be selected in conjunction with the attribute information of different enforcement engines
Target enforcement engine handles the inquiry request, and realization offsets the different inquiry request in cost source and handled respectively, and inquiry is provided
Source is routed to different enforcement engines, and then ensures the stability of search efficiency and each engine cluster.
Data query method provided by the embodiment of the present invention is put down by data query provided by any embodiment of the invention
Platform executes, and has the beneficial effect with Data Query Platform and its each functional module.
Embodiment five
Fig. 5 is a kind of structural schematic diagram for computer equipment that the embodiment of the present invention five provides.Fig. 5, which is shown, to be suitable for being used to
Realize the block diagram of the exemplary computer device 12 of embodiment of the present invention.The computer equipment 12 that Fig. 5 is shown is only one
Example, should not function to the embodiment of the present invention and use scope bring any restrictions.
As shown in figure 5, computer equipment 12 is showed in the form of universal computing device.The component of computer equipment 12 can be with
Including but not limited to: one or more processor or processing unit 16, system storage 28 connect different system components
The bus 18 of (including system storage 28 and processing unit 16).
Bus 18 indicates one of a few class bus structures or a variety of, including memory bus or Memory Controller,
Peripheral bus, graphics acceleration port, processor or the local bus using any bus structures in a variety of bus structures.It lifts
For example, these architectures include but is not limited to industry standard architecture (ISA) bus, microchannel architecture (MAC)
Bus, enhanced isa bus, Video Electronics Standards Association (VESA) local bus and peripheral component interconnection (PCI) bus.
Computer equipment 12 typically comprises a variety of computer system readable media.These media can be it is any can be by
The usable medium that computer equipment 12 accesses, including volatile and non-volatile media, moveable and immovable medium.
System storage 28 may include the computer system readable media of form of volatile memory, such as arbitrary access
Memory (RAM) 30 and/or cache memory 32.Computer equipment 12 may further include it is other it is removable/can not
Mobile, volatile/non-volatile computer system storage medium.Only as an example, storage system 34 can be used for reading and writing not
Movably, non-volatile magnetic media (Fig. 5 do not show, commonly referred to as " hard disk drive ").It, can be with although being not shown in Fig. 5
The disc driver for reading and writing to removable non-volatile magnetic disk (such as " floppy disk ") is provided, and non-volatile to moving
The CD drive of CD (such as CD-ROM, DVD-ROM or other optical mediums) read-write.In these cases, each driving
Device can be connected by one or more data media interfaces with bus 18.Memory 28 may include that at least one program produces
Product, the program product have one group of (for example, at least one) program module, these program modules are configured to perform of the invention each
The function of embodiment.
Program/utility 40 with one group of (at least one) program module 42 can store in such as memory 28
In, such program module 42 include but is not limited to operating system, one or more application program, other program modules and
It may include the realization of network environment in program data, each of these examples or certain combination.Program module 42 is usual
Execute the function and/or method in embodiment described in the invention.
Computer equipment 12 can also be with one or more external equipments 14 (such as keyboard, sensing equipment, display 24
Deng) communication, can also be enabled a user to one or more equipment interact with the computer equipment 12 communicate, and/or with make
The computer equipment 12 any equipment (such as network interface card, the modulatedemodulate that can be communicated with one or more of the other calculating equipment
Adjust device etc.) communication.This communication can be carried out by input/output (I/O) interface 22.Also, computer equipment 12 may be used also
To pass through network adapter 20 and one or more network (such as local area network (LAN), wide area network (WAN) and/or public network
Network, such as internet) communication.As shown, network adapter 20 is logical by other modules of bus 18 and computer equipment 12
Letter.It should be understood that although not shown in the drawings, can in conjunction with computer equipment 12 use other hardware and/or software module, including
But it is not limited to: microcode, device driver, redundant processing unit, external disk drive array, RAID system, tape drive
And data backup storage system etc..
Processing unit 16 by the program that is stored in system storage 28 of operation, thereby executing various function application and
Data processing, such as realize data query method provided by the embodiment of the present invention, comprising:
The corresponding query statement of inquiry request is analyzed, determines the executive plan of the inquiry request, wherein described
Executive plan from estimate occupation condition of the different dimensions to inquiry request for being described；
According to the attribute information of the executive plan and at least two enforcement engines, from least two enforcement engine
Selection target enforcement engine is inquired；
It indicates that the target enforcement engine handles the inquiry request, and obtains the inquiry knot of the target enforcement engine
Fruit.
Embodiment six
The embodiment of the present invention six additionally provides a kind of computer readable storage medium, is stored thereon with computer program, should
The data query method as provided by the embodiment of the present invention is realized when program is executed by processor, comprising:
The corresponding query statement of inquiry request is analyzed, determines the executive plan of the inquiry request, wherein described
Executive plan from estimate occupation condition of the different dimensions to inquiry request for being described；
According to the attribute information of the executive plan and at least two enforcement engines, from least two enforcement engine
Selection target enforcement engine is inquired；
It indicates that the target enforcement engine handles the inquiry request, and obtains the inquiry knot of the target enforcement engine
Fruit.
The computer storage medium of the embodiment of the present invention, can be using any of one or more computer-readable media
Combination.Computer-readable medium can be computer-readable signal media or computer readable storage medium.It is computer-readable
Storage medium for example may be-but not limited to-the system of electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor, device or
Device, or any above combination.The more specific example (non exhaustive list) of computer readable storage medium includes: tool
There are electrical connection, the portable computer diskette, hard disk, random access memory (RAM), read-only memory of one or more conducting wires
(ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-
ROM), light storage device, magnetic memory device or above-mentioned any appropriate combination.In this document, computer-readable storage
Medium can be any tangible medium for including or store program, which can be commanded execution system, device or device
Using or it is in connection.
Computer-readable signal media may include in a base band or as carrier wave a part propagate data-signal,
Wherein carry computer-readable program code.The data-signal of this propagation can take various forms, including but unlimited
In electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be that computer can
Any computer-readable medium other than storage medium is read, which can send, propagates or transmit and be used for
By the use of instruction execution system, device or device or program in connection.
The program code for including on computer-readable medium can transmit with any suitable medium, including --- but it is unlimited
In wireless, electric wire, optical cable, RF etc. or above-mentioned any appropriate combination.
The computer for executing operation of the present invention can be write with one or more programming languages or combinations thereof
Program code, described program design language include object oriented program language-such as Java, Smalltalk, C++,
Further include conventional procedural programming language-such as " C " language or similar programming language.Program code can be with
It fully executes, partly execute on the user computer on the user computer, being executed as an independent software package, portion
Divide and partially executes or executed on a remote computer or server completely on the remote computer on the user computer.?
Be related in the situation of remote computer, remote computer can pass through the network of any kind --- including local area network (LAN) or
Wide area network (WAN)-be connected to subscriber computer, or, it may be connected to outer computer (such as mentioned using Internet service
It is connected for quotient by internet).
Note that the above is only a better embodiment of the present invention and the applied technical principle.It will be appreciated by those skilled in the art that
The invention is not limited to the specific embodiments described herein, be able to carry out for a person skilled in the art it is various it is apparent variation,
It readjusts and substitutes without departing from protection scope of the present invention.Therefore, although being carried out by above embodiments to the present invention
It is described in further detail, but the present invention is not limited to the above embodiments only, without departing from the inventive concept, also
It may include more other equivalent embodiments, and the scope of the invention is determined by the scope of the appended claims.
Claims (15)
1. a kind of Data Query Platform characterized by comprising
Executive plan obtains module and determines the inquiry request for analyzing the corresponding query statement of inquiry request
Executive plan, wherein the executive plan from the pre- occupancy situation of resource of the different dimensions to inquiry request for being described；
Query engine selecting module, for the attribute information according to the executive plan and at least two enforcement engines, from described
Selection target enforcement engine is inquired at least two enforcement engines；
Query result obtains module, for obtaining the query result of the target enforcement engine.
2. platform according to claim 1, which is characterized in that the executive plan obtains module and is specifically used for:
The field in query statement is analyzed according to syntax rule, obtains the tables of data that inquiry is related to；
According to the size and incidence relation for inquiring tables of data in the tables of data being related to and database, determine that the inquiry is asked
The executive plan asked；Wherein, the executive plan includes at least the dependence between the data volume of inquiry, the tables of data that inquiry is related to
The degree of parallelism of relationship and data operation.
3. platform according to claim 2, which is characterized in that the query engine selecting module is also used to:
According to the attribute information of the executive plan and at least two enforcement engines, to the preferential of at least two enforcement engine
Grade is ranked up；
According to the sequence of priority from high to low, the target enforcement engine is selected, and is executed in target enforcement engine
When failure, the enforcement engine for selecting priority to take second place is continued with.
4. platform according to claim 1, which is characterized in that the platform further include:
Query statement correction verification module, for being obtained before module analyzes the query statement in executive plan, to described
The legitimacy of query statement is verified.
5. platform according to claim 1, which is characterized in that the platform further include:
Library table parsing module, for being obtained before module analyzes the query statement in executive plan, to the inquiry
Sentence carries out the parsing of library table, obtains the library and/or table that the inquiry request is related to.
6. platform according to claim 5, which is characterized in that the platform further include:
Library exterior heat degree analysis module records the inquiry time of each library and/or table for the parsing result based on library table parsing module
Number, and according to inquiry times analysis library and/or the inquiry temperature of table.
7. platform according to claim 5, which is characterized in that the platform further include:
Authority Verification module judges whether the inquiry request has and relates to it for the parsing result based on library table parsing module
And the search access right in the library and/or table arrived, and when judging to have permission, turn to obtain resume module by the executive plan.
8. platform according to claim 1, which is characterized in that the platform further include:
Inquiry control module, for obtaining the execution state information of enforcement engine, according to the execution state information to inquiry request
It is managed；Wherein, the execution state information includes at least and executes time threshold, executes occupation condition and execute queue
Resources left situation；The control, which includes at least, stops inquiry or sending alarm.
9. platform according to claim 1, which is characterized in that the platform further include:
Log analysis module is mentioned from the inquiry log for obtaining the inquiry log of the enforcement engine, and based on configuration
Multiple fields are taken, structuring log is formed；
As a result display module shows the query result for obtaining query result and the structuring log of the enforcement engine
With structuring log.
10. platform according to claim 1, which is characterized in that the platform further include:
Query statement detection module, for according to syntax rule and preset standards of grading, to the quality of the query statement into
Row marking.
11. a kind of data query method characterized by comprising
The corresponding query statement of inquiry request is analyzed, determines the executive plan of the inquiry request, wherein the execution
It is designed for being described from estimate occupation condition of the different dimensions to inquiry request；
According to the attribute information of the executive plan and at least two enforcement engines, selected from least two enforcement engine
Target enforcement engine is inquired；
It indicates that the target enforcement engine handles the inquiry request, and obtains the query result of the target enforcement engine.
12. according to the method for claim 11, which is characterized in that described to divide the corresponding query statement of inquiry request
Analysis, determines the executive plan of the inquiry request, comprising:
The field in query statement is analyzed according to syntax rule, obtains the tables of data that inquiry is related to；
According to the size and incidence relation for inquiring tables of data in the tables of data being related to and database, determine that the inquiry is asked
The executive plan asked；Wherein, the executive plan includes at least the dependence between the data volume of inquiry, the tables of data that inquiry is related to
The degree of parallelism of relationship and data operation.
13. according to the method for claim 12, which is characterized in that described to be executed according to the executive plan and at least two
The attribute information of engine, selection target enforcement engine is inquired from least two enforcement engine, comprising:
According to the attribute information of the executive plan and at least two enforcement engines, to the preferential of at least two enforcement engine
Grade is ranked up；
According to the sequence of priority from high to low, the target enforcement engine is selected, and is executed in target enforcement engine
When failure, the enforcement engine for selecting priority to take second place is continued with.
14. a kind of computer equipment characterized by comprising
One or more processors；
Storage device, for storing one or more programs,
When one or more of programs are executed by one or more of processors, so that one or more of processors are real
The now data query method as described in any in claim 11-13.
15. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the program is by processor
The data query method as described in any in claim 11-13 is realized when execution.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910492537.7A CN110222072A (en) | 2019-06-06 | 2019-06-06 | Data Query Platform, method, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910492537.7A CN110222072A (en) | 2019-06-06 | 2019-06-06 | Data Query Platform, method, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110222072A true CN110222072A (en) | 2019-09-10 |
Family
ID=67816019
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910492537.7A Withdrawn CN110222072A (en) | 2019-06-06 | 2019-06-06 | Data Query Platform, method, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110222072A (en) |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110825767A (en) * | 2019-11-15 | 2020-02-21 | 江苏满运软件科技有限公司 | Data query method, platform, equipment and storage medium |
| CN111078670A (en) * | 2019-12-19 | 2020-04-28 | 上海达梦数据库有限公司 | Database execution plan clearing method, device, equipment and storage medium |
| CN111104419A (en) * | 2019-12-24 | 2020-05-05 | 上海众源网络有限公司 | Data query method and device |
| CN111159229A (en) * | 2019-12-31 | 2020-05-15 | 北京奇艺世纪科技有限公司 | Data query method and device |
| CN111190932A (en) * | 2019-12-16 | 2020-05-22 | 北京淇瑀信息科技有限公司 | Privacy cluster query method and device and electronic equipment |
| CN111723112A (en) * | 2020-06-11 | 2020-09-29 | 咪咕文化科技有限公司 | Data task execution method and device, electronic equipment and storage medium |
| CN112115160A (en) * | 2020-08-14 | 2020-12-22 | 苏宁云计算有限公司 | Query request scheduling method and device and computer system |
| CN112286954A (en) * | 2020-09-25 | 2021-01-29 | 北京邮电大学 | Multi-dimensional data analysis method and system based on hybrid engine |
| CN112307061A (en) * | 2019-10-31 | 2021-02-02 | 北京京东尚科信息技术有限公司 | Method and device for querying data |
| CN112527830A (en) * | 2020-12-24 | 2021-03-19 | 平安普惠企业管理有限公司 | Data query method, device, equipment and storage medium |
| CN112579536A (en) * | 2019-09-30 | 2021-03-30 | 北京国双科技有限公司 | Data query method and device, computer equipment and storage medium |
| CN112711677A (en) * | 2021-01-08 | 2021-04-27 | 北京仿真中心 | Visual management device and method for Neo4j graph database |
| CN112948467A (en) * | 2021-03-18 | 2021-06-11 | 北京中经惠众科技有限公司 | Data processing method and device, computer equipment and storage medium |
| CN113204337A (en) * | 2021-06-08 | 2021-08-03 | 中国银行股份有限公司 | Switching method and device for running engines |
| CN113568931A (en) * | 2020-04-29 | 2021-10-29 | 盛趣信息技术（上海）有限公司 | Route analysis system and method for data access request |
| CN113641487A (en) * | 2021-07-06 | 2021-11-12 | 多点生活(成都)科技有限公司 | Intelligent automatic switching method for SQL task execution engine of big data platform |
| CN113704286A (en) * | 2020-05-20 | 2021-11-26 | 中国移动通信集团重庆有限公司 | Engine selection method and device and computing equipment |
| CN114442940A (en) * | 2022-01-04 | 2022-05-06 | 网易（杭州）网络有限公司 | Data processing method, device, medium and electronic equipment |
| CN115309777A (en) * | 2022-10-10 | 2022-11-08 | 北京奥星贝斯科技有限公司 | Data query method and device |
| CN115599790A (en) * | 2022-11-10 | 2023-01-13 | 星环信息科技（上海）股份有限公司（Cn） | Data storage system, data processing method, electronic device and storage medium |
| WO2023236776A1 (en) * | 2022-06-11 | 2023-12-14 | Huawei Cloud Computing Technologies Co., Ltd. | Method and apparatus for sub-optimal query identification and index optimizations post migration |
-
2019
- 2019-06-06 CN CN201910492537.7A patent/CN110222072A/en not_active Withdrawn
Cited By (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112579536A (en) * | 2019-09-30 | 2021-03-30 | 北京国双科技有限公司 | Data query method and device, computer equipment and storage medium |
| CN112307061A (en) * | 2019-10-31 | 2021-02-02 | 北京京东尚科信息技术有限公司 | Method and device for querying data |
| CN110825767A (en) * | 2019-11-15 | 2020-02-21 | 江苏满运软件科技有限公司 | Data query method, platform, equipment and storage medium |
| CN111190932A (en) * | 2019-12-16 | 2020-05-22 | 北京淇瑀信息科技有限公司 | Privacy cluster query method and device and electronic equipment |
| CN111190932B (en) * | 2019-12-16 | 2023-08-18 | 北京淇瑀信息科技有限公司 | Privacy cluster query method and device and electronic equipment |
| CN111078670A (en) * | 2019-12-19 | 2020-04-28 | 上海达梦数据库有限公司 | Database execution plan clearing method, device, equipment and storage medium |
| CN111078670B (en) * | 2019-12-19 | 2023-06-20 | 上海达梦数据库有限公司 | Database execution plan clearing method, device, equipment and storage medium |
| CN111104419A (en) * | 2019-12-24 | 2020-05-05 | 上海众源网络有限公司 | Data query method and device |
| CN111159229A (en) * | 2019-12-31 | 2020-05-15 | 北京奇艺世纪科技有限公司 | Data query method and device |
| CN111159229B (en) * | 2019-12-31 | 2024-04-26 | 北京奇艺世纪科技有限公司 | Data query method and device |
| CN113568931A (en) * | 2020-04-29 | 2021-10-29 | 盛趣信息技术（上海）有限公司 | Route analysis system and method for data access request |
| CN113704286A (en) * | 2020-05-20 | 2021-11-26 | 中国移动通信集团重庆有限公司 | Engine selection method and device and computing equipment |
| CN111723112A (en) * | 2020-06-11 | 2020-09-29 | 咪咕文化科技有限公司 | Data task execution method and device, electronic equipment and storage medium |
| CN112115160A (en) * | 2020-08-14 | 2020-12-22 | 苏宁云计算有限公司 | Query request scheduling method and device and computer system |
| CN112286954B (en) * | 2020-09-25 | 2022-08-16 | 北京邮电大学 | Multi-dimensional data analysis method and system based on hybrid engine |
| CN112286954A (en) * | 2020-09-25 | 2021-01-29 | 北京邮电大学 | Multi-dimensional data analysis method and system based on hybrid engine |
| CN112527830A (en) * | 2020-12-24 | 2021-03-19 | 平安普惠企业管理有限公司 | Data query method, device, equipment and storage medium |
| CN112711677A (en) * | 2021-01-08 | 2021-04-27 | 北京仿真中心 | Visual management device and method for Neo4j graph database |
| CN112948467A (en) * | 2021-03-18 | 2021-06-11 | 北京中经惠众科技有限公司 | Data processing method and device, computer equipment and storage medium |
| CN112948467B (en) * | 2021-03-18 | 2023-10-10 | 北京中经惠众科技有限公司 | Data processing method and device, computer equipment and storage medium |
| CN113204337A (en) * | 2021-06-08 | 2021-08-03 | 中国银行股份有限公司 | Switching method and device for running engines |
| CN113641487A (en) * | 2021-07-06 | 2021-11-12 | 多点生活(成都)科技有限公司 | Intelligent automatic switching method for SQL task execution engine of big data platform |
| CN114442940A (en) * | 2022-01-04 | 2022-05-06 | 网易（杭州）网络有限公司 | Data processing method, device, medium and electronic equipment |
| WO2023236776A1 (en) * | 2022-06-11 | 2023-12-14 | Huawei Cloud Computing Technologies Co., Ltd. | Method and apparatus for sub-optimal query identification and index optimizations post migration |
| CN115309777A (en) * | 2022-10-10 | 2022-11-08 | 北京奥星贝斯科技有限公司 | Data query method and device |
| CN115599790A (en) * | 2022-11-10 | 2023-01-13 | 星环信息科技（上海）股份有限公司（Cn） | Data storage system, data processing method, electronic device and storage medium |
| CN115599790B (en) * | 2022-11-10 | 2024-03-15 | 星环信息科技(上海)股份有限公司 | Data storage system, data processing method, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110222072A (en) | Data Query Platform, method, equipment and storage medium | |
| CN107273280B (en) | Log processing method and device, electronic equipment and storage medium | |
| US9189524B2 (en) | Obtaining partial results from a database query | |
| US20150134637A1 (en) | System and Method for Sharding a Graph Database | |
| US10331549B2 (en) | Collaborative computer aided test plan generation | |
| US8498995B1 (en) | Optimizing data retrieval during event data query processing | |
| US8489550B2 (en) | Multi-tenancy data storage and access method and apparatus | |
| US9465790B2 (en) | SVO-based taxonomy-driven text analytics | |
| CN108959564A (en) | Data warehouse metadata management method, readable storage medium storing program for executing and computer equipment | |
| CN108519967A (en) | Chart method for visualizing, device, terminal and storage medium | |
| US20180218041A1 (en) | Validation of query plan | |
| US7730024B2 (en) | Distributed data mining using analysis services servers | |
| US10031901B2 (en) | Narrative generation using pattern recognition | |
| CN107733894B (en) | Comparison method, system, equipment and storage medium of logical interface message | |
| CN111813804A (en) | Data query method and device, electronic equipment and storage medium | |
| KR101331452B1 (en) | Method for providing database management and the database management server there of | |
| US9336674B1 (en) | Notifying a user utilizing smart alerting techniques | |
| US20200265042A1 (en) | Candidate name generation | |
| EP4044031A1 (en) | Cloud orchestration system and method based on read-write separation and auto-scaling | |
| CN109376173A (en) | A kind of data query method, apparatus, electronic equipment and storage medium | |
| CN111767269A (en) | Health detection method, device and equipment of database instance and storage medium | |
| CN108133143B (en) | Data leakage prevention method and system for cloud desktop application environment | |
| CN107609179B (en) | Data processing method and equipment | |
| US20150302034A1 (en) | Correlating database and storage performance views | |
| CN109144965A (en) | A kind of log processing method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20190910 |
|
| WW01 | Invention patent application withdrawn after publication |