CN110019298A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN110019298A CN110019298A CN201711053329.4A CN201711053329A CN110019298A CN 110019298 A CN110019298 A CN 110019298A CN 201711053329 A CN201711053329 A CN 201711053329A CN 110019298 A CN110019298 A CN 110019298A
- Authority
- CN
- China
- Prior art keywords
- structured query
- query sentence
- sentence
- query
- resource quantity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of data processing method and device.This method comprises: reading the query characteristics that structured query sentence includes；The query characteristics for including by structured query sentence are input in the memory prediction model pre-established, obtain prediction result, wherein, memory prediction model is obtained according to historical data training, and historical data includes resource quantity needed for the query characteristics and structured query sentence that structured query sentence includes；Required resource quantity when determining that structured query sentence executes according to prediction result；Structured query sentence is stored in resource pool corresponding with resource quantity and is run.Through the invention, the more accurate effect of resource estimation quantity to database has been reached.
Description
Technical field
The present invention relates to computer fields, in particular to a kind of data processing method and device.
Background technique
Impala be one based on distributed file system (Hadoop Distribute File System, referred to as
) or the distribution of HBase storage system, interactive database HDFS.Because the resource estimation mode error of impala is too big, it is
Cluster can be stable operation, can only be controlled by the setting of default each sql (Structured Query Language,
Structured query sentence) resource quantity, when sql resource actually required be greater than impala estimation resource quantity when, sql will
It can be cancelled automatically.If the actually required resource of some sql that the same resource pool executes parallel is especially big, this also will affect the money
The execution of other sql in the pond of source.
There are such problems for existing resource estimation method: resource quantity evaluated error is too big, cannot reasonably dispatch
Operation in sql to suitable resource pool.
For the problem that the resource estimation quantitative error of database in the related technology causes greatly cluster unstable, at present not yet
It puts forward effective solutions.
Summary of the invention
The main purpose of the present invention is to provide a kind of data processing method and device, to solve the resource estimation of database
The problem that quantitative error causes greatly cluster unstable.
To achieve the goals above, according to an aspect of the invention, there is provided a kind of data processing method, this method packet
It includes: reading the query characteristics that structured query sentence includes；The query characteristics that the structured query sentence includes are input to
In the memory prediction model pre-established, prediction result is obtained, wherein memory prediction model is obtained according to historical data training
, historical data includes resource quantity needed for the query characteristics and structured query sentence that structured query sentence includes；Root
Required resource quantity when determining that the structured query sentence executes according to the prediction result；By the structured query sentence
It is stored in resource pool corresponding with the resource quantity and runs.
Further, the memory prediction pre-established is input in the query characteristics for including by the structured query sentence
Before in model, the method also includes: obtain historical data, wherein the historical data includes multiple groups structuralized query language
Resource quantity needed for the query characteristics and the multiple groups structured query sentence of sentence；It is established in described according to the historical data
Deposit prediction model.
Further, the resource quantity corresponds to multiple grades, and the inquiry for including by the structured query sentence is special
Sign is input in the memory prediction model pre-established, and obtaining prediction result includes: to include by the structured query sentence
Query characteristics are input in the memory prediction model pre-established, the grade for the resource quantity predicted.
Further, reading the structured query sentence query characteristics that include includes: to pass through Java receiving client
When the structured query sentence that database connection JDBC mode is submitted, it is special to read the inquiry that the structured query sentence includes
Sign, wherein the query characteristics include join feature and select feature.
To achieve the goals above, according to another aspect of the present invention, a kind of data processing equipment is additionally provided, the device
It include: reading unit, the query characteristics for including for reading structured query sentence；Input unit is used for the structuring
The query characteristics that query statement includes are input in the memory prediction model pre-established, obtain prediction result, wherein memory is pre-
Surveying model is obtained according to historical data training, and historical data includes the query characteristics and structure that structured query sentence includes
Resource quantity needed for changing query statement；Determination unit, for determining the structured query sentence according to the prediction result
Required resource quantity when execution；Running unit, it is corresponding with the resource quantity for being stored in the structured query sentence
Resource pool in run.
Further, described device further include: acquiring unit, in the inquiry for including by the structured query sentence
Before feature is input in the memory prediction model pre-established, historical data is obtained, wherein the historical data includes multiple groups
Resource quantity needed for the query characteristics of structured query sentence and the multiple groups structured query sentence；Unit is established, is used for
The memory prediction model is established according to the historical data.
Further, the resource quantity corresponds to multiple grades, and the input unit is used for: by the structuralized query
The query characteristics that sentence includes are input in the memory prediction model pre-established, the grade for the resource quantity predicted.
Further, the reading unit is used for: being mentioned receiving client by Java database connection JDBC mode
When the structured query sentence of friendship, the query characteristics that the structured query sentence includes are read, wherein the query characteristics packet
Include join feature and select feature.
To achieve the goals above, according to another aspect of the present invention, a kind of storage medium is additionally provided, including storage
Program, wherein equipment where controlling the storage medium in described program operation executes data processing side of the present invention
Method.
To achieve the goals above, according to another aspect of the present invention, a kind of processor is additionally provided, for running journey
Sequence, wherein described program executes data processing method of the present invention when running.
The query characteristics that the present invention includes by reading structured query sentence；The inquiry for including by structured query sentence
Feature is input in the memory prediction model pre-established, obtains prediction result, wherein memory prediction model is according to history number
It is obtained according to training, historical data includes money needed for the query characteristics and structured query sentence that structured query sentence includes
Source quantity；Required resource quantity when determining that structured query sentence executes according to prediction result；Structured query sentence is deposited
Enter and run in resource pool corresponding with resource quantity, the resource estimation quantitative error for solving database causes greatly cluster unstable
The problem of, and then reached the more accurate effect of resource estimation quantity to database.
Detailed description of the invention
The attached drawing constituted part of this application is used to provide further understanding of the present invention, schematic reality of the invention
It applies example and its explanation is used to explain the present invention, do not constitute improper limitations of the present invention.In the accompanying drawings:
Fig. 1 is the flow chart of data processing method according to an embodiment of the present invention；
Fig. 2 is the schematic diagram of data processing equipment according to an embodiment of the present invention.
Specific embodiment
It should be noted that in the absence of conflict, the features in the embodiments and the embodiments of the present application can phase
Mutually combination.The present invention will be described in detail below with reference to the accompanying drawings and embodiments.
In order to make those skilled in the art more fully understand application scheme, below in conjunction in the embodiment of the present application
Attached drawing, the technical scheme in the embodiment of the application is clearly and completely described, it is clear that described embodiment is only
The embodiment of the application a part, instead of all the embodiments.Based on the embodiment in the application, ordinary skill people
Member's every other embodiment obtained without making creative work, all should belong to the model of the application protection
It encloses.
It should be noted that the description and claims of this application and term " first " in above-mentioned attached drawing, "
Two " etc. be to be used to distinguish similar objects, without being used to describe a particular order or precedence order.It should be understood that using in this way
Data be interchangeable under appropriate circumstances, so as to embodiments herein described herein.In addition, term " includes " and " tool
Have " and their any deformation, it is intended that cover it is non-exclusive include, for example, containing a series of steps or units
Process, method, system, product or equipment those of are not necessarily limited to be clearly listed step or unit, but may include without clear
Other step or units listing to Chu or intrinsic for these process, methods, product or equipment.
For ease of description, below to the invention relates to several terms be illustrated:
Impala is a distribution based on HDFS or HBase storage system, interactive database.
Feature extraction refers to be gone out to describe the association attributes of sample according to sample extraction.
The embodiment of the invention provides a kind of data processing methods.
Fig. 1 is the flow chart of data processing method according to an embodiment of the present invention, as shown in Figure 1, this method includes following
Step:
Step S102: the query characteristics that structured query sentence includes are read；
Step S104: the query characteristics for including by structured query sentence are input to the memory prediction model pre-established
In, obtain prediction result, wherein memory prediction model is obtained according to historical data training, and historical data includes structuring
Resource quantity needed for query characteristics and structured query sentence that query statement includes；
Step S106: required resource quantity when determining that structured query sentence executes according to prediction result；
Step S108: structured query sentence is stored in resource pool corresponding with resource quantity and is run.
The query characteristics that the embodiment includes using structured query sentence is read；It is looked into what structured query sentence included
It askes feature to be input in the memory prediction model pre-established, obtains prediction result, wherein memory prediction model is according to history
Data training obtains, and historical data includes needed for the query characteristics and structured query sentence that structured query sentence includes
Resource quantity；Required resource quantity when determining that structured query sentence executes according to prediction result；By structured query sentence
It is stored in resource pool corresponding with resource quantity and runs, the resource estimation quantitative error for solving database causes greatly cluster unstable
Fixed problem, and then reached the more accurate effect of resource estimation quantity to database.
The technical solution of the embodiment of the present invention can be applied in impala database, execute as a kind of optimization impala
The method of cluster stability when inquiry.In structured query sentence (Structured Query Language, referred to as sql)
Comprising there are many query characteristics, for example, can wrap containing features such as join, select, reading can be defeated by feature after feature
Enter and carry out model calculation into the model pre-established, obtain prediction result, then is determined to execute according to the result of prediction and be somebody's turn to do
Sql can be deposited into corresponding resource pool run in this way by resource quantity required for sql, due to passing through model prediction energy
More accurately prediction result is accessed, thus sql can be deposited into most suitable resource pool and be run, can be prevented
Sql resource actually required, which is greater than impala quantity, leads to the problem of being cancelled, and promotes the resource estimation quantity of impala, rationally
Scheduling sql executed to suitable resource pool, the operation of impala cluster can be made more stable.
Before the query characteristics for including by structured query sentence are input in the memory prediction model pre-established, obtain
Take historical data, wherein historical data includes the query characteristics and multiple groups structured query sentence of multiple groups structured query sentence
Required resource quantity；Memory prediction model is established according to historical data.
Model foundation needs to establish by multiple groups historical data, and every group of historical data includes that the query characteristics of sql, sql are held
The quantity of the characteristic parameter of required scanning and corresponding memory source when row, after getting historical data, to historical data
Classified and arranged, memory prediction model can be established based on historical data.Characteristic parameter can be scanning needed for sql is executed
The features such as number of files, the number of partitions of table, total number of partitions, hash number, agg number.
Optionally, resource quantity corresponds to multiple grades, and the query characteristics for including by structured query sentence are input to pre-
In the memory prediction model first established, obtaining prediction result includes: that the query characteristics for including are input to by structured query sentence
In the memory prediction model pre-established, the grade for the resource quantity predicted.Resource quantity can be divided into multiple grades,
The resource quantity predicted can be the grade for the resource quantity predicted, may not need to obtain resource quantity and direct
To grade, model can simplify in this way.
In the technical solution of the embodiment of the present invention, structured query sentence is being stored in resource pool corresponding with resource quantity
Before middle operation, impala cluster memory is divided into the different brackets such as the memory pool of multiple grades, such as 200G, 400G
Memory pool, obtained by model some sql sentence needs memory source quantity after, can determine in memory
The suitable resource pool of grade is run for the sql, and then sql is put into corresponding resource pool and is run.
Optionally, reading the structured query sentence query characteristics that include includes: to pass through the side JDBC receiving client
When the structured query sentence that formula is submitted, the structured query sentence query characteristics that include are read, wherein query characteristics include
Join feature and select feature.
As a kind of optional embodiment, can by client by JDBC (Java DataBase Connectivity,
Java, database connection) mode submit sql, upon receipt, the query characteristics that sql includes can be read.
The embodiment of the invention also provides a kind of preferred embodiment, which includes following part:
1. extracting the number of files, the number of partitions of table, total number of partitions, hash that scan needed for sql is executed by explain sentence
The features such as number, agg number, the features such as parsing sql self-contained join, select.
2. memory prediction model is established according to certain sql historical data, according to test sample tuning model or parameter.This
Place is preferably selected sorting algorithm, and the prediction of memory is divided into multiple grades.
3. client submits sql by modes such as JDBC, feature is first extracted, is then predicted according to prediction model
As a result sql is put into corresponding resource pool, setting sql executes required resource, finally executes sql.
After handling in this way, for benefit of both mainly having for impala cluster.First is that the stability of cluster will
It being strengthened, sql can be put into the resource pool of resource abundance according to the prediction result that prediction model obtains and execute by client,
Both the rate that runs succeeded that sql can be improved can also reduce influence of the big sql to other concurrent sql.Second is that O&M colleague can be with
According to the increase and decrease of the historical data decision cluster scale of prediction model, the soft and hardware cost of cluster is saved.
It should be noted that step shown in the flowchart of the accompanying drawings can be in such as a group of computer-executable instructions
It is executed in computer system, although also, logical order is shown in flow charts, and it in some cases, can be with not
The sequence being same as herein executes shown or described step.
The embodiment of the invention provides a kind of data processing equipment, which can be used for executing the number of the embodiment of the present invention
According to processing method.
Fig. 2 is the schematic diagram of data processing equipment according to an embodiment of the present invention, as shown in Fig. 2, the device includes:
Reading unit 10, the query characteristics for including for reading structured query sentence；
Input unit 20, the query characteristics for including by structured query sentence are input to the memory prediction pre-established
In model, prediction result is obtained, wherein memory prediction model is obtained according to historical data training, and historical data includes knot
Resource quantity needed for query characteristics and structured query sentence that structure query statement includes；
Determination unit 30, required resource quantity when for determining that structured query sentence executes according to prediction result；
Running unit 40 is run for being stored in structured query sentence in resource pool corresponding with resource quantity.
The embodiment uses reading unit 10, the query characteristics for including for reading structured query sentence；Input unit
20, the query characteristics for including by structured query sentence are input in the memory prediction model pre-established, are predicted
As a result, wherein memory prediction model is obtained according to historical data training, and historical data includes that structured query sentence includes
Query characteristics and structured query sentence needed for resource quantity；Determination unit 30, for determining structure according to prediction result
Change resource quantity required when query statement executes；Running unit 40 is used for structured query sentence deposit and resource quantity
It is run in corresponding resource pool, solves the problems, such as that the resource estimation quantitative error of database causes greatly cluster unstable, in turn
The more accurate effect of resource estimation quantity to database is reached.
Optionally, the device further include: acquiring unit, in the query characteristics input for including by structured query sentence
Before into the memory prediction model pre-established, historical data is obtained, wherein historical data includes multiple groups structuralized query language
Resource quantity needed for the query characteristics and multiple groups structured query sentence of sentence；Unit is established, for establishing according to historical data
Memory prediction model.
Optionally, resource quantity corresponds to multiple grades, and input unit 20 by what structured query sentence included for looking into
It askes feature to be input in the memory prediction model pre-established, the grade for the resource quantity predicted.
Optionally, reading unit 10 is used for: connecting what JDBC mode was submitted by Java database receiving client
When structured query sentence, read the structured query sentence query characteristics that include, wherein query characteristics include join feature and
Select feature.
The data processing equipment includes processor and memory, above-mentioned reading unit, input unit, determination unit, fortune
Row unit etc. stores in memory as program unit, executes above procedure unit stored in memory by processor
To realize corresponding function.
Include kernel in processor, is gone in memory to transfer corresponding program unit by kernel.Kernel can be set one
Or more, come by adjusting kernel parameter more acurrate to the resource estimation quantity of database.
Memory may include the non-volatile memory in computer-readable medium, random access memory (RAM) and/
Or the forms such as Nonvolatile memory, if read-only memory (ROM) or flash memory (flash RAM), memory include that at least one is deposited
Store up chip.
The embodiment of the invention provides a kind of storage mediums, are stored thereon with program, real when which is executed by processor
The existing data processing method.
The embodiment of the invention provides a kind of processor, the processor is for running program, wherein described program operation
Data processing method described in Shi Zhihang.
The embodiment of the invention provides a kind of equipment, equipment include processor, memory and storage on a memory and can
The program run on a processor, processor performs the steps of when executing program reads structured query sentence looking into of including
Ask feature；The query characteristics for including by structured query sentence are input in the memory prediction model pre-established, are predicted
As a result, wherein memory prediction model is obtained according to historical data training, and historical data includes that structured query sentence includes
Query characteristics and structured query sentence needed for resource quantity；When determining that structured query sentence executes according to prediction result
Required resource quantity；Structured query sentence is stored in resource pool corresponding with resource quantity and is run.
Obtain historical data, wherein historical data includes the query characteristics and multiple groups structure of multiple groups structured query sentence
Resource quantity needed for changing query statement；Memory prediction model is established according to historical data.
The query characteristics for including by structured query sentence are input in the memory prediction model pre-established, are predicted
Resource quantity grade.
Reading the structured query sentence query characteristics that include includes: to be connected by Java database receiving client
When the structured query sentence that JDBC mode is submitted, the query characteristics that structured query sentence includes are read.Equipment herein
It can be server, PC, PAD, mobile phone etc..
Present invention also provides a kind of computer program products, when executing on data processing equipment, are adapted for carrying out just
The program of beginningization there are as below methods step: the query characteristics that structured query sentence includes are read；By structured query sentence packet
The query characteristics contained are input in the memory prediction model pre-established, obtain prediction result, wherein memory prediction model is root
It is obtained according to historical data training, historical data includes the query characteristics and structured query sentence that structured query sentence includes
Required resource quantity；Required resource quantity when determining that structured query sentence executes according to prediction result；Structuring is looked into
It askes and is run in sentence deposit resource pool corresponding with resource quantity.
Obtain historical data, wherein historical data includes the query characteristics and multiple groups structure of multiple groups structured query sentence
Resource quantity needed for changing query statement；Memory prediction model is established according to historical data.
The query characteristics for including by structured query sentence are input in the memory prediction model pre-established, are predicted
Resource quantity grade.
Reading the structured query sentence query characteristics that include includes: to be connected by Java database receiving client
When the structured query sentence that JDBC mode is submitted, the query characteristics that structured query sentence includes are read.
It should be understood by those skilled in the art that, embodiments herein can provide as method, system or computer program
Product.Therefore, complete hardware embodiment, complete software embodiment or reality combining software and hardware aspects can be used in the application
Apply the form of example.Moreover, it wherein includes the computer of computer usable program code that the application, which can be used in one or more,
The computer program implemented in usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) produces
The form of product.
The application is referring to method, the process of equipment (system) and computer program product according to the embodiment of the present application
Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions
The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer programs
Instruct the processor of general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices to produce
A raw machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for real
The device for the function of being specified in present one or more flows of the flowchart and/or one or more blocks of the block diagram.
These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates,
Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or
The function of being specified in multiple boxes.
These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting
Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or
The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one
The step of function of being specified in a box or multiple boxes.
In a typical configuration, calculating equipment includes one or more processors (CPU), input/output interface, net
Network interface and memory.
Memory may include the non-volatile memory in computer-readable medium, random access memory (RAM) and/
Or the forms such as Nonvolatile memory, such as read-only memory (ROM) or flash memory (flash RAM).Memory is computer-readable Jie
The example of matter.
Computer-readable medium includes permanent and non-permanent, removable and non-removable media can be by any method
Or technology come realize information store.Information can be computer readable instructions, data structure, the module of program or other data.
The example of the storage medium of computer includes, but are not limited to phase change memory (PRAM), static random access memory (SRAM), moves
State random access memory (DRAM), other kinds of random access memory (RAM), read-only memory (ROM), electric erasable
Programmable read only memory (EEPROM), flash memory or other memory techniques, read-only disc read only memory (CD-ROM) (CD-ROM),
Digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape magnetic disk storage or other magnetic storage devices
Or any other non-transmission medium, can be used for storage can be accessed by a computing device information.As defined in this article, it calculates
Machine readable medium does not include temporary computer readable media (transitory media), such as the data-signal and carrier wave of modulation.
It should also be noted that, the terms "include", "comprise" or its any other variant are intended to nonexcludability
It include so that the process, method, commodity or the equipment that include a series of elements not only include those elements, but also to wrap
Include other elements that are not explicitly listed, or further include for this process, method, commodity or equipment intrinsic want
Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including element
There is also other identical elements in process, method, commodity or equipment.
It will be understood by those skilled in the art that embodiments herein can provide as method, system or computer program product.
Therefore, complete hardware embodiment, complete software embodiment or embodiment combining software and hardware aspects can be used in the application
Form.It is deposited moreover, the application can be used to can be used in the computer that one or more wherein includes computer usable program code
The shape for the computer program product implemented on storage media (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.)
Formula.
The above is only embodiments herein, are not intended to limit this application.To those skilled in the art,
Various changes and changes are possible in this application.It is all within the spirit and principles of the present application made by any modification, equivalent replacement,
Improve etc., it should be included within the scope of the claims of this application.
Claims (10)
1. a kind of data processing method characterized by comprising
Read the query characteristics that structured query sentence includes；
The query characteristics that the structured query sentence includes are input in the memory prediction model pre-established, are predicted
As a result, wherein the memory prediction model is obtained according to historical data training, and the historical data includes structuralized query
Resource quantity needed for query characteristics and the structured query sentence that sentence includes；
Required resource quantity when determining that the structured query sentence executes according to the prediction result；
The structured query sentence is stored in resource pool corresponding with the resource quantity and is run.
2. the method according to claim 1, wherein in the query characteristics for including by the structured query sentence
Before being input in the memory prediction model pre-established, the method also includes:
Obtain historical data, wherein the historical data includes the query characteristics and the multiple groups of multiple groups structured query sentence
Resource quantity needed for structured query sentence；
The memory prediction model is established according to the historical data.
3. the method according to claim 1, wherein the resource quantity corresponds to multiple grades, by the knot
The query characteristics that structure query statement includes are input in the memory prediction model pre-established, are obtained prediction result and are included:
The query characteristics that the structured query sentence includes are input in the memory prediction model pre-established, are predicted
Resource quantity grade.
4. the method according to claim 1, wherein reading the query characteristics packet that structured query sentence includes
It includes:
When receiving the structured query sentence that client is submitted by Java database connection JDBC mode, the knot is read
The query characteristics that structure query statement includes, wherein the query characteristics include join feature and select feature.
5. a kind of data processing equipment characterized by comprising
Reading unit, the query characteristics for including for reading structured query sentence；
Input unit, the query characteristics for including by the structured query sentence are input to the memory prediction mould pre-established
In type, prediction result is obtained, wherein the memory prediction model is obtained according to historical data training, the historical data
Resource quantity needed for the query characteristics and the structured query sentence that include including structured query sentence；
Determination unit, required resource quantity when for determining that the structured query sentence executes according to the prediction result；
Running unit is run for being stored in the structured query sentence in resource pool corresponding with the resource quantity.
6. device according to claim 5, which is characterized in that described device further include:
Acquiring unit, for being input to the memory prediction pre-established in the query characteristics for including by the structured query sentence
Before in model, historical data is obtained, wherein the historical data includes query characteristics and the institute of multiple groups structured query sentence
Resource quantity needed for stating multiple groups structured query sentence；
Unit is established, for establishing the memory prediction model according to the historical data.
7. device according to claim 5, which is characterized in that the resource quantity corresponds to multiple grades, the input
Unit is used for:
The query characteristics that the structured query sentence includes are input in the memory prediction model pre-established, are predicted
Resource quantity grade.
8. device according to claim 5, which is characterized in that the reading unit is used for:
When receiving the structured query sentence that client is submitted by Java database connection JDBC mode, the knot is read
The query characteristics that structure query statement includes, wherein the query characteristics include join feature and select feature.
9. a kind of storage medium, which is characterized in that the storage medium includes the program of storage, wherein run in described program
When control the storage medium where equipment perform claim require any one of 1 to 4 described in data processing method.
10. a kind of processor, which is characterized in that the processor is for running program, wherein right of execution when described program is run
Benefit require any one of 1 to 4 described in data processing method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711053329.4A CN110019298B (en) | 2017-10-31 | 2017-10-31 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711053329.4A CN110019298B (en) | 2017-10-31 | 2017-10-31 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110019298A true CN110019298A (en) | 2019-07-16 |
| CN110019298B CN110019298B (en) | 2021-07-30 |
Family
ID=67186736
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711053329.4A Active CN110019298B (en) | 2017-10-31 | 2017-10-31 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110019298B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109324905A (en) * | 2018-09-30 | 2019-02-12 | 拉卡拉支付股份有限公司 | Database operation method, device, electronic equipment and storage medium |
| CN111046059A (en) * | 2019-12-09 | 2020-04-21 | 中国建设银行股份有限公司 | Low-efficiency SQL statement analysis method and system based on distributed database cluster |
| CN112699140A (en) * | 2019-10-23 | 2021-04-23 | 阿里巴巴集团控股有限公司 | Data processing method, device, equipment and storage medium |
| CN114579604A (en) * | 2022-03-15 | 2022-06-03 | 北京梦诚科技有限公司 | Database transaction implementation method and system of application layer |
| CN116561374A (en) * | 2023-07-11 | 2023-08-08 | 腾讯科技（深圳）有限公司 | Resource determination method, device, equipment and medium based on semi-structured storage |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101408900A (en) * | 2008-11-24 | 2009-04-15 | 中国科学院地理科学与资源研究所 | Distributed space data enquiring and optimizing method under gridding calculation environment |
| CN101969536A (en) * | 2002-01-30 | 2011-02-09 | 索尼公司 | Data processing apparatus and data processing method used for the data processing apparatus |
| CN102004671A (en) * | 2010-11-15 | 2011-04-06 | 北京航空航天大学 | Resource management method of data center based on statistic model in cloud computing environment |
| CN103186603A (en) * | 2011-12-29 | 2013-07-03 | 中国移动通信集团浙江有限公司 | Method, system and equipment for determining influence of SQL statements on performance of key businesses |
| CN105868274A (en) * | 2016-03-22 | 2016-08-17 | 努比亚技术有限公司 | Resource data querying and processing method and device thereof |
| CN105893224A (en) * | 2015-01-26 | 2016-08-24 | 阿里巴巴集团控股有限公司 | Resource measurement method and device |
| US20160328446A1 (en) * | 2015-05-04 | 2016-11-10 | Dell Software, Inc. | Method of Optimizing Complex SQL Statements Using a Region Divided Preferential SQL Rewrite Operation |
| US20170161307A1 (en) * | 2015-12-02 | 2017-06-08 | Speedment, Inc. | Methods and systems for mapping object oriented/functional languages to database languages |
| CN106844678A (en) * | 2017-01-24 | 2017-06-13 | 山东浪潮商用系统有限公司 | A kind of exchange method of Mybatis data sources and connection pool |
| CN107194680A (en) * | 2016-03-14 | 2017-09-22 | 阿里巴巴集团控股有限公司 | A kind of configurableization resource quantity control method and device |
-
2017
- 2017-10-31 CN CN201711053329.4A patent/CN110019298B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101969536A (en) * | 2002-01-30 | 2011-02-09 | 索尼公司 | Data processing apparatus and data processing method used for the data processing apparatus |
| CN101408900A (en) * | 2008-11-24 | 2009-04-15 | 中国科学院地理科学与资源研究所 | Distributed space data enquiring and optimizing method under gridding calculation environment |
| CN102004671A (en) * | 2010-11-15 | 2011-04-06 | 北京航空航天大学 | Resource management method of data center based on statistic model in cloud computing environment |
| CN103186603A (en) * | 2011-12-29 | 2013-07-03 | 中国移动通信集团浙江有限公司 | Method, system and equipment for determining influence of SQL statements on performance of key businesses |
| CN105893224A (en) * | 2015-01-26 | 2016-08-24 | 阿里巴巴集团控股有限公司 | Resource measurement method and device |
| US20160328446A1 (en) * | 2015-05-04 | 2016-11-10 | Dell Software, Inc. | Method of Optimizing Complex SQL Statements Using a Region Divided Preferential SQL Rewrite Operation |
| US20170161307A1 (en) * | 2015-12-02 | 2017-06-08 | Speedment, Inc. | Methods and systems for mapping object oriented/functional languages to database languages |
| CN107194680A (en) * | 2016-03-14 | 2017-09-22 | 阿里巴巴集团控股有限公司 | A kind of configurableization resource quantity control method and device |
| CN105868274A (en) * | 2016-03-22 | 2016-08-17 | 努比亚技术有限公司 | Resource data querying and processing method and device thereof |
| CN106844678A (en) * | 2017-01-24 | 2017-06-13 | 山东浪潮商用系统有限公司 | A kind of exchange method of Mybatis data sources and connection pool |
Non-Patent Citations (1)
| Title |
|---|
| 国冰磊等: "结构化查询语言动态功耗解析及建模", 《计算机应用》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109324905A (en) * | 2018-09-30 | 2019-02-12 | 拉卡拉支付股份有限公司 | Database operation method, device, electronic equipment and storage medium |
| CN112699140A (en) * | 2019-10-23 | 2021-04-23 | 阿里巴巴集团控股有限公司 | Data processing method, device, equipment and storage medium |
| CN112699140B (en) * | 2019-10-23 | 2023-12-26 | 阿里巴巴集团控股有限公司 | Data processing method, device, equipment and storage medium |
| CN111046059A (en) * | 2019-12-09 | 2020-04-21 | 中国建设银行股份有限公司 | Low-efficiency SQL statement analysis method and system based on distributed database cluster |
| CN111046059B (en) * | 2019-12-09 | 2023-06-30 | 中国建设银行股份有限公司 | Low-efficiency SQL statement analysis method and system based on distributed database cluster |
| CN114579604A (en) * | 2022-03-15 | 2022-06-03 | 北京梦诚科技有限公司 | Database transaction implementation method and system of application layer |
| CN116561374A (en) * | 2023-07-11 | 2023-08-08 | 腾讯科技（深圳）有限公司 | Resource determination method, device, equipment and medium based on semi-structured storage |
| CN116561374B (en) * | 2023-07-11 | 2024-02-23 | 腾讯科技（深圳）有限公司 | Resource determination method, device, equipment and medium based on semi-structured storage |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110019298B (en) | 2021-07-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110019298A (en) | Data processing method and device | |
| JP6709574B2 (en) | Terminal rule engine device and terminal rule operating method | |
| US11580441B2 (en) | Model training method and apparatus | |
| CN105630590B (en) | A kind of business information processing method and processing device | |
| JP6779231B2 (en) | Data processing method and system | |
| US20200034750A1 (en) | Generating artificial training data for machine-learning | |
| CN106557486A (en) | A kind of storage method and device of data | |
| CN108536650A (en) | Generate the method and apparatus that gradient promotes tree-model | |
| Chadha et al. | Towards federated learning using faas fabric | |
| Jangiti et al. | Scalable and direct vector bin-packing heuristic based on residual resource ratios for virtual machine placement in cloud data centers | |
| CN108415912A (en) | Data processing method based on MapReduce model and equipment | |
| CN109343793A (en) | Data migration method and device | |
| CN103064955A (en) | Inquiry planning method and device | |
| CN106648839A (en) | Method and device for processing data | |
| CN107391528A (en) | Front end assemblies Dependency Specification searching method and equipment | |
| CN115730507A (en) | Model engine construction method, kernel function processing method, device and storage medium | |
| CN110069453A (en) | Operation/maintenance data treating method and apparatus | |
| Nguyen et al. | A novel nature-inspired algorithm for optimal task scheduling in fog-cloud blockchain system | |
| CN108880896A (en) | A kind of program gray scale dissemination method and device based on mainframe | |
| CN108711074B (en) | Service classification method, device, server and readable storage medium | |
| CN117763024A (en) | Data fragment extraction method and device | |
| CN110765216A (en) | Data mining method and device, computer equipment and computer readable storage medium | |
| CN110083437A (en) | Handle the method and device of block chain affairs | |
| CN108256694A (en) | Based on Fuzzy time sequence forecasting system, the method and device for repeating genetic algorithm | |
| CN109165325A (en) | Method, apparatus, equipment and computer readable storage medium for cutting diagram data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100083 No. 401, 4th Floor, Haitai Building, 229 North Fourth Ring Road, Haidian District, Beijing Applicant after: Beijing Guoshuang Technology Co.,Ltd. Address before: 100086 Beijing city Haidian District Shuangyushu Area No. 76 Zhichun Road cuigongfandian 8 layer A Applicant before: Beijing Guoshuang Technology Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |