CN110019298B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN110019298B CN110019298B CN201711053329.4A CN201711053329A CN110019298B CN 110019298 B CN110019298 B CN 110019298B CN 201711053329 A CN201711053329 A CN 201711053329A CN 110019298 B CN110019298 B CN 110019298B
- Authority
- CN
- China
- Prior art keywords
- structured query
- query statement
- prediction model
- historical data
- statement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method and a data processing device. The method comprises the following steps: reading the query features contained in the structured query statement; inputting query features contained in the structured query statement into a pre-established memory prediction model to obtain a prediction result, wherein the memory prediction model is obtained by training according to historical data, and the historical data comprises the query features contained in the structured query statement and the quantity of resources required by the structured query statement; determining the quantity of resources required by the execution of the structured query statement according to the prediction result; and storing the structured query statement into a resource pool corresponding to the number of the resources for operation. By the method and the device, the effect of more accurate resource estimation quantity of the database is achieved.
Description
Technical Field
The invention relates to the field of computers, in particular to a data processing method and device.
Background
Impala is a distributed, interactive database based on a distributed File System (HDFS) or HBase storage System. Because the error of the resource estimation method of the impala is too large, in order to enable the cluster to stably operate, the resource quantity of each sql (Structured Query statement) can only be controlled through default setting, and when the actually required resource of the sql is larger than the resource quantity estimated by the impala, the sql is automatically cancelled. If a certain sql executed in parallel by the same resource pool actually requires a large resource, it may also affect the execution of other sql in the resource pool.
The existing resource estimation method has the following problems: the resource quantity estimation error is too large, and the sql cannot be reasonably scheduled to run in a proper resource pool.
Aiming at the problem that the cluster is unstable due to large error of the resource estimation quantity of the database in the related technology, an effective solution is not provided at present.
Disclosure of Invention
The invention mainly aims to provide a data processing method and a data processing device, which are used for solving the problem that a cluster is unstable due to large resource estimation quantity errors of a database.
In order to achieve the above object, according to an aspect of the present invention, there is provided a data processing method including: reading the query features contained in the structured query statement; inputting the query features contained in the structured query statement into a pre-established memory prediction model to obtain a prediction result, wherein the memory prediction model is obtained by training according to historical data, and the historical data comprises the query features contained in the structured query statement and the quantity of resources required by the structured query statement; determining the quantity of resources required by the execution of the structured query statement according to the prediction result; and storing the structured query statement into a resource pool corresponding to the resource quantity for operation.
Further, before inputting the query features included in the structured query statement into a pre-established memory prediction model, the method further includes: acquiring historical data, wherein the historical data comprises query characteristics of a plurality of groups of structured query sentences and the quantity of resources required by the plurality of groups of structured query sentences; and establishing the memory prediction model according to the historical data.
Further, the resource quantity corresponds to a plurality of levels, the query features included in the structured query statement are input into a pre-established memory prediction model, and obtaining a prediction result includes: and inputting the query features contained in the structured query statement into a pre-established memory prediction model to obtain the level of the predicted resource quantity.
Further, reading the query features contained in the structured query statement comprises: when a structured query statement submitted by a client through a Java database connection JDBC mode is received, the query features contained in the structured query statement are read, wherein the query features comprise join features and select features.
In order to achieve the above object, according to another aspect of the present invention, there is also provided a data processing apparatus comprising: the reading unit is used for reading the query features contained in the structured query statement; the input unit is used for inputting the query features contained in the structured query statement into a pre-established memory prediction model to obtain a prediction result, wherein the memory prediction model is obtained by training according to historical data, and the historical data comprises the query features contained in the structured query statement and the quantity of resources required by the structured query statement; the determining unit is used for determining the quantity of resources required by the execution of the structured query statement according to the prediction result; and the operation unit is used for storing the structured query statement into a resource pool corresponding to the resource quantity to operate.
Further, the apparatus further comprises: an obtaining unit, configured to obtain historical data before inputting query features included in the structured query statement into a pre-established memory prediction model, where the historical data includes query features of multiple groups of structured query statements and resource quantities required by the multiple groups of structured query statements; and the establishing unit is used for establishing the memory prediction model according to the historical data.
Further, the number of resources corresponds to a plurality of levels, the input unit is configured to: and inputting the query features contained in the structured query statement into a pre-established memory prediction model to obtain the level of the predicted resource quantity.
Further, the reading unit is configured to: when a structured query statement submitted by a client through a Java database connection JDBC mode is received, the query features contained in the structured query statement are read, wherein the query features comprise join features and select features.
In order to achieve the above object, according to another aspect of the present invention, there is also provided a storage medium including a stored program, wherein when the program runs, a device in which the storage medium is located is controlled to execute the data processing method according to the present invention.
In order to achieve the above object, according to another aspect of the present invention, there is also provided a processor for executing a program, wherein the program executes the data processing method according to the present invention.
The method comprises the steps of reading query features contained in a structured query statement; inputting query features contained in the structured query statement into a pre-established memory prediction model to obtain a prediction result, wherein the memory prediction model is obtained by training according to historical data, and the historical data comprises the query features contained in the structured query statement and the quantity of resources required by the structured query statement; determining the quantity of resources required by the execution of the structured query statement according to the prediction result; the structured query sentences are stored in the resource pool corresponding to the resource quantity to operate, so that the problem of cluster instability caused by large resource estimation quantity errors of the database is solved, and the effect of more accurate resource estimation quantity of the database is achieved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the invention and, together with the description, serve to explain the invention and not to limit the invention. In the drawings:
FIG. 1 is a flow diagram of a data processing method according to an embodiment of the invention;
fig. 2 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention.
Detailed Description
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present invention will be described in detail below with reference to the embodiments with reference to the attached drawings.
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances such that embodiments of the application described herein may be used. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For convenience of description, several terms referred to in the embodiments of the present application are explained below:
impala is a distributed, interactive database based on HDFS or HBase storage systems.
The feature extraction refers to extracting relevant attributes describing the samples according to the samples.
The embodiment of the invention provides a data processing method.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention, as shown in fig. 1, the method including the steps of:
step S102: reading the query features contained in the structured query statement;
step S104: inputting query features contained in the structured query statement into a pre-established memory prediction model to obtain a prediction result, wherein the memory prediction model is obtained by training according to historical data, and the historical data comprises the query features contained in the structured query statement and the quantity of resources required by the structured query statement;
step S106: determining the quantity of resources required by the execution of the structured query statement according to the prediction result;
step S108: and storing the structured query statement into a resource pool corresponding to the number of the resources for operation.
The embodiment adopts the method of reading the query features contained in the structured query statement; inputting query features contained in the structured query statement into a pre-established memory prediction model to obtain a prediction result, wherein the memory prediction model is obtained by training according to historical data, and the historical data comprises the query features contained in the structured query statement and the quantity of resources required by the structured query statement; determining the quantity of resources required by the execution of the structured query statement according to the prediction result; the structured query sentences are stored in the resource pool corresponding to the resource quantity to operate, so that the problem of cluster instability caused by large resource estimation quantity errors of the database is solved, and the effect of more accurate resource estimation quantity of the database is achieved.
The technical scheme of the embodiment of the invention can be applied to an impala database and is used as a method for optimizing cluster stability when the impala executes query. The Structured Query statement (sql) contains various Query features, for example, join, select and other features, after the features are read, the features can be input into a pre-established model for model operation to obtain a prediction result, and then the number of resources required for executing the sql is determined according to the prediction result, so that the sql can be stored into a corresponding resource pool for operation.
Before the query features contained in the structured query sentences are input into a pre-established memory prediction model, acquiring historical data, wherein the historical data comprises the query features of a plurality of groups of structured query sentences and the resource quantity required by the plurality of groups of structured query sentences; and establishing a memory prediction model according to the historical data.
The model building needs to be built through multiple groups of historical data, each group of historical data comprises the query features of sql, the feature parameters required to be scanned when the sql is executed and the number of corresponding memory resources, after the historical data is obtained, the historical data is classified and sorted, and a memory prediction model can be built based on the historical data. The characteristic parameters may be the number of files that sql performs the required scan, the number of partitions of the table, the total number of partitions, the hash number, the agg number, etc.
Optionally, the resource quantity corresponds to multiple levels, and the step of inputting the query features included in the structured query statement into a pre-established memory prediction model to obtain the prediction result includes: and inputting the query features contained in the structured query statement into a pre-established memory prediction model to obtain the level of the predicted resource quantity. The resource quantity can be divided into a plurality of grades, the predicted resource quantity can be the grade of the predicted resource quantity, and the grade can be directly obtained without obtaining the resource quantity, so that the model can be simplified.
In the technical scheme of the embodiment of the invention, before the structured query statement is stored in the resource pool corresponding to the resource quantity to operate, the impala cluster memory is divided into the memory pools with a plurality of levels, such as the memory pools with different levels of 200G, 400G and the like, after the memory resource quantity required by a certain sql statement is obtained through a model, the resource pool with the proper level can be determined in the memory to be used for the sql operation, and then the sql is put into the corresponding resource pool to operate.
Optionally, reading the query features contained in the structured query statement comprises: when a structured query statement submitted by a client in a JDBC mode is received, the query features contained in the structured query statement are read, wherein the query features comprise a join feature and a select feature.
As an alternative, the sql may be submitted by the client by means of JDBC (Java, DataBase Connectivity), and after receipt, the query features contained in the sql may be read.
The embodiment of the invention also provides a preferable implementation mode, which comprises the following parts:
1. extracting features such as the number of files required by the sql to execute scanning, the number of partitions of a table, the total number of partitions, the hash number, the agg number and the like through the explain statement, and analyzing the features such as join, select and the like contained in the sql.
2. And establishing a memory prediction model according to certain sql historical data, and adjusting the model or parameters according to the test sample. Preferably, a classification algorithm is selected to classify the prediction of the memory into a plurality of classes.
3. When a client submits sql through JDBC and other modes, the features are extracted, then the sql is put into a corresponding resource pool according to a result obtained by prediction of a prediction model, resources required by execution of the sql are set, and finally the sql is executed.
After this process, there are two main benefits for impala clustering. Firstly, the stability of the cluster is enhanced, and the client can put the sql into a resource pool with sufficient resources to execute according to a prediction result obtained by a prediction model, so that the execution success rate of the sql can be improved, and the influence of the large sql on other concurrent sql can be reduced. And secondly, operation and maintenance colleagues can decide the increase and decrease of the cluster scale according to the historical data of the prediction model, so that the software and hardware costs of the cluster are saved.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.
The embodiment of the invention provides a data processing device, which can be used for executing the data processing method of the embodiment of the invention.
Fig. 2 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention, as shown in fig. 2, the apparatus including:
a reading unit 10, configured to read a query feature included in the structured query statement;
the input unit 20 is configured to input query features included in the structured query statement into a pre-established memory prediction model to obtain a prediction result, where the memory prediction model is obtained by training according to historical data, and the historical data includes the query features included in the structured query statement and a resource quantity required by the structured query statement;
a determining unit 30, configured to determine, according to the prediction result, the number of resources required for executing the structured query statement;
and the running unit 40 is used for storing the structured query statement into a resource pool corresponding to the number of the resources to run.
The embodiment adopts a reading unit 10, configured to read a query feature included in a structured query statement; the input unit 20 is configured to input query features included in the structured query statement into a pre-established memory prediction model to obtain a prediction result, where the memory prediction model is obtained by training according to historical data, and the historical data includes the query features included in the structured query statement and a resource quantity required by the structured query statement; a determining unit 30, configured to determine, according to the prediction result, the number of resources required for executing the structured query statement; the operation unit 40 is configured to store the structured query statement into a resource pool corresponding to the number of resources to operate, so that the problem of cluster instability caused by a large error in the number of resource estimation of the database is solved, and an effect of more accurate number of resource estimation of the database is achieved.
Optionally, the apparatus further comprises: the device comprises an acquisition unit, a storage unit and a processing unit, wherein the acquisition unit is used for acquiring historical data before the query features contained in the structured query sentences are input into a pre-established memory prediction model, and the historical data comprises the query features of a plurality of groups of structured query sentences and the resource quantity required by the plurality of groups of structured query sentences; and the establishing unit is used for establishing a memory prediction model according to the historical data.
Optionally, the resource quantity corresponds to multiple levels, and the input unit 20 is configured to input the query features included in the structured query statement into a pre-established memory prediction model, so as to obtain the level of the predicted resource quantity.
Optionally, the reading unit 10 is configured to: when a structured query statement submitted by a client through a Java database connection JDBC mode is received, the query features contained in the structured query statement are read, wherein the query features comprise join features and select features.
The data processing device comprises a processor and a memory, wherein the reading unit, the input unit, the determining unit, the running unit and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.
The processor comprises a kernel, and the kernel calls the corresponding program unit from the memory. The kernel can be set to be one or more, and the resource estimation quantity of the database is more accurate by adjusting the kernel parameters.
The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.
An embodiment of the present invention provides a storage medium on which a program is stored, the program implementing the data processing method when executed by a processor.
The embodiment of the invention provides a processor, which is used for running a program, wherein the data processing method is executed when the program runs.
The embodiment of the invention provides equipment, which comprises a processor, a memory and a program which is stored on the memory and can run on the processor, wherein the processor executes the program and realizes the following steps: reading the query features contained in the structured query statement; inputting query features contained in the structured query statement into a pre-established memory prediction model to obtain a prediction result, wherein the memory prediction model is obtained by training according to historical data, and the historical data comprises the query features contained in the structured query statement and the quantity of resources required by the structured query statement; determining the quantity of resources required by the execution of the structured query statement according to the prediction result; and storing the structured query statement into a resource pool corresponding to the number of the resources for operation.
Acquiring historical data, wherein the historical data comprises query characteristics of a plurality of groups of structured query sentences and the quantity of resources required by the plurality of groups of structured query sentences; and establishing a memory prediction model according to the historical data.
And inputting the query features contained in the structured query statement into a pre-established memory prediction model to obtain the level of the predicted resource quantity.
Reading the query features contained in the structured query statement includes: and when receiving a structured query statement submitted by a client in a Java database connection JDBC mode, reading the query features contained in the structured query statement. The device herein may be a server, a PC, a PAD, a mobile phone, etc.
The present application further provides a computer program product adapted to perform a program for initializing the following method steps when executed on a data processing device: reading the query features contained in the structured query statement; inputting query features contained in the structured query statement into a pre-established memory prediction model to obtain a prediction result, wherein the memory prediction model is obtained by training according to historical data, and the historical data comprises the query features contained in the structured query statement and the quantity of resources required by the structured query statement; determining the quantity of resources required by the execution of the structured query statement according to the prediction result; and storing the structured query statement into a resource pool corresponding to the number of the resources for operation.
Acquiring historical data, wherein the historical data comprises query characteristics of a plurality of groups of structured query sentences and the quantity of resources required by the plurality of groups of structured query sentences; and establishing a memory prediction model according to the historical data.
And inputting the query features contained in the structured query statement into a pre-established memory prediction model to obtain the level of the predicted resource quantity.
Reading the query features contained in the structured query statement includes: and when receiving a structured query statement submitted by a client in a Java database connection JDBC mode, reading the query features contained in the structured query statement.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (8)
1. A data processing method, comprising:
reading the query features contained in the structured query statement;
inputting the query features contained in the structured query statement into a pre-established memory prediction model to obtain a prediction result, wherein the memory prediction model is obtained by training according to historical data, and the historical data comprises the query features contained in the structured query statement and the quantity of resources required by the structured query statement;
determining the quantity of resources required by the execution of the structured query statement according to the prediction result;
storing the structured query statement into a resource pool corresponding to the resource quantity for operation;
before inputting the query features contained in the structured query statement into a pre-established memory prediction model, the method further comprises:
acquiring historical data, wherein the historical data comprises query characteristics of a plurality of groups of structured query sentences and the quantity of resources required by the plurality of groups of structured query sentences;
and establishing the memory prediction model according to the historical data.
2. The method of claim 1, wherein the number of resources corresponds to a plurality of levels, and inputting the query features contained in the structured query statement into a pre-established memory prediction model to obtain a prediction result comprises:
and inputting the query features contained in the structured query statement into a pre-established memory prediction model to obtain the level of the predicted resource quantity.
3. The method of claim 1, wherein reading the query features contained in the structured query statement comprises:
when a structured query statement submitted by a client through a Java database connection JDBC mode is received, the query features contained in the structured query statement are read, wherein the query features comprise join features and select features.
4. A data processing apparatus, comprising:
the reading unit is used for reading the query features contained in the structured query statement;
the input unit is used for inputting the query features contained in the structured query statement into a pre-established memory prediction model to obtain a prediction result, wherein the memory prediction model is obtained by training according to historical data, and the historical data comprises the query features contained in the structured query statement and the quantity of resources required by the structured query statement;
the determining unit is used for determining the quantity of resources required by the execution of the structured query statement according to the prediction result;
the operation unit is used for storing the structured query statement into a resource pool corresponding to the resource quantity to operate;
wherein the apparatus further comprises:
an obtaining unit, configured to obtain historical data before inputting query features included in the structured query statement into a pre-established memory prediction model, where the historical data includes query features of multiple groups of structured query statements and resource quantities required by the multiple groups of structured query statements;
and the establishing unit is used for establishing the memory prediction model according to the historical data.
5. The apparatus of claim 4, wherein the number of resources corresponds to a plurality of levels, and wherein the input unit is configured to:
and inputting the query features contained in the structured query statement into a pre-established memory prediction model to obtain the level of the predicted resource quantity.
6. The apparatus of claim 4, wherein the reading unit is configured to:
when a structured query statement submitted by a client through a Java database connection JDBC mode is received, the query features contained in the structured query statement are read, wherein the query features comprise join features and select features.
7. A storage medium, characterized in that the storage medium comprises a stored program, wherein when the program runs, a device in which the storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 3.
8. A processor, characterized in that the processor is configured to run a program, wherein the program is configured to execute the data processing method according to any one of claims 1 to 3 when running.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711053329.4A CN110019298B (en) | 2017-10-31 | 2017-10-31 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711053329.4A CN110019298B (en) | 2017-10-31 | 2017-10-31 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110019298A CN110019298A (en) | 2019-07-16 |
| CN110019298B true CN110019298B (en) | 2021-07-30 |
Family
ID=67186736
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711053329.4A Active CN110019298B (en) | 2017-10-31 | 2017-10-31 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110019298B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109324905B (en) * | 2018-09-30 | 2021-03-30 | 拉卡拉支付股份有限公司 | Database operation method and device, electronic equipment and storage medium |
| CN112699140B (en) * | 2019-10-23 | 2023-12-26 | 阿里巴巴集团控股有限公司 | Data processing method, device, equipment and storage medium |
| CN111046059B (en) * | 2019-12-09 | 2023-06-30 | 中国建设银行股份有限公司 | Low-efficiency SQL statement analysis method and system based on distributed database cluster |
| CN114579604B (en) * | 2022-03-15 | 2022-09-20 | 北京梦诚科技有限公司 | Database transaction implementation method and system of application layer |
| CN116561374B (en) * | 2023-07-11 | 2024-02-23 | 腾讯科技（深圳）有限公司 | Resource determination method, device, equipment and medium based on semi-structured storage |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101969536B (en) * | 2002-01-30 | 2013-02-20 | 索尼公司 | Data processing apparatus and data processing method used for the data processing apparatus |
| CN101408900B (en) * | 2008-11-24 | 2011-03-16 | 中国科学院地理科学与资源研究所 | Distributed space data enquiring and optimizing method under gridding calculation environment |
| CN102004671B (en) * | 2010-11-15 | 2013-03-13 | 北京航空航天大学 | Resource management method of data center based on statistic model in cloud computing environment |
| CN103186603B (en) * | 2011-12-29 | 2016-04-27 | 中国移动通信集团浙江有限公司 | Determine that SQL statement is on the method for the impact of the performance of key business, system and equipment |
| CN105893224B (en) * | 2015-01-26 | 2019-04-09 | 阿里巴巴集团控股有限公司 | A kind of resource measurement method and device |
| US9934278B2 (en) * | 2015-05-04 | 2018-04-03 | Quest Software Inc. | Method of optimizing complex SQL statements using a region divided preferential SQL rewrite operation |
| US10621152B2 (en) * | 2015-12-02 | 2020-04-14 | Speedment, Inc. | Methods and systems for mapping object oriented/functional languages to database languages |
| CN107194680B (en) * | 2016-03-14 | 2020-07-03 | 阿里巴巴集团控股有限公司 | Configurable resource quantity control method and device |
| CN105868274A (en) * | 2016-03-22 | 2016-08-17 | 努比亚技术有限公司 | Resource data querying and processing method and device thereof |
| CN106844678A (en) * | 2017-01-24 | 2017-06-13 | 山东浪潮商用系统有限公司 | A kind of exchange method of Mybatis data sources and connection pool |
-
2017
- 2017-10-31 CN CN201711053329.4A patent/CN110019298B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN110019298A (en) | 2019-07-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110019298B (en) | Data processing method and device | |
| CN106407207B (en) | Real-time newly-added data updating method and device | |
| KR102134952B1 (en) | Data processing method and system | |
| EP3226155A1 (en) | System and method for database migration with target platform scalability | |
| CN111241073B (en) | Data quality inspection method and device | |
| CN110134738B (en) | Distributed storage system resource estimation method and device | |
| CN115129782A (en) | Partition level connection method and device for distributed database | |
| CN110895544A (en) | Interface data processing method, device, system and storage medium | |
| CN110737717A (en) | database migration method and device | |
| CN111125088B (en) | Multi-level data processing method and device | |
| CN112580915A (en) | Project milestone determination method and device, storage medium and electronic equipment | |
| CN109284268A (en) | A kind of method, system and the electronic equipment of fast resolving log | |
| US20190087156A1 (en) | Sorting a table in analytical databases | |
| CN111078905A (en) | Data processing method, device, medium and equipment | |
| CN111125087A (en) | Data storage method and device | |
| US11481517B2 (en) | System and method for determining permission profiles for computer executable functions | |
| CN114817209A (en) | Monitoring rule processing method and device, processor and electronic equipment | |
| CN115033551A (en) | Database migration method and device, electronic equipment and storage medium | |
| CN109902067B (en) | File processing method and device, storage medium and computer equipment | |
| CN112463785A (en) | Data quality monitoring method and device, electronic equipment and storage medium | |
| CN112749189A (en) | Data query method and device | |
| CN106557469B (en) | Method and device for processing data in data warehouse | |
| CN113687973B (en) | Control method, equipment and medium for log dynamic output | |
| CN116561099A (en) | Open Gauss database performance optimization method and system for NUMA CPU architecture | |
| CN110852539A (en) | Drawing processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 100083 No. 401, 4th Floor, Haitai Building, 229 North Fourth Ring Road, Haidian District, Beijing Applicant after: Beijing Guoshuang Technology Co.,Ltd. Address before: 100086 Beijing city Haidian District Shuangyushu Area No. 76 Zhichun Road cuigongfandian 8 layer A Applicant before: Beijing Guoshuang Technology Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |