CN111858606A - Data processing method and device and electronic equipment - Google Patents
Data processing method and device and electronic equipment Download PDFInfo
- Publication number
- CN111858606A CN111858606A CN202010727974.5A CN202010727974A CN111858606A CN 111858606 A CN111858606 A CN 111858606A CN 202010727974 A CN202010727974 A CN 202010727974A CN 111858606 A CN111858606 A CN 111858606A
- Authority
- CN
- China
- Prior art keywords
- data
- tree
- data index
- target
- index range
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 34
- 238000012545 processing Methods 0.000 claims abstract description 43
- 238000000034 method Methods 0.000 claims description 40
- 238000004590 computer program Methods 0.000 claims description 5
- 238000010276 construction Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 description 12
- 238000004891 communication Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000006399 behavior Effects 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2246—Trees, e.g. B+trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2379—Updates performed during online database operations; commit processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
- G06F9/526—Mutual exclusion algorithms
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data processing method, a data processing device and electronic equipment, wherein the data processing method comprises the following steps: acquiring a preset index range, wherein the preset index range is a preset range interval of index key values for indexing data in a database; dividing a preset index range into a plurality of data index ranges, and constructing a corresponding B-tree for each data index range; and executing the specified data operation on the data based on each data index range and the B tree corresponding to each data index range. According to the above description, after the preset index range is divided into the plurality of data index ranges, when the designated data operation is executed on the data in the database, the data can be executed in a partitioned and concurrent manner, so that the data processing efficiency is improved, the data processing time is saved, and the technical problems of low efficiency and serious time consumption of the conventional data processing method are solved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method and apparatus, and an electronic device.
Background
With the rapid development of mobile internet, social networks and electronic commerce, the amount of information is increasing day by day, and thus, massive data is stored in the database. Many data tables in the database store tens of millions or even billions of data, and the data processing is usually realized by means of indexes. Currently, commonly used indexes include B-trees or B + trees.
For a data table with mass data, a B-tree is corresponding to the data table, and when data query or data update is performed for one time, the data query or data update is often implemented in a binary search manner based on the B-tree. The number of queries for a single lookup is not large, but in the case of high concurrency (for example, 1000 data queries are performed simultaneously), the number of queries is multiplied (the number of queries is the number of queries for a single lookup x the number of concurrency). In addition, when data updating (including data writing and data reading) is performed on the data table once, the whole B tree corresponding to the data table is in a locked state, and when the data table is updated 1000 times at the same time, the data updating process can only be performed in series, which seriously affects the efficiency of data updating and consumes time seriously.
Disclosure of Invention
In view of this, the present invention provides a data processing method, an apparatus and an electronic device, so as to alleviate the technical problems of low efficiency and time consumption of the existing data processing method.
In a first aspect, an embodiment of the present invention provides a data processing method, including: acquiring a preset index range, wherein the preset index range is a preset range interval of index key values for indexing data in a database; dividing the preset index range into a plurality of data index ranges, and constructing a corresponding B-tree for each data index range; and executing specified data operation on the data based on each data index range and the B tree corresponding to each data index range.
Further, performing the specified data operation on the data based on each of the data index ranges and the B-tree corresponding to each of the data index ranges includes: when the specified data operation is a data updating operation, determining a data index key value of data to be updated; determining a data index range to which a data index key value of the data to be updated belongs in the plurality of data index ranges; constructing a first target B-tree node on the B-tree corresponding to the data index range to which the data index key value of the data to be updated belongs; and updating the data to be updated according to the index key value of the first target B-tree node and the storage position corresponding to the first target B-tree node.
Further, performing the specified data operation on the data based on each data index range and the B-tree corresponding to each data index range further includes: when the specified data operation is a data reading operation, determining a data index key value of data to be read; determining a data index range to which a data index key value of the data to be read belongs in the plurality of data index ranges; determining a B tree corresponding to a data index range to which a data index key value of the data to be read belongs; and traversing and searching the B tree corresponding to the data index range to which the data index key value of the data to be read belongs to obtain the storage position of the data to be read in the database, and reading the data to be read according to the storage position.
Further, before performing the specified data operation on the data based on each of the data index ranges and the B-tree corresponding to each of the data index ranges, the method further includes: determining a target data index range corresponding to the specified data operation; and controlling the B tree corresponding to the target data index range to be in a locked state.
Further, controlling the B-tree corresponding to the target data index range to be in a locked state includes: setting a locking identifier for the B tree corresponding to the target data index range; and controlling the B tree corresponding to the target data index range to be in a locking state according to the locking identifier.
In a second aspect, an embodiment of the present invention further provides a data processing method, including: acquiring a data index key value of data to be read; determining a first target data index range in a plurality of data index ranges according to the data index key values, wherein each data index range contains the data index key value of each data in the database; each data index range corresponds to a B tree; and determining the storage position of the data to be read in the database based on the first target data index range and the corresponding first target B-tree, and reading the data to be read according to the storage position.
Furthermore, each data index range corresponds to one B-tree pointer; determining the storage location of the data to be read in the database based on the first target data index range and the corresponding first target B-tree thereof comprises: determining the first target B-tree according to the B-tree pointers of the first target data index range; and determining the storage position of the data to be read in the database based on the first target B-tree.
Further, before the determining the storage location of the data to be read in the database based on the first target B-tree, the method further includes: setting a locking identifier for the first target B-tree; and controlling the first target B-tree to be in a locking state according to the locking identifier.
In a third aspect, an embodiment of the present invention further provides a data processing method, including: acquiring a data index key value of data to be updated; determining a second target data index range to which the data to be updated belongs in a plurality of data index ranges according to the data index key values, wherein each data index range contains the data index key value of each data in the database; each data index range corresponds to a B tree; constructing a second target B-tree node on the B-tree corresponding to the second target data index range; and updating the data to be updated according to the index key value of the second target B-tree node and the storage position corresponding to the second target B-tree node.
Further, before constructing a second target B-tree node on the B-tree corresponding to the second target data index range, the method further includes: setting a locking identifier for the B-tree corresponding to the second target data index range; and controlling the B tree corresponding to the second target data index range to be in a locking state according to the locking identifier.
In a fourth aspect, an embodiment of the present invention further provides a data processing apparatus, including: the device comprises a first obtaining unit, a second obtaining unit and a third obtaining unit, wherein the first obtaining unit is used for obtaining a preset index range, and the preset index range is a preset range interval of index key values for indexing data in a database; the dividing unit is used for dividing the preset index range into a plurality of data index ranges and constructing a corresponding B-tree for each data index range; and the operation execution unit is used for executing specified data operation on the data based on each data index range and the B tree corresponding to each data index range.
In a fifth aspect, an embodiment of the present invention further provides a data processing apparatus, including: the second acquisition unit is used for acquiring a data index key value of data to be read; a first determining unit, configured to determine a first target data index range in multiple data index ranges according to the data index key values, where each data index range includes data index key values of respective data in a database; each data index range corresponds to a B tree; and the second determining unit is used for determining the storage position of the data to be read in the database based on the first target data index range and the corresponding first target B-tree thereof, and reading the data to be read according to the storage position.
In a sixth aspect, an embodiment of the present invention further provides a data processing apparatus, including: the third acquisition unit is used for acquiring a data index key value of the data to be updated; a third determining unit, configured to determine, according to the data index key values, a second target data index range to which the data to be updated belongs in multiple data index ranges, where each data index range includes the data index key value of each data in the database; each data index range corresponds to a B tree; the construction unit is used for constructing a second target B-tree node on the B-tree corresponding to the second target data index range; and the updating unit is used for updating the data to be updated according to the index key value of the second target B-tree node and the storage position corresponding to the second target B-tree node.
In a seventh aspect, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the method in any one of the first aspect, the second aspect, or the third aspect.
In an eighth aspect, the present invention further provides a computer-readable medium having non-volatile program code executable by a processor, where the program code causes the processor to execute the method of any one of the first aspect, the second aspect, or the third aspect.
In the embodiment of the invention, firstly, a preset index range is obtained, wherein the preset index range is a preset range interval of index key values for indexing data in a database; dividing a preset index range into a plurality of data index ranges, and constructing a corresponding B-tree for each data index range; and executing the specified data operation on the data based on each data index range and the B tree corresponding to each data index range. According to the above description, after the preset index range is divided into the plurality of data index ranges, when the designated data operation is executed on the data in the database, the data can be executed in a partitioned and concurrent manner, so that the data processing efficiency is improved, the data processing time is saved, and the technical problems of low efficiency and serious time consumption of the conventional data processing method are solved.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a linked list provided by an embodiment of the present invention;
FIG. 3 is a flowchart illustrating operations performed on data in a database according to an embodiment of the present invention;
FIG. 4 is a flow chart illustrating another operation performed on data in a database according to an embodiment of the present invention;
FIG. 5 is a flow chart illustrating another operation performed on data in a database according to an embodiment of the present invention;
FIG. 6 is a flow chart of a data processing method according to an embodiment of the present invention;
FIG. 7 is a flow chart of a data processing method according to an embodiment of the present invention;
FIG. 8 is a diagram illustrating a data processing apparatus according to an embodiment of the present invention;
FIG. 9 is a schematic diagram of another data processing apparatus according to an embodiment of the present invention;
FIG. 10 is a diagram of another data processing apparatus according to an embodiment of the present invention;
fig. 11 is a schematic view of an electronic device according to an embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The first embodiment is as follows:
in accordance with an embodiment of the present invention, there is provided an embodiment of a data processing method, it should be noted that the steps illustrated in the flowchart of the accompanying drawings may be performed in a computer system such as a set of computer executable instructions, and that while a logical order is illustrated in the flowchart, in some cases, the steps illustrated or described may be performed in an order different than that herein.
Fig. 1 is a flow chart of a data processing method according to an embodiment of the present invention, as shown in fig. 1, the method including the steps of:
step S102, a preset index range is obtained, wherein the preset index range is a preset range interval of index key values for indexing data in a database.
In the application, the preset index range may be a range interval of index key values key for indexing data in the database, which is customized by a user.
For ease of understanding, with reference to fig. 2, the data processing method of the present invention is described:
in fig. 2, the range from 0 to 10099 is a preset index range, which is not specifically limited in the embodiment of the present invention.
And step S104, dividing a preset index range into a plurality of data index ranges, and constructing a corresponding B-tree for each data index range.
After the preset index range is obtained, the preset index range is further divided into a plurality of data index ranges. When the data index range is divided, a preset range division interval is obtained first, and then the preset index range is divided into a plurality of data index ranges according to the preset range division interval, wherein the plurality of data index ranges can form a target linked list.
It should be noted that there may be one or more preset range division regions. When the preset range division areas are multiple, the index range to be divided corresponding to each preset range division area can be defined, and then the index range to be divided corresponding to each preset range division area is divided according to each preset range division area. When one preset range division area is provided, the range of each data index range obtained by division is equal in size; when the preset range division areas are multiple, the range size of the data index ranges obtained by dividing the preset range division areas is equal, and the range size of the data index ranges obtained by dividing the preset range division areas is different.
As shown in fig. 2, the division interval of the preset range is 100, and the data index range obtained by final division includes: 0-99, 100-199, 200-299, 300-399, …, 10000-10099, and further obtaining a target linked list consisting of a plurality of data index ranges. Each data index range of the target linked list comprises a data field and a pointer field, data in the data field is used for representing the index range, the pointer field comprises two pointers, one pointer points to a B-tree corresponding to the data index range (the pointer can be called as a B-tree pointer), and the other pointer points to the next node in the target linked list.
And step S106, executing specified data operation on the data based on each data index range and the B-tree corresponding to each data index range.
After obtaining the plurality of data index ranges, the specified data operation can be executed on the data in the database based on each data index range and the B-tree corresponding to each data index range. When the specified data operation is executed, the partitions can be executed concurrently. For example, for 1000 data index ranges, when the specified data operation is performed, the corresponding data index ranges are different, so that the specified data operation can be performed on the data in the 1000 data index ranges at the same time, the data processing efficiency is improved, and the data processing time is saved.
In the embodiment of the invention, firstly, a preset index range is obtained, wherein the preset index range is a preset range interval of index key values for indexing data in a database; dividing a preset index range into a plurality of data index ranges, and constructing a corresponding B-tree for each data index range; and executing the specified data operation on the data based on each data index range and the B tree corresponding to each data index range. According to the above description, after the preset index range is divided into the plurality of data index ranges, when the designated data operation is executed on the data in the database, the data can be executed in a partitioned and concurrent manner, so that the data processing efficiency is improved, the data processing time is saved, and the technical problems of low efficiency and serious time consumption of the conventional data processing method are solved.
The foregoing has outlined the data processing method of the present invention and the detailed description that follows is provided.
In an alternative embodiment of the present application, referring to fig. 3, in the step S106, performing the specified data operation on the data based on each data index range and the B-tree corresponding to each data index range includes the following steps:
step S301, when the specified data operation is a data updating operation, determining a data index key value of data to be updated to obtain a first data index key value;
step S302, determining a data index range to which a data index key value of data to be updated belongs in a plurality of data index ranges;
step S303, constructing a first target B-tree node on a B-tree corresponding to a data index range to which a data index key value of the data to be updated belongs;
step S304, updating the data to be updated according to the index key value of the first target B-tree node and the storage position corresponding to the first target B-tree node.
The following describes the process of data update with a specific example:
if the data index key value of the data to be updated is 201, the data index range to which the data index key value 201 belongs can be determined to be the data index range of 200-299 in fig. 2 according to a plurality of data index ranges obtained by dividing the preset range, and then a corresponding B-tree node (i.e., a first target B-tree node) is established on the B-tree corresponding to the data index range of 200-299, that is, the corresponding node is inserted into the B-tree according to the building principle of the B-tree. After the first target node B is established, the data to be updated may be updated according to the data index key value of the first target node B and the storage location corresponding to the first target node B.
In another alternative embodiment of the present application, referring to fig. 4, in step S106, the performing the specified data operation on the data based on each data index range and the B-tree corresponding to each data index range further includes the following steps:
step S401, when the specified data operation is a data reading operation, determining a data index key value of data to be read;
step S402, determining a data index range to which a data index key value of data to be read belongs in a plurality of data index ranges;
step S403, determining a B tree corresponding to a data index range to which a data index key value of data to be read belongs;
step S404, traversing and searching the B-tree corresponding to the data index range of the data index key value of the data to be read, obtaining the storage position of the data to be read in the database, and reading the data to be read according to the storage position.
The following describes the process of data query with a specific example:
if the data index key value of the data to be read is 201, the data index range to which the data index key value 201 belongs can be determined to be the data index range of 200-299 in fig. 2 according to a plurality of data index ranges obtained by dividing the preset range, then traversal search is performed on the B-tree corresponding to the data index range of 200-299, the storage position of the data to be read in the database is obtained, and the data to be read stored in the storage position is read.
In another alternative embodiment of the present application, referring to fig. 5, before performing the specified data operation on the data based on each data index range and the B-tree corresponding to each data index range, the method further includes the following steps:
step S501, determining a target data index range corresponding to the specified data operation;
specifically, the data indexing ranges are obtained by dividing the regions according to a preset range, then the data indexing key values corresponding to the specified data operation are determined, and further the target data indexing range to which the data indexing key values belong is determined in the data indexing ranges, wherein the target data indexing range is the target data indexing range corresponding to the specified data operation. As shown in FIG. 2, if the division interval of the preset range is 100, the division range of the obtained multiple data indexes is 0 to 99, 100 to 199, 200 to 299, 300 to 399, …, 10000 to 10099 respectively. If the data index key value corresponding to the specified data operation is 201, it can be determined that the target data index range to which 201 belongs is 200-299, that is, the data index range to which the target data index range corresponding to the specified data operation is 200-299.
Step S502, controlling the B tree corresponding to the target data index range to be in a locking state;
specifically, a locking identifier may be set for the B-tree corresponding to the target data index range, where the locking identifier is used to represent that the B-tree corresponding to the target data index range is in a locked state; and controlling the B tree corresponding to the target data index range to be in a locking state according to the locking identification.
When the B-tree corresponding to the target data index range is in a locked state, the specified data operation can be executed on the data in the database based on the target data index range and the B-tree corresponding to the target data index range.
If the specified data operation is a data updating operation, constructing a corresponding B-tree node on the target B-tree (namely the B-tree corresponding to the target data index range); if the specified data operation is a data reading operation, traversing and searching a target B-tree (namely the B-tree corresponding to the target data index range) to obtain the storage position of the data to be read in the database, and reading the data to be read according to the storage position.
It can be seen from the above process of executing the specified data operation that, when data is processed, only the B-trees corresponding to the control target data index ranges are in the locked state, and it is not necessary to make all the B-trees corresponding to the data index ranges in the locked state. Therefore, when the index ranges of the parallel target data are different, corresponding operations can be executed on a plurality of corresponding B trees simultaneously, conflict is avoided, and the data processing efficiency is improved.
The data processing method of the invention has the following advantages: the B trees corresponding to different data index ranges can be operated simultaneously, data concurrent processing corresponding to different partitions can be realized, and the data processing efficiency is improved; the height of the B tree is reduced, the query times are reduced, further, the query of useless disks is avoided, and the query times of IO are reduced.
Example two:
fig. 6 is a flowchart of a data processing method according to an embodiment of the present invention, as shown in fig. 6, the method including the steps of:
step S602, acquiring a data index key value of data to be read;
step S604, determining a first target data index range in a plurality of data index ranges according to data index key values, wherein each data index range contains the data index key value of each data in the database; each data index range corresponds to a B-tree.
Referring to FIG. 2, the first behavior of FIG. 2 is a target linked list in which a plurality of data index ranges as described above are included. If the data index key value is 201, it may be determined that a first target data index range of the data index key value of 201 in the target linked list is a data index range of 200 to 299 in fig. 2.
Step S606, determining a storage location of the data to be read in the database based on the first target data index range and the corresponding first target B-tree, and reading the data to be read according to the storage location.
In an optional embodiment, if each data index range corresponds to one B-tree pointer, determining a storage location of the data to be read in the database based on the first target data index range and the corresponding first target B-tree includes the following processes:
determining a first target B-tree according to the B-tree pointers of the first target data index range; and determining the storage position of the data to be read in the database based on the first target B-tree.
Before determining the storage position of the data to be read in the database based on the first target B-tree, a locking identifier may also be set for the first target B-tree, and the first target B-tree is controlled to be in a locking state according to the locking identifier.
In the data processing method, when the data index key value of the data to be read is in different data index ranges, the data reading process can be executed in parallel, and the data reading efficiency is improved.
Example three:
fig. 7 is a flowchart of a data processing method according to an embodiment of the present invention, as shown in fig. 7, the method including the steps of:
step S702, acquiring a data index key value of data to be updated;
step S704, determining a second target data index range to which the data to be updated belongs in a plurality of data index ranges according to the data index key values, wherein each data index range contains the data index key value of each data in the database; each data index range corresponds to a B-tree.
Referring to FIG. 2, the first behavior of FIG. 2 is a target linked list in which a plurality of data index ranges as described above are included. If the data index key value is 201, it may be determined that the second target data index range of the data index key value 201 in the target linked list is the data index range of 200 to 299 in fig. 2.
Step S706, constructing a second target B-tree node on the B-tree corresponding to the second target data index range; specifically, when a second target B-tree node on the B-tree corresponding to the second target data index range is constructed, the index key value of the second target B-tree node may be set as the data index key value of the data to be updated, and the storage location corresponding to the second target B-tree node is set as the storage location of the data to be updated in the database, so as to implement the data updating operation on the data to be updated.
Step S708, the data to be updated is updated according to the index key value of the second target B-tree node and the storage location corresponding to the second target B-tree node.
Each data index range corresponds to one B-tree pointer, and specifically, the B-tree corresponding to the second target data index range may be determined according to the B-tree pointer of the second target data index range.
In an optional embodiment of the present application, before constructing the second target B-tree node on the B-tree corresponding to the second target data index range, the method further includes: setting a locking identifier for the B-tree corresponding to the second target data index range; and controlling the B tree corresponding to the second target data index range to be in a locking state according to the locking identifier.
Specifically, in the present application, a lock identifier may be set for the B-tree corresponding to the second target data index range, where the lock identifier is used to represent that the B-tree corresponding to the second target data index range is in a lock state. Then, when the target data index range is in the locked state, a second target B-tree node on the B-tree corresponding to the second target data index range is constructed.
In the data processing method, when the data index key values of the data to be updated are in different data index ranges, the data updating process can be executed in parallel, and the data updating efficiency is improved.
Example four:
the embodiment of the present invention further provides a data processing apparatus, where the data processing apparatus is mainly configured to execute the data processing method provided in the first embodiment of the present invention, and the data processing apparatus provided in the embodiment of the present invention is specifically described below.
Fig. 8 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention, as shown in fig. 8, the data processing apparatus mainly includes: a first acquisition unit 81, a dividing unit 82, and an operation execution unit 83, wherein:
a first obtaining unit 81, configured to obtain a preset index range, where the preset index range is a preset range interval of index key values for indexing data in a database;
a dividing unit 82, configured to divide the preset index range into multiple data index ranges, and construct a corresponding B-tree for each data index range;
and an operation executing unit 83, configured to execute a specified data operation on the data based on each data index range and the B-tree corresponding to each data index range.
In the embodiment of the invention, firstly, a preset index range is obtained, wherein the preset index range is a preset range interval of index key values for indexing data in a database; dividing a preset index range into a plurality of data index ranges, and constructing a corresponding B-tree for each data index range; and executing the specified data operation on the data based on each data index range and the B tree corresponding to each data index range. According to the above description, after the preset index range is divided into the plurality of data index ranges, when the designated data operation is executed on the data in the database, the data can be executed in a partitioned and concurrent manner, so that the data processing efficiency is improved, the data processing time is saved, and the technical problems of low efficiency and serious time consumption of the conventional data processing method are solved.
Optionally, the operation performing unit 83 is further configured to: when the specified data operation is a data updating operation, determining a data index key value of data to be updated; determining a data index range to which a data index key value of the data to be updated belongs in the plurality of data index ranges; constructing a first target B-tree node on the B-tree corresponding to the data index range to which the data index key value of the data to be updated belongs; and updating the data to be updated according to the index key value of the first target B-tree node and the storage position corresponding to the target B-tree node.
Optionally, the operation performing unit 83 is further configured to: when the specified data operation is a data reading operation, determining a data index key value of data to be read; determining a data index range to which a data index key value of the data to be read belongs in the plurality of data index ranges; determining a B tree corresponding to a data index range to which a data index key value of the data to be read belongs; and traversing and searching the B tree corresponding to the data index range to which the data index key value of the data to be read belongs to obtain the storage position of the data to be read in the database, and reading the data to be read according to the storage position.
Optionally, the apparatus is further configured to: determining a target data index range corresponding to a specified data operation before performing the specified data operation on data based on each data index range and the B-tree corresponding to each data index range; and controlling the B tree corresponding to the target data index range to be in a locked state.
Optionally, the operation execution unit is further configured to: setting a locking identifier for the B tree corresponding to the target data index range; and controlling the B tree corresponding to the target data index range to be in a locking state according to the locking identifier.
Example five:
the embodiment of the present invention further provides a data processing apparatus, which is mainly used for executing the data processing method provided in the second embodiment of the present invention, and the data processing apparatus provided in the embodiment of the present invention is specifically described below.
Fig. 9 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention, as shown in fig. 9, the data processing apparatus mainly includes: a second obtaining unit 91, a first determining unit 92, and a second determining unit 93, wherein:
a second obtaining unit 91, configured to obtain a data index key value of data to be read;
a first determining unit 92, configured to determine a first target data index range in a plurality of data index ranges according to the data index key values, where each data index range includes the data index key value of each data in the database; each data index range corresponds to a B tree;
a second determining unit 93, configured to determine, based on the first target data index range and the first target B-tree corresponding to the first target data index range, a storage location of the data to be read in the database, and read the data to be read according to the storage location.
Optionally, each data index range corresponds to one B-tree pointer; the second determining unit 93 is further configured to: determining the first target B-tree according to the B-tree pointers of the first target data index range; and determining the storage position of the data to be read in the database based on the first target B-tree.
Optionally, the apparatus is further configured to: before determining the storage position of the data to be read in the database based on the first target B-tree, setting a locking identifier for the first target B-tree; and controlling the first target B-tree to be in a locking state according to the locking identifier.
In the data processing device, when the data index key values of the data to be read are in different data index ranges, the data reading process can be executed in parallel, and the data reading efficiency is improved.
Example six:
the embodiment of the present invention further provides a data processing apparatus, where the data processing apparatus is mainly configured to execute the data processing method provided in the third embodiment of the present invention, and the data processing apparatus provided in the embodiment of the present invention is specifically described below.
Fig. 10 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention, as shown in fig. 10, the data processing apparatus mainly includes: a third obtaining unit 101, a third determining unit 102, a constructing unit 103 and an updating unit 104, wherein:
a third obtaining unit 101, configured to obtain a data index key of data to be updated;
a third determining unit 102, configured to determine, according to the data index key values, a second target data index range to which the data to be updated belongs in multiple data index ranges, where each data index range includes data index key values of each data in a database; each data index range corresponds to a B tree;
a constructing unit 103, configured to construct a second target B-tree node on the B-tree corresponding to the second target data index range;
and the updating unit 104 is configured to update the data to be updated according to the index key value of the second target B-tree node and the storage location corresponding to the second target B-tree node.
Optionally, the apparatus is further configured to: before constructing a second target B-tree node on the B-tree corresponding to the second target data index range, setting a locking identifier for the B-tree corresponding to the second target data index range; and controlling the B tree corresponding to the second target data index range to be in a locking state according to the locking identifier.
In the data processing device, when the data index key values of the data to be updated are in different data index ranges, the data updating process can be executed in parallel, and the data updating efficiency is improved.
Example seven:
referring to fig. 11, an embodiment of the present invention further provides an electronic device 100, including: the processor 110, the memory 111, the bus 112 and the communication interface 113, wherein the processor 110, the communication interface 113 and the memory 111 are connected through the bus 112; the processor 110 is adapted to execute executable modules, such as computer programs, stored in the memory 111.
The Memory 111 may include a high-speed Random Access Memory (RAM) and may also include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 113 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used.
The bus 112 may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 11, but that does not indicate only one bus or one type of bus.
The memory 111 is used for storing a program, the processor 110 executes the program after receiving an execution instruction, and the method executed by the apparatus defined by the flow process disclosed in any of the foregoing embodiments of the present invention may be applied to the processor 110, or implemented by the processor 110.
The processor 110 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 110. The Processor 110 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component. The various methods, steps and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 111, and the processor 110 reads the information in the memory 111 and completes the steps of the method in combination with the hardware thereof.
In addition, in the description of the embodiments of the present invention, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (15)
1. A data processing method, comprising:
acquiring a preset index range, wherein the preset index range is a preset range interval of index key values for indexing data in a database;
dividing the preset index range into a plurality of data index ranges, and constructing a corresponding B-tree for each data index range;
and executing specified data operation on the data based on each data index range and the B tree corresponding to each data index range.
2. The method of claim 1, wherein performing the specified data operation on the data based on each of the data index ranges and the B-tree corresponding to each of the data index ranges comprises:
when the specified data operation is a data updating operation, determining a data index key value of data to be updated;
determining a data index range to which a data index key value of the data to be updated belongs in the plurality of data index ranges;
constructing a first target B-tree node on the B-tree corresponding to the data index range to which the data index key value of the data to be updated belongs;
and updating the data to be updated according to the index key value of the first target B-tree node and the storage position corresponding to the first target B-tree node.
3. The method of claim 1, wherein performing the specified data operation on the data based on each of the data index ranges and the B-tree corresponding to each of the data index ranges further comprises:
when the specified data operation is a data reading operation, determining a data index key value of data to be read;
determining a data index range to which a data index key value of the data to be read belongs in the plurality of data index ranges;
determining a B tree corresponding to a data index range to which a data index key value of the data to be read belongs;
and traversing and searching the B tree corresponding to the data index range to which the data index key value of the data to be read belongs to obtain the storage position of the data to be read in the database, and reading the data to be read according to the storage position.
4. The method of any of claims 1-3, wherein prior to performing the specified data operation on the data based on each of the data index ranges and the B-tree to which each of the data index ranges corresponds, the method further comprises:
determining a target data index range corresponding to the specified data operation;
and controlling the B tree corresponding to the target data index range to be in a locked state.
5. The method of claim 4, wherein controlling the B-tree corresponding to the target data index range to be in a locked state comprises:
setting a locking identifier for the B tree corresponding to the target data index range;
and controlling the B tree corresponding to the target data index range to be in a locking state according to the locking identifier.
6. A data processing method, comprising:
acquiring a data index key value of data to be read;
determining a first target data index range in a plurality of data index ranges according to the data index key values, wherein each data index range contains the data index key value of each data in the database; each data index range corresponds to a B tree;
and determining the storage position of the data to be read in the database based on the first target data index range and the corresponding first target B-tree, and reading the data to be read according to the storage position.
7. The method of claim 6, wherein each data index range corresponds to a B-tree pointer;
determining the storage location of the data to be read in the database based on the first target data index range and the corresponding first target B-tree thereof comprises:
determining the first target B-tree according to the B-tree pointers of the first target data index range;
and determining the storage position of the data to be read in the database based on the first target B-tree.
8. The method of claim 7, wherein prior to said determining a storage location of said data to be read in a database based on said first target B-tree, said method further comprises:
setting a locking identifier for the first target B-tree;
and controlling the first target B-tree to be in a locking state according to the locking identifier.
9. A data processing method, comprising:
acquiring a data index key value of data to be updated;
determining a second target data index range to which the data to be updated belongs in a plurality of data index ranges according to the data index key values, wherein each data index range contains the data index key value of each data in the database; each data index range corresponds to a B tree;
constructing a second target B-tree node on the B-tree corresponding to the second target data index range;
and updating the data to be updated according to the index key value of the second target B-tree node and the storage position corresponding to the second target B-tree node.
10. The method of claim 9, wherein prior to constructing the second target B-tree node on the B-tree corresponding to the second target data index range, the method further comprises:
setting a locking identifier for the B-tree corresponding to the second target data index range;
and controlling the B tree corresponding to the second target data index range to be in a locking state according to the locking identifier.
11. A data processing apparatus, comprising:
the device comprises a first obtaining unit, a second obtaining unit and a third obtaining unit, wherein the first obtaining unit is used for obtaining a preset index range, and the preset index range is a preset range interval of index key values for indexing data in a database;
the dividing unit is used for dividing the preset index range into a plurality of data index ranges and constructing a corresponding B-tree for each data index range;
and the operation execution unit is used for executing specified data operation on the data based on each data index range and the B tree corresponding to each data index range.
12. A data processing apparatus, comprising:
the second acquisition unit is used for acquiring a data index key value of data to be read;
a first determining unit, configured to determine a first target data index range in multiple data index ranges according to the data index key values, where each data index range includes data index key values of respective data in a database; each data index range corresponds to a B tree;
and the second determining unit is used for determining the storage position of the data to be read in the database based on the first target data index range and the corresponding first target B-tree thereof, and reading the data to be read according to the storage position.
13. A data processing apparatus, comprising:
the third acquisition unit is used for acquiring a data index key value of the data to be updated;
a third determining unit, configured to determine, according to the data index key values, a second target data index range to which the data to be updated belongs in multiple data index ranges, where each data index range includes the data index key value of each data in the database; each data index range corresponds to a B tree;
the construction unit is used for constructing a second target B-tree node on the B-tree corresponding to the second target data index range;
and the updating unit is used for updating the data to be updated according to the index key value of the second target B-tree node and the storage position corresponding to the second target B-tree node.
14. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the data processing method of any of claims 1-10 when executing the computer program.
15. A computer-readable medium having non-volatile program code executable by a processor, characterized in that the program code causes the processor to perform the data processing method of any of the preceding claims 1-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010727974.5A CN111858606A (en) | 2020-07-24 | 2020-07-24 | Data processing method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010727974.5A CN111858606A (en) | 2020-07-24 | 2020-07-24 | Data processing method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111858606A true CN111858606A (en) | 2020-10-30 |
Family
ID=72950208
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010727974.5A Pending CN111858606A (en) | 2020-07-24 | 2020-07-24 | Data processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111858606A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114238704A (en) * | 2022-02-21 | 2022-03-25 | 北京金山云网络技术有限公司 | Tree index splitting method, data access method and device and electronic equipment |
| WO2024051271A1 (en) * | 2022-09-08 | 2024-03-14 | 华为技术有限公司 | Data processing method and device |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6374232B1 (en) * | 1996-08-29 | 2002-04-16 | Oracle Corp. | Method and mechanism for retrieving values from a database |
| US20110238667A1 (en) * | 2010-03-29 | 2011-09-29 | Sybase, Inc. | B-Tree Ordinal Approximation |
| CN103745008A (en) * | 2014-01-28 | 2014-04-23 | 河海大学 | Sorting method for big data indexing |
| CN104112011A (en) * | 2014-07-16 | 2014-10-22 | 深圳市国泰安信息技术有限公司 | Method and device for extracting mass data |
| CN106503196A (en) * | 2016-10-26 | 2017-03-15 | 云南大学 | The structure and querying method of extensible storage index structure in cloud environment |
| CN106657174A (en) * | 2015-10-28 | 2017-05-10 | 阿里巴巴集团控股有限公司 | Data synchronizing and updating methods and data synchronizing and updating devices |
| CN109117433A (en) * | 2017-06-23 | 2019-01-01 | 菜鸟智能物流控股有限公司 | Index tree object creation method and index method and related device thereof |
| CN110716933A (en) * | 2019-09-29 | 2020-01-21 | 浙江大学 | Novel urban rail train big data-oriented high-flexibility distributed index method |
| CN111125120A (en) * | 2019-12-30 | 2020-05-08 | 广州数锐智能科技有限公司 | Stream data-oriented fast indexing method, device, equipment and storage medium |
-
2020
- 2020-07-24 CN CN202010727974.5A patent/CN111858606A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6374232B1 (en) * | 1996-08-29 | 2002-04-16 | Oracle Corp. | Method and mechanism for retrieving values from a database |
| US20110238667A1 (en) * | 2010-03-29 | 2011-09-29 | Sybase, Inc. | B-Tree Ordinal Approximation |
| CN103745008A (en) * | 2014-01-28 | 2014-04-23 | 河海大学 | Sorting method for big data indexing |
| CN104112011A (en) * | 2014-07-16 | 2014-10-22 | 深圳市国泰安信息技术有限公司 | Method and device for extracting mass data |
| CN106657174A (en) * | 2015-10-28 | 2017-05-10 | 阿里巴巴集团控股有限公司 | Data synchronizing and updating methods and data synchronizing and updating devices |
| CN106503196A (en) * | 2016-10-26 | 2017-03-15 | 云南大学 | The structure and querying method of extensible storage index structure in cloud environment |
| CN109117433A (en) * | 2017-06-23 | 2019-01-01 | 菜鸟智能物流控股有限公司 | Index tree object creation method and index method and related device thereof |
| CN110716933A (en) * | 2019-09-29 | 2020-01-21 | 浙江大学 | Novel urban rail train big data-oriented high-flexibility distributed index method |
| CN111125120A (en) * | 2019-12-30 | 2020-05-08 | 广州数锐智能科技有限公司 | Stream data-oriented fast indexing method, device, equipment and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114238704A (en) * | 2022-02-21 | 2022-03-25 | 北京金山云网络技术有限公司 | Tree index splitting method, data access method and device and electronic equipment |
| WO2024051271A1 (en) * | 2022-09-08 | 2024-03-14 | 华为技术有限公司 | Data processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111552692B (en) | Plus-minus cuckoo filter | |
| WO2021072874A1 (en) | Dual array-based location query method and apparatus, computer device, and storage medium | |
| CN111858606A (en) | Data processing method and device and electronic equipment | |
| CN111078689B (en) | Data processing method and system of discontinuous pre-ordering traversal tree algorithm | |
| CN110727702B (en) | Data query method, device, terminal and computer readable storage medium | |
| KR20190079354A (en) | Partitioned space based spatial data object query processing apparatus and method, storage media storing the same | |
| CN111209252A (en) | File metadata storage method and device and electronic equipment | |
| CN116126864A (en) | Index construction method, data query method and related equipment | |
| CN113779286B (en) | Method and device for managing graph data | |
| CN104794130A (en) | Inter-table correlation query method and device | |
| CN114840487A (en) | Metadata management method and device for distributed file system | |
| CN112231400B (en) | Access method, device, equipment and storage medium of distributed database | |
| CN117056428A (en) | Automatic expansion method, device and server of partition table | |
| CN108376054B (en) | Processing method and device for indexing identification data | |
| CN116450607A (en) | Data processing method, device and storage medium | |
| CN113256301B (en) | Data shielding method, device, server and medium | |
| EP3736705A1 (en) | Date query method and device | |
| CN111858607B (en) | Data processing method, device, electronic equipment and computer readable medium | |
| CN114595215A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN108984615B (en) | Data query method and system and storage medium | |
| CN111651466A (en) | Data sampling method and device | |
| CN113127480A (en) | Data management method and device | |
| CN111858607A (en) | Data processing method and device, electronic equipment and computer readable medium | |
| CN110874246A (en) | Module loading method, system and equipment | |
| CN116108023B (en) | Data retrieval method and device for HBase database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |