CN111914125A

CN111914125A - Data processing method and device, storage medium and electronic equipment

Info

Publication number: CN111914125A
Application number: CN202010690105.XA
Authority: CN
Inventors: 梁喆; 李磊
Original assignee: Beijing Aixin Technology Co ltd
Current assignee: Beijing Aixin Technology Co ltd
Priority date: 2020-07-16
Filing date: 2020-07-16
Publication date: 2020-11-10

Abstract

The application relates to the technical field of data processing, and provides a data processing method and device, a storage medium and electronic equipment. The data processing method comprises the following steps: acquiring data to be processed and an initialized heap space, wherein each data storage unit in the heap space represents a node in a heap structure, and the type of the heap structure is a small top heap or a large top heap; comparing each data to be processed with the data stored in the heap top node, if the comparison result shows that the data to be processed meets the condition of adding the heap structure, storing the data to be processed into the heap top node, comparing the data with the data stored in other nodes of the heap structure, and determining the storage position of the data in the nodes of the heap structure according to the comparison result so that the heap structure still maintains the original type after storing the data to be processed in the nodes of the heap structure; data stored in a node of the heap structure is read. The data processing method solves the TopK problem by constructing a heap, and greatly reduces the time complexity of the TopK algorithm.

Description

Data processing method and device, storage medium and electronic equipment

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a data processing method and apparatus, a storage medium, and an electronic device.

Background

How to quickly find the maximum or minimum K data (TopK for short) from a large amount of data is one of the common problems in the field of data processing. There are many business scenarios involving the TopK problem: for example, in the face recognition task, K persons most similar to a certain image are quickly found from the face feature library, and for example, in the search task, K webpages (or audio, video, images, files) with the highest degree of matching with the label are quickly retrieved from the target library according to the search label.

In the prior art, several ordering algorithms are usually used to solve the TopK problem. However, although the conventional algorithms such as the insert ordering algorithm are simple in logic, the time complexity is high, and the TopK problem is difficult to solve in a short time, so that the corresponding business progress is affected.

Disclosure of Invention

An embodiment of the present invention provides a data processing method and apparatus, a storage medium, and an electronic device, so as to solve the above technical problems.

In order to achieve the above purpose, the present application provides the following technical solutions:

in a first aspect, an embodiment of the present application provides a data processing method, including: acquiring a plurality of data to be processed and an initialized heap space, wherein each data storage unit in the heap space represents a node in a heap structure, and the type of the heap structure is a small top heap or a large top heap; comparing each data to be processed with data stored in a heap top node, if the comparison result shows that the data to be processed meets the condition of adding the heap structure, storing the data to be processed into the heap top node, comparing the data to be processed with data stored in other nodes of the heap structure, and determining the storage position of the data in the nodes of the heap structure according to the comparison result so that the heap structure still maintains the original type after storing the data to be processed in the nodes of the heap structure; data stored in a node of the heap structure is read.

The data processing method solves the TopK problem by constructing the heap, and the time complexity of the TopK algorithm is greatly reduced because each data to be processed is added into the heap structure (including the added heap structure adjustment operation) and only logK comparison is required on average. Moreover, the method is different from a general heap sorting algorithm, a complete binary tree is not required to be constructed step by using the data to be processed, the storage space occupied by the heap structure is allocated and initialized in advance (which is equivalent to the complete binary tree being constructed), and the data to be processed is only required to be added into the heap structure, so that the data processing efficiency is improved.

In an implementation manner of the first aspect, the determining, when the type of the heap structure is a small top heap, the comparing each piece of to-be-processed data with data stored in a heap top node, if a comparison result indicates that the to-be-processed data meets a condition of adding the heap structure, storing the to-be-processed data in the heap top node, comparing the to-be-processed data with data stored in other nodes of the heap structure, and determining a storage location of the data in the node of the heap structure according to the comparison result includes: comparing each data to be processed with data stored in a heap top node, and if the data to be processed is larger than the data stored in the heap top node, storing the data to be processed into the heap top node; starting from the heap top node, if the current node has a child node, comparing the data stored in the current node with the data stored in the child node, if the node corresponding to the minimum data is the current node, terminating the comparison process, if the node corresponding to the minimum data is the child node of the current node, exchanging the data stored in the current node with the data stored in the child node, and taking the child node as a new current node to continue to compare until the current node does not have a child node or the comparison process is terminated.

In the implementation mode, when the data to be processed is added into the heap structure, the data stored in the nodes are compared layer by layer, so that the flow process can be performed through hardware, the data processing speed is further increased, the algorithm throughput rate is improved, and the equipment power consumption is reduced.

In an implementation manner of the first aspect, the number of the data to be processed is N, and before the obtaining of the plurality of data to be processed and the initializing of the heap space, the method further includes: distributing K data storage units in a storage medium as the heap space, and initializing the data storage units in the heap space to be the theoretical minimum value of the data to be processed; or distributing M data storage units in a storage medium as the heap space, initializing the first K data storage units in the heap space to the theoretical minimum value of the data to be processed, and initializing the last M-K data storage units in the heap space to the theoretical maximum value of the data to be processed; where K is the amount of data that needs to be obtained by data processing, M > K.

Two ways of allocating heap space are given in the above implementation. The way, which is simply summarized, is how much space is needed to allocate, and the advantage is that the storage space is saved. The second way is to allocate more space than needed, which has the advantage of simple logic and may be more efficient in allocating storage space.

In one implementation of the first aspect, M-2ⁿ-1, n-ceil (log2(K +1)), where ceil denotes the ceiling operation and n denotes the total number of levels of the heap structure.

The heap structure can be logically regarded as a complete binary tree, but not necessarily a full binary tree, and when allocating memory space, the allocation can be performed according to the memory space required by the minimum full binary tree containing the complete binary tree, and the allocation of the memory space in this way not only saves the memory space but also is relatively easy to allocate and manage because the number of data memory units representing each layer of nodes of the heap structure is a power of 2.

In an implementation manner of the first aspect, the heap space is a continuous storage space in the storage medium, and the data storage units representing each layer of nodes of the heap structure are sequentially arranged in the heap space.

In the above implementation, because the heap space addresses are continuous and the nodes of the heap structure are also continuously distributed, the address conversion of the parent node and the child node can be efficiently realized, and the data access can be efficiently performed, and because the addresses of the data storage units are continuously distributed, the allocation of the heap space from the storage medium is also very efficient.

In one implementation manner of the first aspect, the reading data stored in the node of the heap structure includes: sorting the data stored in the nodes of the heap structure according to a preset mode, and sequentially reading the sorted data; and if the type of the heap structure is a small top heap, sorting the data in an ascending order, and if the type of the heap structure is a large top heap, sorting the data in a descending order.

When reading data in the heap structure, the data may be read directly (for the TopK problem), or the data stored in the nodes of the heap structure may be sorted in a predetermined manner and the sorted data may be read sequentially (for the sorting problem).

In an implementation manner of the first aspect, the type of the heap structure is a small top heap, and sorting data stored in nodes of the heap structure according to a predetermined manner and sequentially reading the sorted data includes: reading data stored in a heap top node; starting from the heap top node, if a current node has a child node, determining smaller data from data stored in the child node of the current node, storing the smaller data in the current node, and taking the child node corresponding to the smaller data as a new current node to continue comparison until a comparison process is terminated when the child node does not exist in the current node, and storing a theoretical maximum value of the data to be processed in the current node; and continuously reading the data stored in the heap top node until all the data stored in the nodes of the heap structure are read.

In the implementation mode, when the data to be processed in the heap structure is sequenced and read, the data stored in the nodes are compared layer by layer, so that the flow process can be carried out through hardware, the data processing speed is further accelerated, the algorithm throughput rate is improved, and the operation power consumption is reduced.

In a second aspect, an embodiment of the present application provides a data processing apparatus, including a first operation module and a storage module, where the first operation module is connected to the storage module; a heap space is allocated in the storage module, each data storage unit in the heap space represents a node in a heap structure, and the type of the heap structure is small top heap or large top heap; the first operation module is used for comparing each piece of data to be processed with the data stored in the heap top node, if the comparison result shows that the data to be processed meets the condition of adding the heap structure, the data to be processed is stored in the heap top node, the data to be processed is compared with the data stored in other nodes of the heap structure, and the storage position of the data in the nodes of the heap structure is determined according to the comparison result, so that the heap structure still maintains the original type after the data to be processed is stored in the nodes of the heap structure.

In an implementation manner of the second aspect, the first operation module includes a plurality of first operation sub-modules, the plurality of first operation sub-modules are connected in sequence, the storage module includes a plurality of storage sub-modules, a storage unit in each storage sub-module represents a layer of node in the heap structure, the type of the heap structure is a small top heap, and each first operation sub-module can access data stored in one storage sub-module corresponding to the first operation sub-module; a first one of the plurality of first operation sub-modules is configured to: comparing each data to be processed with data stored in a heap top node, storing the data to be processed into the heap top node when the data to be processed is larger than the data stored in the heap top node, and outputting the data stored in the heap top node and the address of the heap top node to a second first operation submodule; the first operation submodule is further configured to: when data stored in a second layer node returned by the second first operation submodule is received, storing the data stored in the second layer node into the heap top node; the other first operation sub-modules except the first one of the plurality of first operation sub-modules are used for: calculating the address of the child node of the previous layer node according to the address of the previous layer node output by the previous first operation submodule, reading the data stored in the child node of the previous layer node from the storage submodule corresponding to the first operation submodule according to the address of the child node of the previous layer node, comparing the data stored in the previous layer node output by the previous first operation submodule with the data stored in the child node thereof, terminating the output of the data to the next first operation submodule when the node corresponding to the minimum data is the previous layer node, returning the data stored in the child node and the address of the child node to the previous first operation submodule when the node corresponding to the minimum data is the child node of the previous layer node, and storing the data stored in the previous layer node into the child node, and the first operation submodule is used for outputting the data stored in the child node and the address of the child node to the next first operation submodule when the first operation submodule exists in the next first operation submodule; the first operation submodule other than the first operation submodule is further configured to: when the first operation submodule exists in a next first operation submodule and receives data stored in a next-layer node returned by the next first operation submodule and the address of the next-layer node, calculating the address of a father node of the next-layer node according to the address of the next-layer node, and storing the data stored in the next-layer node into the father node of the next-layer node according to the address of the father node of the next-layer node.

In the implementation manner, by cascading a plurality of first operation sub-modules, and in the process of adding the data to be processed to the heap structure, each first operation sub-module is only responsible for the access and operation of one layer of nodes of the heap structure, the flow process of solving the TopK problem is realized, that is, the data processing speed is accelerated by hardware.

In one implementation manner of the second aspect, the apparatus further includes a second operation module, and the second operation module is connected to the storage module; the second operation module is used for sequencing the data stored in the nodes of the heap structure according to a preset mode and sequentially reading the sequenced data; and if the type of the heap structure is a small top heap, sorting the data in an ascending order, and if the type of the heap structure is a large top heap, sorting the data in a descending order.

In an implementation manner of the second aspect, the second operation module includes a plurality of second operation sub-modules, the plurality of second operation sub-modules are connected in sequence, the storage module includes a plurality of storage sub-modules, a storage unit in each storage sub-module represents a layer of node in the heap structure, the type of the heap structure is a small top heap, and each second operation sub-module can access data stored in one storage sub-module corresponding to the second operation sub-module; a first one of the plurality of second operation sub-modules is configured to: reading data stored in the heap top node, and outputting the address of the heap top node to a second operation submodule; the first second operation submodule is further configured to: storing the data stored in the second layer node returned by the second operation submodule into the heap top node; the second operation sub-modules except the first one of the plurality of second operation sub-modules are used for: calculating the address of the child node of the previous layer node according to the address of the previous layer node output by the previous second operation submodule, and reads the data stored in the child node of the previous layer node from the storage submodule corresponding to the second operation submodule according to the address of the child node of the previous layer node, and is used for determining smaller data from the data stored in the child node of the previous layer node and returning the smaller data and the address of the child node storing the smaller data to the previous second operation submodule, and is used for outputting the address of the child node storing the smaller data to the next second arithmetic submodule when the second arithmetic submodule exists in the next second arithmetic submodule, and when the child node storing the small data is a leaf node of the heap structure, storing the theoretical maximum value of the data to be processed into the child node storing the small data; the second operation submodule except the first operation submodule is also used for: when the second operation submodule exists in a next second operation submodule, the address of the father node of the next layer node is calculated according to the address of the next layer node returned by the next second operation submodule, and the data stored in the next layer node is stored in the father node of the next layer node according to the address of the father node of the next layer node.

In the implementation manner, by cascading a plurality of second operation sub-modules, and in the process of sequencing and reading the stored to-be-processed data in the heap structure, each second operation sub-module is only responsible for the access and operation of one layer of nodes of the heap structure, the flow process of solving the sequencing problem is realized, that is, the data processing speed is accelerated through hardware.

In a third aspect, an embodiment of the present application provides a computer-readable storage medium, where computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are read and executed by a processor, the computer program instructions perform the method provided by the first aspect or any one of the possible implementation manners of the first aspect.

In a fourth aspect, an embodiment of the present application provides an electronic device, including: a memory in which computer program instructions are stored, and a processor, where the computer program instructions are read and executed by the processor to perform the method provided by the first aspect or any one of the possible implementation manners of the first aspect.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.

Fig. 1 is a flowchart illustrating a data processing method provided in an embodiment of the present application;

FIG. 2 shows a schematic diagram of an initialized stack structure;

FIGS. 3(A) through 3(D) are schematic diagrams illustrating the process of adding data to a heap structure;

FIGS. 4(A) through 4(D) are schematic diagrams illustrating a process of reading data from a heap structure;

fig. 5 is a block diagram illustrating a data processing apparatus according to an embodiment of the present application;

fig. 6 shows a block diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.

The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element. The terms "first," "second," "third," and the like are used solely to distinguish one from another and are not to be construed as indicating or implying relative importance.

Fig. 1 shows a flowchart of a data processing method provided in an embodiment of the present application. The method may be implemented in software or hardware, and if implemented in software, the method steps may be executed by the electronic device shown in fig. 6, and the specific structure of the electronic device will be described later, and if implemented in hardware, the method steps may be executed by the data processing apparatus shown in fig. 5, and the specific structure of the data processing apparatus will be described later. In introducing fig. 1, the principle of the data processing method is mainly explained, without limiting in which way the method is implemented. Referring to fig. 1, the method includes:

step S110: and acquiring a plurality of data to be processed and initialized heap space.

The TopK problem can be described simply as: and finding the maximum or minimum K data from the N data, and sorting and outputting the K data after finding the K data according to the requirement. In the TopK problem, N is generally larger than K, since it is not necessary to select from N data when N ═ K. However, when K data are sorted and output for the case where N is K, the TopK problem becomes a pure data sorting problem. For the above situations, the application provides corresponding solutions.

The above-mentioned N data are to-be-processed data, and the application does not limit the type of the to-be-processed data, for example, the to-be-processed data may be a numerical value, a vector, or the like, but the to-be-processed data should be measurable, so that the size comparison between the to-be-processed data may be performed.

There are many business scenarios involving the TopK problem: for example, in the face recognition task, K persons most similar to the target image are quickly found from the face feature library, and the data to be processed may be the similarity between the features of the target image and the features in the face feature library. For another example, in the search task, the K webpages (or audio, video, image, file) that match the tags to the highest degree are quickly retrieved from the target library according to the search tags (which can be expressed as a vector), and the like, and the data to be processed at this time may be the matching degree of the search tags and the webpages (or audio, video, image, file).

The heap space is a space allocated from a storage medium (e.g., a computer memory, a register, etc.) for storing the heap structure, and the obtaining of the heap space in step S110 may refer to a storage location (including a start position, a length, etc.) of the obtained heap space in the storage medium. Note that the allocation here includes not only the case of dynamic allocation, such as allocation by a program, but also the case of static allocation, such as setting a hardware memory module exclusively as heap space. The stack structure is first described below:

the heap structure is a data structure, and the heap structure is composed of a plurality of nodes, each node can store one data, and the heap structure can be logically regarded as a complete binary tree, that is, the nodes have parent-child relationships. For example, the heap structure in fig. 2 includes 7 nodes A, B, C, D, E, F, G, which are divided into 3 levels, the parent-child relationships between the nodes are shown by arrows, and the number of nodes in each level increases by a power of 2 from the top node a of the heap, that is, 1, 2, and 4 nodes are included respectively. The heap structure in fig. 2 can be logically viewed as a 3-level full binary tree, and it can be understood that if the heap structure is not a full binary tree, some nodes are missing on the right side of the lowest level, and the number of nodes on the last level is not a power of 2.

The pile structure has two types, namely a small top pile and a large top pile, wherein the small top pile is as follows: the data stored in each node of the heap structure is not larger than the data stored in the child nodes (if the child nodes exist), and a subtree taking any one node of the small top heap as a root node is also the small top heap; the large top stack means that: the data stored in each node of the heap structure is no smaller than the data stored in its children nodes (if any), and the subtree with any one node of the large top heap as the root node is also the small top heap. The scheme of this application both can be suitable for little top heap and also can be suitable for big top heap, if will solve maxK problem (select biggest K data from N data promptly), then can adopt little top heap, if will solve the minK problem (select minimum K data from N data promptly), then can adopt big top heap. Hereinafter, for simplicity, a case where the type of the heap structure is a small top heap is mainly exemplified.

Each data storage unit in the heap space represents a node in the heap structure, i.e. the data stored in a certain node of the heap structure is actually stored in the data storage unit representing the node, and the size of each data storage unit should be at least the same as the bit width of the data (although it may be larger), for example, for an unsigned number of 8 bits, the size of each data storage unit should be at least 1 byte, and for a floating point number of 64 bits, the size of each data storage unit should be at least 4 bytes.

In some implementations, the heap space is a contiguous storage space in the storage medium, and the data storage units representing each level node of the heap structure are arranged sequentially in the heap space. For example, the 1 st data storage unit of the heap space represents a level 1 node (i.e., the heap top node) of the heap structure, the 2 nd, 3 rd data storage units represent a level 2 node of the heap structure, the 4 th, 5 th, 6 th, 7 th data storage units represent a level 3 node of the heap structure, and so on. The advantages of this implementation are that the address conversion of the parent and child nodes can be realized efficiently, so as to perform data access efficiently, and the allocation of heap space from the storage medium is also very efficient due to the continuous distribution of addresses of the data storage units. For example, if the address of a node is Addr (the address can be understood as the serial number of a data storage unit), the addresses of the left and right child nodes are Addr × 2 and Addr × 2+1, respectively, and the address of the parent node is Addr/2 (only taking an integer part). It will be appreciated that in other implementations, the data storage units in the heap space may also be non-contiguous in address, e.g., each data storage unit is individually subject to a space application.

The size of the heap space to be applied and the initialization method of the heap space are related to K in the TopK problem. Two implementations are listed below:

the first method is as follows: k data storage units are allocated from the storage medium as heap space, and the data storage units in the heap space are initialized to a theoretical minimum (for small top heaps) or a theoretical maximum (for large top heaps) of the data to be processed. For example, if the data to be processed are 8-bit unsigned numbers, the theoretical minimum value is 0x00, and the theoretical maximum value is 0 xFF. As will be described in the method below, the largest or smallest K data to be obtained in the TopK problem is stored in heap space.

The second method comprises the following steps: allocating M data storage units from the storage medium as heap space, initializing the first K data storage units in the heap space to a theoretical minimum (for small top heaps) or a theoretical maximum (for large top heaps) of the data to be processed, and initializing the last M-K data storage units in the heap space to a theoretical maximum (for large top heaps) or a theoretical minimum (for small top heaps) of the data to be processed, wherein M > K. As will be described in the following method, the first K data locations in the heap space store the largest or smallest K data that is ultimately to be obtained in the TopK problem.

The way, which is simply summarized, is how much space is needed to allocate, and the advantage is that the storage space is saved. The second approach is to allocate more space than is needed: for example, always allocating a fixed memory space has the advantage of simple logic, and the efficiency of allocating memory space may be higher; for another example, it is mentioned before that the heap structure may be logically regarded as a complete binary tree, but not necessarily a full binary tree, and when allocating the storage space, the allocation may be performed according to the storage space required by the minimum full binary tree containing the complete binary tree, for example, the number of layers n-ceil (log2(K +1)) (ceil represents an rounding operation) of the heap structure is calculated first, and since the number of nodes that can be contained at most in each layer is known, M-2 may be calculatedⁿ1, performing the memory allocation in this way not only saves memory but also is relatively easy to manage since the number of data storage units representing each level node of the heap structure is a power of 2.

In the second mode, the last M-K data storage units in the heap space adopt an initialization mode different from the initialization mode of the first K data storage units. Although these data storage units are not directly used for storing the data to be processed, when the storage locations of the data in the stack structure are adjusted (see step S120 in detail), the data stored in these data storage units can still participate in the data comparison process, which will be described later.

For example, the data to be processed are unsigned numbers of 8 bits, K is 7, and the heap space is initialized in the first mode, so that the initialization result is as shown in fig. 2, the data stored in each node of the heap structure is initialized to 0x00, and since 0x00 is the theoretical minimum value of the unsigned numbers, it can be ensured that the data to be processed can be normally added to the heap structure in step S120.

Step S120: and comparing each data to be processed with the data stored in the heap top node, if the comparison result shows that the data to be processed meets the condition of adding the heap structure, storing the data to be processed into the heap top node, comparing the data to be processed with the data stored in other nodes of the heap structure, and determining the storage position of the data in the nodes of the heap structure according to the comparison result so that the heap structure still maintains the original type after storing the data to be processed in the nodes of the heap structure.

According to the expectation of the method, the K maximum or minimum data to be finally obtained in the TopK problem are stored in the heap space, and the process is implemented step by step, and whether a piece of data to be processed meets the condition of adding the heap structure is examined in each step, and if the condition is met, the data is added into the heap structure and the storage position of the data in the nodes of the heap structure is adjusted accordingly, so that the type of the heap structure is kept unchanged, for example, for solving the maxK problem, the adopted heap structure always maintains a small top heap, and for solving the minK problem, the adopted heap structure always maintains a large top heap. As to how the storage location of data in the nodes of the heap structure is adjusted, it will be explained later.

Taking the case that the type of the heap structure is the small top heap as an example, according to the above method, assuming that the current step i is executed (i can take any value from 1 to N), the small top heap stores the largest K data of the i-1 data to be processed that have been examined (of course, if i-1< K, some nodes in the small top heap also maintain the initial values). For the small top heap, the condition that the ith to-be-processed data is added into the heap structure is that the data is larger than the data stored in the heap top node, because the data stored in the heap top node of the small top heap is the minimum of the data stored in the current heap structure, if the data stored in the heap top node is smaller than the ith to-be-processed data, it indicates that the ith to-be-processed data should be added into the current heap structure, and the data stored in the heap top node should be removed from the current heap structure, so that the ith to-be-processed data can be stored into the heap top node, and the data originally stored in the heap top node is replaced. However, after the ith to-be-processed data is added to the heap structure, the small top heap condition may not be satisfied any more, so that the to-be-processed data needs to be compared with data stored in other nodes of the heap structure, and the storage location of the data in the node of the heap structure is adjusted according to the comparison result (of course, adjustment may not be needed), so that the heap structure satisfies the small top heap condition again. For a large top heap, the condition for adding certain data to be processed into the heap structure is that the data is smaller than the data stored in the heap top node, and the principle can be analyzed similarly.

Step S120 is further described below using a small top stack as an example:

firstly, comparing the data to be processed with the data stored in the heap top node, if the data to be processed is larger than the data stored in the heap top node, storing the data to be processed into the heap top node, and at this time, because the data originally stored in the heap top node is covered, the heap structure is removed.

Then, the heap top node is used as the current node to start adjusting the storage location of the data in the node of the heap structure, because the current heap structure may not meet the requirement of the small top heap (and may not need to be adjusted) with the addition of the data to be processed, which is as follows:

and taking the heap top node as a current node, and if the current node has a child node, comparing the data stored in the current node with the data stored in the child node to determine the minimum data in the data.

If the node corresponding to the minimum data is the current node, it indicates that the sub-tree using the current node as the root node is already the small top heap, or the current node is the suitable storage location of the data to be processed, and no further adjustment is needed, so that the comparison process can be terminated (i.e. comparing the data stored in the current node with the data stored in the child nodes thereof).

If the node corresponding to the minimum data is a child node of the current node, it indicates that the sub-tree using the current node as the root node is not the small top heap, or the current node is not a suitable storage location for the data to be processed, and a certain location in the sub-tree using the child node as the root node is a suitable storage location for the data to be processed, so that the data stored in the current node and the data stored in the child node can be exchanged, and the child node is used as a new current node to continue the above comparison process until the child node does not exist in the current node (i.e., the current node is a leaf node), or the condition that the comparison process is terminated is satisfied although the child node exists in the current node.

The above steps are explained with reference to fig. 3(a) to 3(D), assuming that at a certain time, the state in the stack structure is as shown in fig. 3 (a). The current data to be processed is 0x06, which is compared with the data 0x00 stored in node a, and since 0x06 > 0x00, 0x06 is stored in node a, as shown in fig. 3 (B). Then, the data stored in the node a is compared with the data stored in the child node B, C thereof to obtain the minimum value of 0x01, which corresponds to the node C, and then the data stored in the node a and the node C are exchanged, and the result is shown in fig. 3 (C). Then, the data stored in the node C is compared with the data stored in the child node F, G thereof to obtain the minimum value of 0x02, which corresponds to the node G, and then the data stored in the node C and the node G are exchanged, with the result as shown in fig. 3 (D). Since node G is already a leaf node, the comparison is not continued, and node G is a suitable location to store 0x 06. Meanwhile, through the above series of adjustments, the stack structure in fig. 3(D) is still a small top stack.

As mentioned previously, M (M > K) data storage units may be allocated when allocating heap space, and if M is large enough, it may be guaranteed that the lowest level nodes of the heap structure are all nodes initialized to the theoretical maximum value of the data to be processed. For example, if M is equal to 20, in the examples of fig. 3(a) to 3(D), the D, E, F, G nodes each include two child nodes initialized to 0xFF, and since the data stored in the D, E, F, G node may not be greater than 0xFF, the comparison process is inevitably terminated when the comparison process reaches the D, E, F, G node, and at this time, the condition of whether the current node has a child node (because there is an inevitable child node) may not be considered, which is beneficial to simplifying the determination logic in the data location adjustment process in the stack structure, and also does not affect the result of the data location adjustment in the stack structure.

Step S130: data stored in a node of the heap structure is read.

According to the explanation in step S120, during data processing, the largest K data (for small top heaps) or the smallest K data (for large top heaps) of the processed data are always stored in the heap structure. Therefore, after all the data to be processed are processed, the data stored in the nodes of the heap structure are read, and the maximum K data (for small top heaps) or the minimum K data (for large top heaps) in all the data to be processed are obtained, namely the TopK problem is solved.

For example, in the face recognition task, the similarity between the feature of the target image and each feature in the face feature library is processed, so that the maximum K similarities can be obtained, and then K persons corresponding to the K similarities can be found according to the records in the face feature library.

For another example, in the search task, after the matching degree between the search tag and each web page (or audio, video, image, file) in the target library is processed, the maximum K matching degrees can be obtained, and then the web page corresponding to the K matching degrees in the target library can be presented to the user as the search result.

The data processing method solves the TopK problem by constructing the heap, and the time complexity of the TopK algorithm is greatly reduced because each data to be processed is added into the heap structure (including the added heap structure adjustment operation) and only logK comparison is required on average. In addition, although the above method uses the heap structure, it is different from the general heap sorting algorithm, it is not necessary to gradually construct a complete binary tree by using the data to be processed, the storage space occupied by the heap structure is allocated and initialized in advance (which is equivalent to that the complete binary tree has been constructed), and it is only necessary to add the data to be processed into the heap structure, thereby being beneficial to improving the data processing efficiency.

In addition, in the method for adding the data to be processed to the heap structure in step S120, the data stored in the nodes are compared layer by layer, so that the hardware can perform pipeline operations, thereby further increasing the data processing speed, improving the algorithm throughput rate, and reducing the power consumption of the device. As described in detail below with respect to fig. 5.

Further, according to requirements, the data read in step S130 may be directly read, or the data stored in the nodes of the heap structure may be sequentially read after being sorted according to a predetermined manner. For the latter, if the type of heap structure is small top heap, the data may be sorted in ascending order, and if the type of heap structure is large top heap, the data may be sorted in descending order. Since the small top heap or the large top heap is already constructed in step S120, the existing heap sorting algorithm may be used to sort the data here, although the present application also provides a new sorting method below.

Note that the sorting is performed for K pieces of data to be processed stored in the nodes of the heap structure, and if N is equal to K, the sorting is performed for the whole data to be processed, and the heap structure is used for sorting instead of solving the TopK problem.

Taking a small top heap as an example, the sorting process in step S130 can be implemented as follows:

step (1): and reading the data stored in the top node of the heap as the first data to be output, namely outputting the smallest data in the heap structure firstly.

Step (2): taking the heap top node as a current node, supplementing the data to be read next in the heap structure into the heap top node, and correspondingly adjusting the position of the stored data in the node of the heap structure, wherein the method specifically comprises the following steps:

taking the heap top node as the current node, if the current node has a child node, comparing the data stored in the child node of the current node (if only one child node exists, the comparison is not carried out), determining the smaller data in the data, and storing the smaller data in the current node. Obviously, the smaller data is the smallest data in the heap structure other than the data read in step (1), according to the characteristics of the small top heap.

And then, taking the child node corresponding to the smaller data as a new current node, continuing the comparison process until the comparison process is terminated when the child node does not exist in the current node (namely the current node is a leaf node), and storing the theoretical maximum value of the data to be processed into the current node. The reason for filling data in this way is that, since the data originally stored in the current node is already stored in its parent node, the data stored in the current node must be invalidated, and it is avoided that a conclusion that the data stored in the current node is small is obtained again when the data stored in the current node and the data stored in its siblings are compared in the following. It should be noted that during the sorting process, the nodes filled with the theoretical maximum are considered to be removed from the heap structure, so that if the child nodes of a node are all filled with the theoretical maximum of the data to be processed, the node should be considered to be a leaf node.

After step (2) is performed, the stack structure still maintains a small top stack.

And (3): and (3) repeating the steps (1) and (2) until all the data stored in the nodes of the heap structure are read. Because the minimum data in the current heap structure is read every time the step (1) is executed, the data stored in the nodes of the heap structure are read out in an ascending order, which is equivalent to the ascending order sorting of the K data to be processed. If K data to be processed are expected to be arranged in descending order, the order of the K data to be processed is just required to be reversed after the data are read out.

The above steps are explained with reference to fig. 4(a) to 4(D), and it is assumed that the state in the stack structure after the execution of step S120 is completed is as shown in fig. 4 (a). First, the data stored in the node a is read, as shown in fig. 4(B), it is noted that, in practice, after the data stored in the node a is read, the data may still remain in the node a and is only in a state of being capable of being overwritten, so that fig. 4(B) does not show the data at the node a only for easy understanding, and is not necessarily consistent with the actual situation. Then, the data stored in the child node B, C of the node a is compared to obtain the smaller value of 0x02, and 0x02 is stored in the node a corresponding to the node C, and the result is shown in fig. 4 (C). Then, the data stored in the child node F, G of the node C is compared to obtain a smaller value 0x05, and corresponding to the node F, 0x05 is stored in the node C, and 0xFF is stored in the node F, with the result shown in fig. 4 (D). It can be seen that through the above series of adjustments, the heap structure in fig. 4(D) is still a small top heap (certainly not very strict, since node F is no longer a complete binary tree after it is considered to be removed from the heap structure), and that at the top of the heap is the data to be read next.

In the sorting method, since each piece of data to be processed (including the read heap structure adjustment operation) is read from the heap structure, only logK comparison is required to be performed on average, so that the time complexity of the sorting algorithm is greatly reduced. In addition, although the above method uses a heap structure, unlike a general heap sorting algorithm, it is not necessary to gradually construct a complete binary tree using data to be processed.

In addition, in the sorting method provided in step S130, the comparison of the data stored in the nodes is performed layer by layer, so that the pipeline operation can be performed by hardware, the data processing speed is further increased, the algorithm throughput is improved, and the power consumption of the device is reduced. As described in detail below with respect to fig. 5.

For example, in the face recognition task, after the similarity between the feature of the target image and each feature in the face feature library is processed, the maximum K similarities can be obtained, and then K persons corresponding to the K similarities can be found according to the records in the face feature library, and the K similarities can be sorted in a descending order, and the corresponding K persons can be sequentially output as a recommended recognition result according to the sorting result (of course, only the person with the first similarity can be output as the recognition result).

For another example, in the search task, after the matching degree between the search tag and each webpage (or audio, video, image, file) in the target library is processed, the largest K matching degrees can be obtained, further, the K webpages corresponding to the K matching degrees in the target library can be determined, the K matching degrees can be sorted in a descending order, and the corresponding K webpages can be sequentially presented to the user as the search result according to the sorting order.

Fig. 5 is a block diagram of a data processing apparatus according to an embodiment of the present application, where the aforementioned data processing method can be implemented by hardware of the data processing apparatus, so that the hardware of the data processing process can be accelerated.

Referring to fig. 5, the apparatus includes a first operation module and a storage module, and the first operation module is connected to the storage module. In fig. 5, the first operation module is denoted as HC, and the storage module is denoted as RAM. The first operation module may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Micro Control Unit (MCU), a Network Processor (NP), or other conventional processors; the Application Specific Processor may also be a special purpose Processor, including a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, and discrete hardware components. The storage module may be a Random Access Memory (RAM) or a Register (Register).

The storage module is used for data storage and access, a heap space is allocated in the storage module (or allocation is not needed, and a data storage unit in the storage module is a heap space), each data storage unit in the heap space represents one node in a heap structure, and the type of the heap structure is a small top heap or a large top heap. Before the apparatus is used to solve the TopK problem, the heap space should be initialized, the initialization method being described above.

The first operation module is used for comparing each piece of data to be processed with the data stored in the heap top node, if the comparison result shows that the data to be processed meets the condition of adding the heap structure, the data to be processed is stored in the heap top node, the data to be processed is compared with the data stored in other nodes of the heap structure, and the storage position of the data in the nodes of the heap structure is determined according to the comparison result, so that the heap structure still maintains the original type after the data to be processed is stored in the nodes of the heap structure. After the first operation module has processed all the data to be processed, the data stored in the node of the stack structure may be read out (the first operation module may be responsible for reading or may be responsible for reading by other components), and the result is the largest or smallest K data to be processed among the N data to be processed, that is, the solution of the TopK problem is obtained.

In some implementations, the first operation module may further include a plurality of first operation sub-modules sequentially connected to each other, the storage module may further include a plurality of storage sub-modules, a storage unit in each storage sub-module represents a layer of node in the heap structure, and each first operation sub-module can access data stored in one storage sub-module corresponding to the first operation sub-module.

Referring to fig. 5, the first operation module further includes 3 first operation sub-modules, which are sequentially connected to the first operation sub-module and are respectively denoted as HC1, HC2, and HC3, and the memory module further includes 3 memory sub-modules, which are respectively denoted as RAM1, RAM2, and RAM 3. RAM1 corresponds to level 1 of the heap structure of fig. 2, and includes 1 data storage unit or 1 node in level 1 of the heap structure; RAM2 corresponds to level 2 of the heap structure of FIG. 2, and contains 2 data storage units or 2 nodes in level 2 of the heap structure; RAM3 corresponds to level 3 of the heap structure of fig. 2, and includes 4 data storage units or 4 nodes in level 3 of the heap structure. HC1, HC2, and HC3 are responsible for operations on nodes in layer 1, layer 2, and layer 3 of the stack structure in the process of solving the TopK problem, respectively. Each storage submodule can be set to be isolated (but not mandatory), HC1 can only access RAM1, HC2 can only access RAM2, HC3 can only access RAM3, so for example, if HC2 accesses data in RAM1, the data can only be transferred to HC2 by HC1, and HC2 cannot directly access the data.

The use of the first arithmetic sub-module is described below using a small top heap as an example:

the plurality of first operation sub-modules can be roughly divided into two types: the first operation sub-module is similar to the first operation sub-module, and the other first operation sub-modules are similar to the first operation sub-module, which have certain differences in functions and are described separately. It should be understood that, for the sake of uniform design and manufacture, the two types of first operation sub-modules may also adopt the same implementation in hardware.

The first operational submodule has the functions of:

(1) and comparing the data to be processed with the data stored in the heap top node, and storing the data to be processed into the heap top node when the data to be processed is larger than the data stored in the heap top node.

(2) And outputting the data stored in the heap top node and the address of the heap top node to a second first operation submodule.

(3) And when the data stored in the second layer node returned by the second first operation submodule is received, storing the data stored in the second layer node into the heap top node.

The function of any other first operation submodule is as follows:

(1) and reading the data stored in the child node of the previous layer node from the storage sub-module corresponding to the first operation sub-module according to the address of the child node of the previous layer node output by the previous first operation sub-module. The sub-node of the previous node is located in the current layer (referring to the layer in the heap structure corresponding to the first operation sub-module), and according to the foregoing, if the address of the previous node is Addr, the addresses of the left and right sub-nodes of the previous node are Addr × 2 and Addr × 2+1, respectively, and according to the two addresses, the first operation sub-module can obtain the data stored in the left and right sub-nodes from the corresponding storage sub-module.

(2) Comparing the data stored in the previous layer node output by the previous first operation sub-module with the data stored in the child nodes of the previous layer node, and when the node corresponding to the minimum data is the previous layer node, stopping outputting the data to the next first operation sub-module, and not returning the data to the previous operation sub-module; when the node corresponding to the minimum data is a child node of the previous layer of nodes, returning the data stored in the child node and the address of the child node to the previous first operation submodule, and storing the data stored in the previous layer of nodes into the child node; and outputting the data stored in the child node and the address of the child node to the next first operation submodule when the first operation submodule exists in the next first operation submodule.

(3) When the first operation submodule exists in a next first operation submodule and receives data stored in a next layer node returned by the next first operation submodule and an address of the next layer node, calculating an address of a father node of the next layer node according to the address of the next layer node, and storing the data stored in the next layer node into the father node of the next layer node according to the address of the father node of the next layer node. According to the foregoing, if the address of the next-layer node is Addr, the address of the next-layer node parent node is Addr/2 (only taking an integer part), and the first operation sub-module can store the data received from the next first operation sub-module into the corresponding storage sub-module according to the address.

The function of the first arithmetic submodule will be described below by taking a specific example, and assuming that the state in the stack structure at a certain time is as shown in fig. 3(a), the data processing apparatus takes the form of fig. 5.

1 st clock cycle: the HC1 compares 0x06 with data 0x00 stored in node a, stores 0x06 in node a because 0x06 > 0x00, and outputs the data 0x06 stored in node a and the address of node a to the HC2 (in fig. 5, the HC1 outputs data to the HC2 as indicated by push 1).

2 nd clock cycle: the HC2 calculates the address of its child node B, C from the address of the node a output from the HC1, and reads data 0x03 and 0x01 stored in the node B, C from the RAM 2; the HC2 compares the data 0x06 stored in the node a output from the HC1 with the data 0x03 and 0x01 stored in the node B, C, determines the smallest data 0x01 among them, and the corresponding node C; the HC2 returns the data 0x01 stored in the node C and the address of the node C to the HC1 (in fig. 5, pop2 indicates that the HC2 returns data to the HC 1); the HC2 stores the data 0x06 stored in the node a output from the HC1 in the node C, and outputs the data 0x06 stored in the node C and the address of the node C to the HC3 (in fig. 5, the HC2 outputs the data to the HC3 as indicated by a push 2).

3 rd clock cycle: the HC3 calculates the address of its child node F, G from the address of the node C output from the HC2, and reads data 0x05 and 0x02 stored in the node F, G from the RAM 3; the HC3 compares the data 0x06 stored in the node C output by the HC2 with the data 0x05 and 0x02 stored in the node F, G, determines the smallest data 0x02 among them, and the corresponding node G; the HC3 returns the data 0x02 stored in the node G and the address of the node G to the HC2 (in fig. 5, pop3 indicates that the HC3 returns data to the HC 2); the HC3 stores the data 0x06 stored in the node C output by the HC2 into the node G, and the HC3 is the last first operation submodule and is not output backward any more.

The HC1 stores data 0x01 stored in the node C returned by the HC2 (since there is only one data storage unit in the RAM1, the HC1 may be available and unavailable for the address of the node C returned by the HC2, or the HC2 may not return the address of the node C to the HC1, depending on the implementation) into the node a; the HC1 compares another data to be processed 0x09 with the data stored in the node a, 0x01, stores 0x09 into the node a since 0x09 > 0x01, and outputs the data stored in the node a, 0x09, and the address of the node a to the HC 2.

4 th clock cycle: the HC2 calculates the address of its parent node C from the address of the node G returned by the HC3, and stores the data 0x02 stored in the node G returned by the HC3 into the node C. The HC2 calculates the address of its child node B, C from the address of the node a output from the HC1, and reads data 0x03 and 0x02 stored in the node B, C from the RAM 2; the HC2 compares the data 0x09 stored in the node a output from the HC1 with the data 0x03 and 0x02 stored in the node B, C, determines the smallest data 0x02 among them, and the corresponding node C; the HC2 returns the data 0x02 stored in the node C and the address of the node C to the HC 1; the HC2 stores the data 0x09 stored in the node a output from the HC1 into the node C, and outputs the data 0x09 stored in the node C and the address of the node C to the HC 3.

Clock cycle 5: the HC3 calculates the address of its child node F, G from the address of the node C output from the HC2, and reads data 0x05 and 0x06 stored in the node F, G from the RAM 3; the HC3 compares the data 0x09 stored in the node C output by the HC2 with the data 0x05 and 0x06 stored in the node F, G, determines the smallest data 0x05 among them, and the corresponding node F; the HC3 returns the data 0x05 stored in the node F and the address of the node F to the HC 2; the HC3 stores the data 0x09 stored in the node C output by the HC2 into the node F, and the HC3 is already the last first operation submodule and is not output backward any more.

The HC1 stores the data 0x02 stored in the node C returned by the HC2 into the node A; the HC1 compares another data to be processed 0x0B with the data stored in the node a, 0x02, stores 0x0B into the node a since 0x0B > 0x02, and outputs the data stored in the node a, 0x0B, and the address of the node a to the HC 2.

Other clock cycles may be similarly analyzed and are not further enumerated. Note that the behavior of the first operation submodule in each clock cycle is not necessarily exactly the same as that listed above, and for example, it is also possible that at the 1 st clock cycle, the HC2 has already calculated the address of its child node B, C from the address of the node a output by the HC1, and therefore the above-described processing flow of the data processing apparatus is merely an example. It is easy to see that, in the above processing flow of the data processing apparatus, one to-be-processed data may be added to the heap structure every 2 clock cycles, and the first operation sub-modules corresponding to each layer of the heap structure perform pipeline operations in a cascade manner, and the next to-be-processed data may not be added to the heap structure until one to-be-processed data is stored in the final position, but may be processed in parallel to some extent, and the data processing efficiency is extremely high.

In some implementations, the data processing apparatus further includes a second operation module, and the second operation module is connected to the storage module. The second operation module is denoted as SC in fig. 5. The second operation module can adopt a hardware structure similar to that of the first operation module, even the second operation module and the first operation module are not excluded from being the same module, and different functions can be realized according to different setting modes.

The second operation module is used for sorting the data stored in the nodes of the heap structure according to a preset mode after the first operation module adds the data to be processed into the heap structure, and reading the sorted data in sequence; if the type of the heap structure is a small top heap, the data are sorted in an ascending order, and if the type of the heap structure is a large top heap, the data are sorted in a descending order. Of course, the responsibility for data reading by other components is not excluded.

In some implementations, the second operation module may further include a plurality of second operation sub-modules connected in sequence, and each of the second operation sub-modules may access data stored in one of the storage sub-modules corresponding to the second operation sub-module.

Referring to fig. 5, the second operation module further includes 3 second operation sub-modules, which are sequentially connected to each other and are respectively denoted as SC1, SC2, and SC 3. The SC1, SC2, and SC3 are responsible for operations on nodes in the level 1, level 2, and level 3 of the heap structure during the sorting process, respectively. Each storage submodule may be arranged to be isolated (but not mandatory), SC1 only having access to RAM1, SC2 only having access to RAM2, SC3 only having access to RAM 3.

The use of the second arithmetic sub-module is described below using a small top heap as an example:

the plurality of second operation sub-modules can be roughly divided into two types: the first second operation sub-module is one type, and the other second operation sub-modules are one type, which have certain differences in functions and are introduced separately. It should be understood that, for the sake of uniform design and manufacture, the two types of second arithmetic sub-modules may also adopt the same implementation in hardware.

The first second operation submodule has the functions of:

(1) reading data stored in the heap top node, and outputting the address of the heap top node to a second operation submodule;

(2) and storing the data stored in the second-layer node returned by the second operation submodule into the heap top node.

The function of any other second operation submodule is as follows:

(1) and reading the data stored in the child node of the previous layer node from the storage sub-module corresponding to the second operation sub-module according to the address of the child node of the previous layer node output by the previous second operation sub-module.

(2) Determining smaller data from the data stored in the child node of the previous layer of nodes, and returning the smaller data and the address of the child node storing the smaller data to the previous second operation submodule; when the second operation submodule exists in a next second operation submodule, the address of the child node storing the smaller data is output to the next second operation submodule; and when the child node storing the small data is a leaf node of the heap structure, storing the theoretical maximum value of the data to be processed into the child node storing the small data.

(3) And when the second operation submodule exists in a next second operation submodule, calculating the address of the father node of the next-layer node according to the address of the next-layer node returned by the next second operation submodule, and storing the data stored in the next-layer node into the father node of the next-layer node according to the address of the father node of the next-layer node.

The function of the first arithmetic submodule will be described below by taking a specific example, and assuming that the state in the stack structure at a certain time is as shown in fig. 4(a), the data processing apparatus takes the form of fig. 5.

1 st clock cycle: SC1 reads data 0x01 stored in node a, and outputs the address of node a to SC2 (SC 1 outputs data to SC2 is denoted by push1 in fig. 5).

2 nd clock cycle: the SC2 calculates the address of its child node B, C from the address of node a output by the SC1, and reads data 0x03 and 0x02 stored in the node B, C from the RAM 2; SC2 compares data 0x03 and 0x02 stored in node B, C to determine the smaller data 0x02 and the corresponding node C; the SC2 returns data 0x02 stored in node C and the address of node C to SC1 (the SC2 returns data to SC1 is denoted by pop2 in fig. 5); SC2 outputs the address of node C to SC3 (FIG. 5 shows SC2 outputting data to SC3 with push 2).

3 rd clock cycle: the SC3 calculates the address of its child node F, G from the address of node C output by the SC2, and reads data 0x05 and 0x06 stored in the node F, G from the RAM 3; SC3 compares data 0x05 and 0x06 stored in node F, G to determine the smaller data 0x05 and the corresponding node F; the SC3 returns data 0x05 stored in node F and the address of node F to SC2 (in fig. 5, pop3 indicates that SC3 returns data to SC 2); SC3 stores 0xFF into node F.

SC1 stores data 0x02 stored in node C returned by SC2 into node a, SC1 reads data 0x02 stored in node a, and outputs the address of node a to SC 2.

4 th clock cycle: the SC2 calculates the address of the parent node C according to the address of the node F returned by the SC3, and stores the data 0x05 stored in the node F returned by the SC3 into the node C; the SC2 calculates the address of its child node B, C from the address of node a output by the SC1, and reads data 0x03 and 0x05 stored in the node B, C from the RAM 2; SC2 compares data 0x03 and 0x05 stored in node B, C to determine the smaller data 0x03 and the corresponding node B; SC2 returns to SC1 the data 0x03 stored in node B and the address of node B; SC2 outputs the address of node B to SC 3.

Clock cycle 5: the SC3 calculates the address of its child node D, E from the address of node B output by the SC2, and reads data 0x07 and 0x04 stored in the node D, E from the RAM 3; SC3 compares data 0x07 and 0x04 stored in node D, E to determine the smaller data 0x04 and the corresponding node E; SC3 returns to SC2 the data 0x04 stored in node E and the address of node E; SC3 stores 0xFF into node E.

SC1 stores data 0x03 stored in node B returned by SC2 into node a, SC1 reads data 0x03 stored in node a, and outputs the address of node a to SC 2.

Other clock cycles may be similarly analyzed and are not further enumerated. Note that the behavior of the second arithmetic sub-module in each clock cycle is not necessarily exactly the same as that enumerated above, and for example, it is also possible that at the 1 st clock cycle, the SC2 has already calculated the address of its child node B, C from the address of node a output by the SC1, and therefore the above-described processing flow of the data processing apparatus is merely an example. It is obvious that, in the above processing flow of the data processing apparatus, one to-be-processed data can be read from the heap structure every 2 clock cycles, and the second operation sub-modules corresponding to each layer of the heap structure perform pipeline operations in a cascade manner, and the next to-be-processed data can be read from the heap structure not until the reading of one to-be-processed data is completed and the adjustment of the heap structure is completed, but rather, the data processing apparatus can perform parallel processing to a certain extent, and has extremely high data processing efficiency.

Fig. 6 shows a possible structure of an electronic device 200 provided in an embodiment of the present application. Referring to fig. 6, the electronic device 200 includes: a processor 210, a memory 220, and a communication interface 230, which are interconnected and in communication with each other via a communication bus 240 and/or other form of connection mechanism (not shown).

The Memory 220 includes one or more (Only one is shown in the figure), which may be, but not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), and the like. Processor 210, and possibly other components, may access, read, and/or write data from memory 220.

Processor 210 includes one or more (only one shown) which may be an integrated circuit chip having signal processing capabilities. The Processor 210 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Micro Control Unit (MCU), a Network Processor (NP), or other conventional processors; the Application Specific Processor may also be a special purpose Processor, including a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, and discrete hardware components.

Communication interface 230 includes one or more (only one shown) that can be used to communicate directly or indirectly with other devices for data interaction. Communication interface 230 may include an interface to communicate wired and/or wireless.

One or more computer program instructions may be stored in the memory 220 and read and executed by the processor 210 to implement the data processing methods provided by the embodiments of the present application and other desired functions.

It will be appreciated that the configuration shown in fig. 6 is merely illustrative and that electronic device 200 may include more or fewer components than shown in fig. 6 or have a different configuration than shown in fig. 6. The components shown in fig. 6 may be implemented in hardware, software, or a combination thereof. The electronic device 200 may be a physical device, such as a PC, a laptop, a tablet, a server, an embedded device, etc., or may be a virtual device, such as a virtual machine, a virtualized container, etc. The electronic device 200 is not limited to a single device, and may be a combination of a plurality of devices or one or more clusters of a large number of devices.

The embodiment of the present application further provides a computer-readable storage medium, where computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are read and executed by a processor of a computer, the computer-readable storage medium executes the data processing method provided by the embodiment of the present application. The computer readable storage medium may be embodied as, for example, the memory 220 in the electronic device 200 in fig. 6.

The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method, comprising:

acquiring a plurality of data to be processed and an initialized heap space, wherein each data storage unit in the heap space represents a node in a heap structure, and the type of the heap structure is a small top heap or a large top heap;

comparing each data to be processed with data stored in a heap top node, if the comparison result shows that the data to be processed meets the condition of adding the heap structure, storing the data to be processed into the heap top node, comparing the data to be processed with data stored in other nodes of the heap structure, and determining the storage position of the data in the nodes of the heap structure according to the comparison result so that the heap structure still maintains the original type after storing the data to be processed in the nodes of the heap structure;

data stored in a node of the heap structure is read.

2. The data processing method according to claim 1, wherein the type of the heap structure is a small top heap, the comparing of each piece of data to be processed with data stored in a heap top node, if a comparison result indicates that the piece of data to be processed satisfies a condition for joining the heap structure, storing the piece of data to be processed in the heap top node, comparing the piece of data to be processed with data stored in other nodes of the heap structure, and determining a storage location of the data in the node of the heap structure according to the comparison result includes:

comparing each data to be processed with data stored in a heap top node, and if the data to be processed is larger than the data stored in the heap top node, storing the data to be processed into the heap top node;

starting from the heap top node, if the current node has a child node, comparing the data stored in the current node with the data stored in the child node, if the node corresponding to the minimum data is the current node, terminating the comparison process, if the node corresponding to the minimum data is the child node of the current node, exchanging the data stored in the current node with the data stored in the child node, and taking the child node as a new current node to continue to compare until the current node does not have a child node or the comparison process is terminated.

3. The data processing method according to claim 2, wherein the number of the to-be-processed data is N, and before the obtaining of the plurality of the to-be-processed data and the initializing of the good heap space, the method further comprises:

distributing K data storage units in a storage medium as the heap space, and initializing the data storage units in the heap space to be the theoretical minimum value of the data to be processed;

or distributing M data storage units in a storage medium as the heap space, initializing the first K data storage units in the heap space to the theoretical minimum value of the data to be processed, and initializing the last M-K data storage units in the heap space to the theoretical maximum value of the data to be processed;

where K is the amount of data that needs to be obtained by data processing, M > K.

4. A data processing method according to claim 3, wherein M-2ⁿ-1, n-ceil (log2(K +1)), where ceil denotes the ceiling operation and n denotes the total number of levels of the heap structure.

5. The data processing method of claim 3, wherein the heap space is a continuous storage space in the storage medium, and the data storage units representing each layer node of the heap structure are arranged in sequence in the heap space.

6. The data processing method of any of claims 1-5, wherein the reading data stored in the nodes of the heap structure comprises:

sorting the data stored in the nodes of the heap structure according to a preset mode, and sequentially reading the sorted data; and if the type of the heap structure is a small top heap, sorting the data in an ascending order, and if the type of the heap structure is a large top heap, sorting the data in a descending order.

7. The data processing method according to claim 6, wherein the type of the heap structure is a small top heap, and the sorting the data stored in the nodes of the heap structure according to the predetermined manner and sequentially reading the sorted data comprises:

reading data stored in a heap top node;

starting from the heap top node, if a current node has a child node, determining smaller data from data stored in the child node of the current node, storing the smaller data in the current node, and taking the child node corresponding to the smaller data as a new current node to continue comparison until a comparison process is terminated when the child node does not exist in the current node, and storing a theoretical maximum value of the data to be processed in the current node;

and continuously reading the data stored in the heap top node until all the data stored in the nodes of the heap structure are read.

8. A data processing device is characterized by comprising a first operation module and a storage module, wherein the first operation module is connected with the storage module;

a heap space is allocated in the storage module, each data storage unit in the heap space represents a node in a heap structure, and the type of the heap structure is small top heap or large top heap;

the first operation module is used for comparing each piece of data to be processed with the data stored in the heap top node, if the comparison result shows that the data to be processed meets the condition of adding the heap structure, the data to be processed is stored in the heap top node, the data to be processed is compared with the data stored in other nodes of the heap structure, and the storage position of the data in the nodes of the heap structure is determined according to the comparison result, so that the heap structure still maintains the original type after the data to be processed is stored in the nodes of the heap structure.

9. The data processing apparatus according to claim 8, wherein the first operation module includes a plurality of first operation sub-modules, the plurality of first operation sub-modules are connected in sequence, the storage module includes a plurality of storage sub-modules, a storage unit in each storage sub-module represents a layer of node in the heap structure, the type of the heap structure is a small top heap, and each first operation sub-module can access data stored in a corresponding storage sub-module;

a first one of the plurality of first operation sub-modules is configured to: comparing each data to be processed with data stored in a heap top node, storing the data to be processed into the heap top node when the data to be processed is larger than the data stored in the heap top node, and outputting the data stored in the heap top node and the address of the heap top node to a second first operation submodule;

the first operation submodule is further configured to: when data stored in a second layer node returned by the second first operation submodule is received, storing the data stored in the second layer node into the heap top node;

the other first operation sub-modules except the first one of the plurality of first operation sub-modules are used for: calculating the address of the child node of the previous layer node according to the address of the previous layer node output by the previous first operation submodule, reading the data stored in the child node of the previous layer node from the storage submodule corresponding to the first operation submodule according to the address of the child node of the previous layer node, comparing the data stored in the previous layer node output by the previous first operation submodule with the data stored in the child node thereof, terminating the output of the data to the next first operation submodule when the node corresponding to the minimum data is the previous layer node, returning the data stored in the child node and the address of the child node to the previous first operation submodule when the node corresponding to the minimum data is the child node of the previous layer node, and storing the data stored in the previous layer node into the child node, and the first operation submodule is used for outputting the data stored in the child node and the address of the child node to the next first operation submodule when the first operation submodule exists in the next first operation submodule;

the first operation submodule other than the first operation submodule is further configured to: when the first operation submodule exists in a next first operation submodule and receives data stored in a next-layer node returned by the next first operation submodule and the address of the next-layer node, calculating the address of a father node of the next-layer node according to the address of the next-layer node, and storing the data stored in the next-layer node into the father node of the next-layer node according to the address of the father node of the next-layer node.

10. The data processing apparatus according to claim 8, further comprising a second operation module, the second operation module being connected to the storage module;

the second operation module is used for sequencing the data stored in the nodes of the heap structure according to a preset mode and sequentially reading the sequenced data; and if the type of the heap structure is a small top heap, sorting the data in an ascending order, and if the type of the heap structure is a large top heap, sorting the data in a descending order.

11. The data processing apparatus according to claim 10, wherein the second operation module includes a plurality of second operation sub-modules, the plurality of second operation sub-modules are connected in sequence, the storage module includes a plurality of storage sub-modules, a storage unit in each storage sub-module represents a layer of node in the heap structure, the type of the heap structure is a small top heap, and each second operation sub-module can access data stored in a corresponding storage sub-module;

a first one of the plurality of second operation sub-modules is configured to: reading data stored in the heap top node, and outputting the address of the heap top node to a second operation submodule;

the first second operation submodule is further configured to: storing the data stored in the second layer node returned by the second operation submodule into the heap top node;

the second operation sub-modules except the first one of the plurality of second operation sub-modules are used for: calculating the address of the child node of the previous layer node according to the address of the previous layer node output by the previous second operation submodule, and reads the data stored in the child node of the previous layer node from the storage submodule corresponding to the second operation submodule according to the address of the child node of the previous layer node, and is used for determining smaller data from the data stored in the child node of the previous layer node and returning the smaller data and the address of the child node storing the smaller data to the previous second operation submodule, and is used for outputting the address of the child node storing the smaller data to the next second arithmetic submodule when the second arithmetic submodule exists in the next second arithmetic submodule, and when the child node storing the small data is a leaf node of the heap structure, storing the theoretical maximum value of the data to be processed into the child node storing the small data;

the second operation submodule except the first operation submodule is also used for: when the second operation submodule exists in a next second operation submodule, the address of the father node of the next layer node is calculated according to the address of the next layer node returned by the next second operation submodule, and the data stored in the next layer node is stored in the father node of the next layer node according to the address of the father node of the next layer node.

12. A computer-readable storage medium having computer program instructions stored thereon, which when read and executed by a processor, perform the method of any one of claims 1-7.

13. An electronic device, comprising: a memory having stored therein computer program instructions which, when read and executed by the processor, perform the method of any of claims 1-7.