CN115619964A

CN115619964A - Data processing method and device

Info

Publication number: CN115619964A
Application number: CN202110798664.7A
Authority: CN
Inventors: 徐斌; 何雷骏; 舒嘉明
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2021-07-15
Filing date: 2021-07-15
Publication date: 2023-01-17

Abstract

The application provides a data processing method and device, which can improve the storage efficiency of stored data. The method comprises the following steps: determining a first table entry used for indexing a first data block in a hash table according to a received first message, wherein the first message is used for indicating to store the first data block, the first message comprises an identifier of the first data block and the length of the first data block, and the first table entry is unoccupied; determining address information of at least one memory block for storing the first data block according to the length of the first data block; storing the first data block into the at least one memory block according to the address information of the at least one memory block; and writing the address information of the at least one memory block, the identifier of the first data block and the first indication information into the first table entry.

Description

Data processing method and device

Technical Field

The present application relates to the field of data processing technology, and more particularly, to a method and apparatus for data processing.

Background

Sparsity is a main characteristic of three-dimensional scene data, and for example, in an animation simulation or a light rendering application scene, the three-dimensional scene data is conventionally sparse, locally dense, and possibly completely dense. And dividing the three-dimensional scene into different space grids, wherein the different space grids correspond to different data. Under the condition that three-dimensional scene data are sparse, some spatial grids have no data, and some spatial grids have corresponding data, and different spatial grids under the condition can be called sparse grids.

In the prior art, the space sparse data is regarded as completely dense space grid data to be stored, so that the storage cost and the calculation cost of the data are high. In order to fully utilize the sparsity of data and reduce the storage cost and the calculation cost of the data, a method for only storing the spatial grid data of the data is provided.

For example, an open source voxel data base (OpenVDB) is used to store space grid data, a memory of the OpenVDB is a dynamic application, and when the space grid data changes or a new space grid data needs to be stored, a corresponding memory needs to be temporarily applied. For another example, the GVDB stores space network data, and the memory pool of the GVDB is suitable for storing fixed-length data, that is, the size of one memory block corresponds to the size of one piece of voxel data in space, and the application scenario is limited, and is not suitable for the case where the data size in the storage space grid is variable; in addition, the storage address of the voxel data is determined layer by layer according to the coordinate position of the voxel data in the three-dimensional scene and the bit masks corresponding to different coordinate positions, and the time consumption is long.

Disclosure of Invention

The application provides a data processing method and device, which can reduce the storage time delay.

In a first aspect, a method of data processing is provided, which may be performed by a data processing unit or a detection unit in a data processing chip or device. The method comprises the following steps: determining a first table entry used for indexing a first data block in a hash table according to a received first message, wherein the first message is used for indicating to store the first data block, the first message comprises an identifier of the first data block and the length of the first data block, and the first table entry is unoccupied; determining address information of at least one memory block for storing the first data block according to the length of the first data block, where the length of data storable in the at least one memory block is greater than or equal to the length of the first data block, and the length of data storable in each memory block is fixed; storing the first data block into the at least one memory block according to the address information of the at least one memory block; writing the address information of the at least one memory block, the identifier of the first data block, and first indication information into the first table entry, so as to obtain the first data block from the at least one memory block, where the first indication information is used to indicate that the at least one memory block is occupied.

Based on the foregoing technical solution, when a first data block needs to be stored, an unoccupied first table entry is determined in the hash table, after at least one memory block used for storing the first data block is determined according to the length of the first data block, the first slave data block is stored in the at least one memory block, and address information of the at least one memory block, an identifier of the first data block, and first indication information used for indicating that the at least one memory block is occupied are written into the first table entry, so as to obtain the first data block from the at least one memory block. According to the scheme, at least one memory block for storing the first data block can be directly determined from a storage unit which is applied in advance according to the length of the first data block, and the address information of the memory block for storing the first data block is determined conveniently and efficiently; and writing the address information of the memory block for storing the first data block into the first table entry of the hash table, so as to obtain the stored first data block again according to the address information in the first table entry. Therefore, the scheme can improve the storage efficiency of the stored data. Besides, the scheme can also realize the storage of data blocks with the data length being not fixed.

With reference to the first aspect, in certain implementations of the first aspect, the determining, according to the received first message, a first entry in a hash table for indexing a first data block includes: determining a row number of a table entry used for indexing the first data block in the hash table according to the identifier of the first data block; and determining the unoccupied first table entry from the hash table according to the row number.

With reference to the first aspect, in certain implementations of the first aspect, the determining, according to the identifier of the first data block, a row number of an entry in the hash table, where the entry is used to index the first data block, includes: determining the row number based on a hash function and an identification of the first block, the hash function comprising:

f(k)＝kmodm

wherein k is the identifier of the first data block, m is the length of the hash table, and f (k) is the row number of the first entry in the hash table.

With reference to the first aspect, in some implementation manners of the first aspect, the determining, according to the length of the first data block, address information of at least one memory block used for storing the first data block includes: receiving first information sent by a storage controller, wherein the first information comprises address information of a plurality of unoccupied memory blocks in a storage unit corresponding to the storage controller; according to the length of the first data block, determining address information of the at least one memory block used for storing the first data block from the address information of the plurality of unoccupied memory blocks.

With reference to the first aspect, in certain implementation manners of the first aspect, the determining, according to the length of the first data block, address information of at least one memory block used for storing the first data block includes: sending a first request message to a storage controller, where the first request message is used to obtain address information of the at least one memory block storing the first data block, and the first request message includes a length of the first data block, where the storage controller is used to determine, according to the length of the first data block, the address information of the at least one memory block that is not occupied in a storage unit corresponding to the storage controller; receiving a second response message sent by the storage controller, where the second response message includes address information of the at least one unoccupied memory block.

With reference to the first aspect, in certain implementation manners of the first aspect, the address information of the at least one memory block includes: starting address information of a first memory block of the at least one memory block and the number of the memory blocks, where addresses of each memory block of the at least one memory block are consecutive.

With reference to the first aspect, in certain implementations of the first aspect, the writing the address information of the at least one memory block, the identifier of the first data block, and the first indication information into the first entry includes: writing the start address information of a first memory block of the at least one memory block, the number of the memory blocks, the identifier of the first data block, and the first indication information into the first table entry.

With reference to the first aspect, in certain implementations of the first aspect, the method further includes: determining the first table entry used for indexing the first data block in the hash table according to a received second message, wherein the second message is used for indicating to acquire the first data block, and the second message comprises an identifier of the first data block; and obtaining the first data block from the at least one memory block according to the address information of the at least one memory block written in the first table entry.

With reference to the first aspect, in certain implementations of the first aspect, the method further includes: determining the first table entry used for indexing the first data block in the hash table according to a received third message, where the third message is used to instruct to process the first data block, and the third message includes an identifier of the first data block; acquiring the first data block from the at least one memory block according to the address information of the at least one memory block written in the first table entry; processing the acquired first data block; replacing the first data block in the at least one memory block with the processed first data block.

With reference to the first aspect, in certain implementations of the first aspect, the method further includes: determining the first table entry used for indexing the first data block in the hash table according to a received fourth message, where the fourth message is used for indicating to release or delete the first data block, and the fourth message includes an identifier of the first data block; modifying the first indication information in the first table entry into second indication information, where the second indication information is used to indicate that the at least one memory block is not occupied; sending a second request message to the storage controller, where the second request message is used to request the storage controller to mark the at least one memory block as unoccupied. Based on the scheme, at least one memory block can be reclaimed for storing other data blocks.

In a second aspect, a data processing apparatus is provided, including: a detection unit to: determining a first table entry used for indexing a first data block in a hash table according to a received first message, wherein the first message is used for indicating to store the first data block, the first message comprises an identifier of the first data block and the length of the first data block, and the first table entry is unoccupied; determining address information of at least one memory block for storing the first data block according to the length of the first data block, where the length of data storable in the at least one memory block is greater than or equal to the length of the first data block, and the length of data storable in each memory block is fixed; storing the first data block into the at least one memory block according to the address information of the at least one memory block; writing the address information of the at least one memory block, the identifier of the first data block, and first indication information into the first table entry, so as to obtain the first data block from the at least one memory block, where the first indication information is used to indicate that the at least one memory block is occupied.

With reference to the second aspect, in certain implementations of the second aspect, the detection unit is specifically configured to: determining a row number of a table entry used for indexing the first data block in the hash table according to the identifier of the first data block; and determining the first table entry which is not occupied from the hash table according to the row number.

With reference to the second aspect, in some implementations of the second aspect, the detection unit is specifically configured to: determining the row number based on a hash function and an identification of the first block, the hash function comprising:

f(k)＝kmodm

With reference to the second aspect, in certain implementations of the second aspect, the detection unit is specifically configured to: receiving first information sent by a storage controller, wherein the first information comprises address information of a plurality of unoccupied memory blocks in a storage unit corresponding to the storage controller; according to the length of the first data block, determining address information of the at least one memory block used for storing the first data block from the address information of the plurality of unoccupied memory blocks.

With reference to the second aspect, in certain implementations of the second aspect, the detection unit is specifically configured to: sending a first request message to a storage controller, where the first request message is used to obtain address information of the at least one memory block storing the first data block, and the first request message includes a length of the first data block, where the storage controller is used to determine, according to the length of the first data block, the address information of the at least one memory block that is not occupied in a storage unit corresponding to the storage controller; receiving a second response message sent by the storage controller, where the second response message includes address information of the at least one unoccupied memory block.

With reference to the second aspect, in some implementations of the second aspect, the address information of the at least one memory chunk includes: starting address information of a first memory block of the at least one memory block and the number of the memory blocks, where addresses of each memory block of the at least one memory block are consecutive.

With reference to the second aspect, in certain implementations of the second aspect, the detection unit is specifically configured to: writing the start address information of a first memory block of the at least one memory block, the number of the memory blocks, the identifier of the first data block, and the first indication information into the first table entry.

With reference to the second aspect, in certain implementations of the second aspect, the detection unit is further configured to: determining the first table entry used for indexing the first data block in the hash table according to a received second message, wherein the second message is used for indicating to acquire the first data block, and the second message comprises an identifier of the first data block; and obtaining the first data block from the at least one memory block according to the address information of the at least one memory block written in the first table entry.

With reference to the second aspect, in certain implementations of the second aspect, the apparatus further includes a computing unit; the detecting unit is further configured to determine, according to a received third message, the first entry used for indexing the first data block in the hash table, where the third message is used to instruct to process the first data block, and the third message includes an identifier of the first data block; the detecting unit is further configured to obtain the first data block from the at least one memory block according to the address information of the at least one memory block written in the first table entry; the calculation unit is configured to process the first data block acquired by the detection unit; the computing unit is further configured to replace the first data block in the at least one memory block with the processed first data block.

With reference to the second aspect, in certain implementations of the second aspect, the detection unit is further configured to: determining the first table entry used for indexing the first data block in the hash table according to a received fourth message, where the fourth message is used to indicate that the first data block is released or deleted, and the fourth message includes an identifier of the first data block; modifying the first indication information in the first table entry into second indication information, where the second indication information is used to indicate that the at least one memory block is not occupied; sending a second request message to the storage controller, where the second request message is used to request the storage controller to mark the at least one memory block as unoccupied.

In a third aspect, a data processing apparatus is provided, which comprises at least one processor. The processor is coupled to the memory and is operable to execute instructions in the memory to implement the method of data processing in any of the possible implementations of the first aspect and the first aspect.

With reference to the third aspect, in certain implementations of the third aspect, the data processing apparatus is a chip or a chip system, and the processor may be a processing circuit, a logic circuit, or the like.

With reference to the third aspect, in certain implementations of the third aspect, the data processing apparatus further includes a communication interface, which is an interface circuit, an input and/or output interface, or an input and/or output circuit, and so on.

With reference to the third aspect, in certain implementations of the third aspect, the data processing apparatus further includes the memory.

In a fourth aspect, a data processing chip is provided, which includes a processor, the processor is connected to a memory, the memory is used for storing a computer program, and the processor is used for executing the computer program stored in the memory, so that the chip executes the method in any one of the above-mentioned first aspect and possible implementation manners of the first aspect.

In a fifth aspect, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a data processing apparatus, causes the data processing apparatus to implement the method of data processing in any one of the possible implementations of the first aspect and the first aspect.

A sixth aspect provides a computer program product comprising instructions which, when executed by a computer, cause a data processing apparatus to carry out the method of data processing according to the first aspect as well as any possible implementation manner of the first aspect.

Drawings

Fig. 1 is a schematic diagram of a sparse grid.

Fig. 2 is a schematic diagram of another sparse grid.

Fig. 3 is a schematic view of voxel data.

Fig. 4 is a GVDB schematic.

Fig. 5 is a schematic diagram of three-dimensional scene data slicing.

Fig. 6 is a schematic flow chart of a method of data processing according to an embodiment of the present application.

Fig. 7 is a schematic block diagram of a data processing apparatus according to an embodiment of the present application.

Fig. 8 is a schematic block diagram of another data processing apparatus according to an embodiment of the present application.

Fig. 9 is a schematic block diagram of another data processing apparatus according to an embodiment of the present application.

Detailed Description

The technical solution in the present application will be described below with reference to the accompanying drawings.

Sparsity is a main characteristic of three-dimensional scene data, and for example, in an animation simulation or a light rendering application scene, the three-dimensional scene data is conventionally sparse, locally dense, and possibly completely dense.

And dividing the three-dimensional scene into different space grids, wherein the different space grids correspond to different data. Under the condition that three-dimensional scene data is sparse, some spatial grids have no data, and some spatial grids have corresponding data, and different spatial grids under the condition can be called sparse grids. Sparse meshes may exhibit locally dense features or exhibit conventional sparse features, which are often referred to as spatial sparsity (spatial sparsity); the local dense feature shows that data continuity exists between partial adjacent spatial grids, and local dense occurs; conventional sparse features show no data continuity between all neighboring spatial meshes. As shown in fig. 1, a schematic diagram of a sparse grid is shown, where data continuity exists between different spatial grids and local density occurs. As shown in fig. 2, a schematic diagram of another sparse grid is shown, where there is no data continuity between different spatial grids. Each spatial grid corresponds to data of a different length.

In addition to sparse meshes, there is also completely dense data in the three-dimensional scene, i.e. there is data continuity between all adjacent spatial meshes.

The Material Point Method (MPM) is a common algorithm in animation simulation scenes, and requires sparse grid data and dense material point data. In an actual application example, the MPM simulates 100M particles by using a 40963 spatial grid, and it is considered that 1024GB (16975 b) of memory is required to store data corresponding to all spatial grids, but the actually used memory only occupies 0.016%, that is, 160MB, and the sparsity rate reaches 99% or more.

For the convenience of understanding the embodiments of the present application, brief descriptions will be made on the prior art related to the embodiments of the present application.

As shown in fig. 3, a schematic view of voxel data is shown. To process spatial grid data, the entire three-dimensional space is divided into a grid of cubes, each called a voxel, corresponding to a pixel of a 2-dimensional plane. The data is sparse, occupying only a small portion of the total space. Currently, a typical data structure is called a Voxel Database (VDB) tree, and a Central Processing Unit (CPU) implementation version is called an open voxel database (OpenVDB), and the OpenVDB is used for storing spatial grid data.

OpenVDB is a commonly used data structure in spatial sparsity, and the design concept is as follows:

(1) The data is stored in a tree structure.

(2) A tree with a depth of 4, the width of each layer being a power of 2.

(3) And the layer 1 (root) realizes sparse storage by using hash (hash), and maintains a mapping table from grid coordinates to a root child level (root child). The method is dynamically managed, namely a memory address is applied when a memory space is needed, and the memory of the subtree can be applied by the subtree at the first layer.

(4) Layer 2, layer 3 (internal) are densely stored, and three rows of small boxes from top to bottom in the figure represent an active mask (active mask), a child mask (child mask), and a child pointer address, respectively. The active mask indicates whether to store a value, and the child mask indicates whether to have child nodes.

(5) Layer 4 (leaf) densely stores masks, values. values store fixed-size data.

The memory of the OpenVDB is a dynamic application, and when the spatial grid data changes or new spatial grid data needs to be stored, a corresponding memory needs to be temporarily applied. Therefore, the time overhead of dynamically applying for the memory is relatively large, and is not suitable for a dedicated hardware accelerator architecture, for example: acceleration chips with higher time delay requirements, and the like.

Currently, another VDB data structure is a version suitable for a Graphics Processing Unit (GPU), called GVDB.

The design idea of GVDB is as follows:

(1) Without hash, each layer in the mesh topology is a bit mask (bitmask) + child level (child), which can be regarded as a one-to-one mapping table, and records whether voxel data exists in a three-dimensional space.

Taking data as an example, according to the coordinate position of the voxel data in the three-dimensional scene and bit masks corresponding to different coordinate positions, determining the storage address of the voxel data layer by layer, and acquiring the data to be acquired according to the storage address of the voxel data. As shown in fig. 4, a GVDB diagram is shown.

(2) Dynamic allocation and release of memory is accelerated with memory pools (memory pools): the leaf node value (value) is an address that points to a fixed-size block of memory (block) in the memory pool. The block in the memory pool can be recovered and then used for storing other data.

The memory pool of the GVDB is suitable for storing fixed-length data, namely, one block corresponds to the size of one piece of voxel data in a space, the application scene is limited, and the memory pool is not suitable for the condition that the data volume in a storage space grid is not variable. In addition, the storage addresses of the voxel data are determined layer by layer according to the coordinate positions of the voxel data in the three-dimensional scene and the bit masks corresponding to different coordinate positions, and the time consumption is long. Therefore, the scheme is not suitable for an acceleration chip or device with higher requirement on time delay.

Therefore, the application provides a data processing method which can reduce the time delay of data storage.

In order to facilitate understanding of the present application, a brief description is given of segmentation of three-dimensional scene data in the prior art. The segmentation of the three-dimensional scene data includes uniform segmentation and arbitrary segmentation. As shown in fig. 5, a schematic diagram illustrating slicing of three-dimensional scene data is shown.

Fig. 5 (a) is a schematic diagram of uniform slicing, which means that the space is equally divided into multiple parts, for example, the whole space is assumed to have a size of 100 × 100, wherein each 1 × 1 unit is a minimum unit, and if each axis is sliced by 10 parts, the space is uniformly sliced into 1000 data blocks in total, each referred to as a box (box), and the slicing is assumed to be axial. Each part can be a local dense data block, and the data in one data block comprises non-zero elements or all-zero data; the box identifier (box id) may be any identifier that can uniquely identify the current data block, and may be sequentially numbered according to the z-y-x axis, for example. In this scenario, box id = X Y Z + Z, where X, Y, Z are the spatial coordinates of the box's top left vertex, and X, Y, Z are the size of the space, where the spatial coordinates are integer values.

Fig. 5 (b) is a schematic diagram of arbitrary segmentation, where the arbitrary segmentation indicates that there may be overlap in the segmentations of the space, the sizes of each data block may be unequal, and all data blocks do not need to contain the entire space, nor do they require that the coordinates of the box be integer values; under the scene, box id can not be simply calculated, and only unique identification is needed, for example, the numbers of 0,1,2, \ 8230, and the like are sequentially numbered according to the establishing sequence.

As shown in fig. 6, a schematic flow chart of a method 600 for data processing provided by the embodiment of the present application is shown. The method can reduce the time delay of data storage. The execution subject of the method may be a data processing chip or a data processing unit in a device, the data processing unit comprising a detection unit and a calculation unit. The method specifically comprises the following steps:

and 610, after the detection unit receives a first message used for indicating to store the first data block, determining a first entry used for indexing the first data block in the hash table according to the received first message, where the first message includes an identifier of the first data block and a length of the first data block. The first table entry is an unoccupied table entry, the unoccupied table entry may be a null table entry, and the null table entry may be understood as a table entry into which no information is written. Optionally, the unoccupied table entry may also be a table entry into which information has been written, and the written information includes indication information indicating that the memory block corresponding to the first table entry is unoccupied.

The hash table includes a plurality of consecutive entries, which are stored in an internal buffer of the data processing chip or device. The hash table includes a plurality of rows, each row consisting of a key (K) value and a consecutive plurality of entries of the same size. The length of the K value may be 32 bits, each entry is a fixed size, such as 256 bits, and the specific size may be configured in advance according to the application requirement.

As shown in table 1, the hash table content in the embodiment of the present application is shown. The entry information written into each entry in the hash table may include:

(1) A valid (v) signal, occupying a bit (bit), indicating whether the corresponding memory block is occupied; for example: when v is equal to 1, it may indicate that the corresponding memory block is occupied, and when v is equal to 0, it may indicate that the corresponding memory block is not occupied;

(2) Identification of data block (box id);

(3) Starting address information (address, ad) of at least one memory block stored in the data block;

(4) The number M of consecutive memory blocks in the at least one memory block stored by the data block.

Table 1 hash table content

K1	(v1，box id1，ad1，M1)	(v2，box id2，ad2，M2)	…
				K2	…	…	…
…	…	…	…

Optionally, the entry information of each entry in the hash table may also only include v, an identification box id of the data block, and address information of each memory block used for storing the data block, and the number of consecutive memory blocks used for storing the data block is not required to be included, which is not specifically limited in this application.

Since hash collision may exist, that is, different data identifiers, for example, box id1 and box id2, are calculated by a hash function to obtain the same key value, for example, K1, they are both stored in the entry corresponding to K1, and both box id1 and box id2 are used as entries of K1 and are embodied in the same row of the hash table.

For data of indefinite length, M is variable; for fixed length data, M is the same. Besides these necessary items, other necessary table entry information, such as coordinates of the data block in the three-dimensional scene, may be added to the hash table according to the actual application requirements.

As a possible implementation manner, a row number of an entry in the hash table for indexing the first data block may be determined according to the identifier of the first data block; and determining an unoccupied first table entry from the row of the hash table according to the row number. When there are more empty entries in the multiple entries corresponding to the row number, one empty entry may be selected arbitrarily as the first entry, or one empty entry may be selected sequentially from left to right of the row as the first entry.

Illustratively, the detection unit may determine the line number based on a hash function (hash function) and an identification of the first data block, the hash function being represented by the following formula (1):

f(k)＝kmodm (1)

wherein k is the identifier of the first data block, m is the length of the hash table, and f (k) is the row number of the first table entry in the hash table. f (K) may be equivalent to K. It should be understood that the hash function may also be represented by other formulas as long as hash mapping can be implemented, and this application is not limited in this respect.

The detecting unit determines, according to the length of the first data block, address information of at least one memory block used for storing the first data block, where the length of data storable in the at least one memory block is greater than or equal to the length of the first data block, and the length of data storable in each memory block is fixed. When the length of the first data block is greater than the length of data which can be stored in one memory block, at least two memory blocks need to be determined to store the first data block; when the length of the first data block is smaller than or equal to the length of data that can be stored in one memory block, only one memory block needs to be determined to store the first data block. It should be understood that the length of the data that can be stored in each memory block may also be non-fixed, and the present application is not limited thereto.

Optionally, the address information of the at least one memory block may include: starting address information of a first memory block in the at least one memory block and the number of the memory blocks in the at least one memory block, where addresses of each memory block in the at least one memory block are consecutive.

Illustratively, the detection unit receives first information actively sent by the storage controller, where the first information includes address information of a plurality of unoccupied memory blocks in the storage unit corresponding to the storage controller; the detection unit determines address information of at least one memory block for storing the first data block from a plurality of unoccupied memory blocks according to the length of the first data block to be stored. The determined at least one memory block may be continuous or discontinuous. When the determined at least one memory block is continuous, the address information of the at least one memory block may include start address information of a first memory block in the at least one memory block and a number of memory blocks in the at least one memory block. When the determined at least one memory block is discontinuous, the address information of the at least one memory block may include address information of each memory block.

For example, the detection unit storage controller sends a first request message, where the first request message is used to request to acquire address information of at least one memory block storing the first data block, and the first request message includes a length of the first data block, where the storage controller is used to determine, according to the length of the first data block, the address information of at least one memory block that is not occupied in a storage unit corresponding to the storage controller. The detection unit receives a second response message sent by the storage controller, where the second response message includes address information of at least one unoccupied memory block, and the at least one unoccupied memory block may be continuous or discontinuous. When the at least one unoccupied memory block is discontinuous, the address information of the at least one memory block may be address information of each of the at least one memory block. When the at least one unoccupied memory block is consecutive, the address information of the at least one memory block may be start address information of a first memory block in the at least one memory block and the number of memory blocks in the at least one memory block, or may be address information of each memory block in the at least one memory block.

In different application scenarios, the size of the storage space required by different data blocks is different, and assuming that the size of one memory block is 256B and the size of the storage space required by the first data block is 300B, two memory blocks are required to store the first data block. The number of memory blocks required to store a data block can be understood as the length of the data block.

It should be understood that, if the storage controller receives a first request message for requesting to acquire address information of at least one memory block storing the first data block, and finds that the length of data that can be stored in the memory block that is not occupied in the storage unit corresponding to the storage controller is smaller than the length of the first data block, a second response message returned by the storage controller indicates that the address information application is failed.

The storage unit in the method is applied statically, and it can be understood that the storage unit for storing the data block is applied before the storage of the data block. The storage unit may be a Double Data Rate (DDR) memory or a High Bandwidth Memory (HBM). In the case of DDR, DDR is regarded as a memory cell as a carrier for storing data, and a portion storing data therein is called a "memory pool". In the memory pool, there may be N memory blocks (blocks) of the same size, where N is given at the time of memory application, and the size (size, S) of each memory block is also given in advance, for example: s =256b, n =1024, then the memory space required by the memory pool is 256b × 1024=256kb, and n and S may vary. The storage unit may be a memory independent from the data processing chip or device, or may be a storage module configured inside the data processing chip or device.

The storage controller in the storage unit is an independent logic unit and is used for marking whether the memory blocks in the memory pool are occupied or not. The storage controller has flag information indicating whether each memory block is occupied, for example: occupied is marked with 1, unoccupied is marked with 0; the size S of the memory block in the memory pool; and the number N of the memory blocks in the memory pool. The memory controller records the number of the remaining unoccupied memory blocks in the memory pool in real time, and can provide an interface for providing the address information of the unoccupied memory blocks; the provided address information of the unoccupied memory blocks may include start address information of the unoccupied consecutive M memory blocks.

For example, the detection unit may also determine the address information of at least one memory block used for storing the first data block by itself without the storage controller providing the address information of the unoccupied memory block. This is not a particular limitation of the present application.

630, the detecting unit stores the first data block to the at least one memory block according to the address information of the at least one memory block.

In a possible implementation manner, the first message may include not only the identifier of the first data block and the length of the first data block, but also the first data block, and the detecting unit may store the first data block in the first message to the at least one memory block according to the address information of the at least one memory block.

The detecting unit writes the address information of the at least one memory block, the identifier of the first data block, and the first indication information into the first table entry, and is configured to obtain the first data block from the at least one memory block according to the written information in the first table entry, where the first indication information is used to indicate that the at least one memory block is occupied. It should be understood that the first indication information may be a v signal in the table entry, and when v is equal to 1, it may indicate that the corresponding memory block is occupied.

Exemplarily, when the address information of the at least one memory block includes the start address information of the first memory block in the at least one memory block and the number of the memory blocks in the at least one memory block, the start address information of the first memory block in the at least one memory block, the number of the memory blocks, the identifier of the first data block, and the first indication information are written into the first table entry.

It should be understood that the steps of 630 and 640 may be performed simultaneously, the step of 640 may also be performed before the step of 630, and the present application does not limit the sequence of this step, and all the steps are within the scope of the present application.

In the technical scheme provided by the application, when a first data block needs to be stored, an unoccupied first table entry is determined in a hash table, after at least one memory block used for storing the first data block is determined according to the length of the first data block, the first slave data block is stored in the at least one memory block, and address information of the at least one memory block, an identifier of the first data block, and first indication information used for indicating that the at least one memory block is occupied are written into the first table entry so as to obtain the first data block from the at least one memory block. According to the scheme, at least one memory block for storing the first data block can be directly determined from a storage unit which is applied in advance according to the length of the first data block, and the address information of the memory block for storing the first data block is determined conveniently and efficiently; and the address information of the memory block for storing the first data block is written into the first table entry of the hash table, so that the stored first data block can be obtained again according to the address information in the first table entry. Therefore, the scheme can improve the storage efficiency of the stored data. Besides, the scheme can also realize the storage of data blocks with the data length being not fixed.

Optionally, if the detection unit receives a second message indicating to acquire the first data block, where the second message includes the identifier of the first data block, the detection unit may determine, according to the identifier of the first data block, a first entry used for indexing the first data block in the hash table.

For example, the detection unit may first determine, according to the identifier of the first data block and the hash function, a row number of an entry in the hash table, where the entry is used to index the first data block; and then according to the determined line number, determining whether the written box id in each table entry of the line is the same as the identifier of the first data block, if the box id in a certain table entry is the same as the identifier of the first data block, determining that the table entry is the first table entry, and indicating that the first data block is actually stored in the storage unit. It should be understood that if the written box id in each entry of the row is not the same as the identification of the first data block, it indicates that the first data block does not exist in the storage unit.

The detecting unit obtains the first data from the at least one memory block according to the address information of the at least one memory block written in the first table entry.

Optionally, if the detection unit receives a third message indicating to process the first data block, where the third message includes an identifier of the first data block, the detection unit may determine, according to the identifier of the first data block, a first entry used for indexing the first data block in the hash table.

For example, the detection unit may first determine, according to the identifier of the first data block and the hash function, a row number of an entry in the hash table, where the entry is used to index the first data block; and then according to the determined line number, determining whether the written box id in each table entry of the line is the same as the identifier of the first data block, and if the box id in a certain table entry is the same as the identifier of the first data block, determining that the table entry is the first table entry.

The detection unit acquires the first data block from the at least one memory block according to the address information of the at least one memory block written in the first table entry; the detection unit can send the acquired first data block to the calculation unit and instruct the calculation unit to process the first data block; the computing unit processes the first data block that needs to be processed, and stores the processed first data block into the at least one memory block, that is, the first data block in the at least one memory block is replaced by the processed first data block, which may also be understood as covering the first data block before processing with the processed first data block.

The computing unit includes a plurality of computing engines (PEs), and the PEs may include a vector processing unit, a scalar processing unit, or a matrix processing unit. The specific design may be customized according to the design requirements of different chips or devices.

Optionally, after the detecting unit obtains the first data block to be processed, the detecting unit may temporarily cache the first data block in an internal cache (inner buffer) of the data processing chip or device, the calculating unit may obtain the first data block to be processed from the internal cache, and after the first data block is processed, the calculating unit may store the processed first data block in the at least one memory block according to the address of the at least one memory block.

It should be understood that the internal cache area in the data processing chip or device may be divided into three different types of cache areas according to different functions, for example, a structure cache area, an address cache area and a data cache area, wherein the structure cache area may be used for caching the hash table, the address cache area may be used for caching the address information of the memory block obtained from the hash table, and the data cache area may be used for caching the data block obtained from the storage unit. These caches are only functionally distinct, and may be the same buffer or may be divided into a plurality of independent buffers in physical implementation.

In a possible implementation manner, the detection unit may also process the first data block, i.e. the detection unit may also implement the functions of the above-mentioned calculation unit without the calculation unit. This is not limited in this application.

Optionally, if the detecting unit receives a fourth message indicating to release or delete the first data block, where the fourth message includes an identifier of the first data block, the detecting unit may determine, according to the identifier of the first data block, a first entry used for indexing the first data block in the hash table. It should be understood that the first message/the second message/the third message/the fourth message received by the detection unit in the embodiment of the present application may be sent by a CPU of a data processing apparatus or a chip.

The detection unit modifies the first indication information in the first table entry into second indication information, where the second indication information is used to indicate that at least one memory block is not occupied, or the detection unit deletes all written information in the first table entry, so that the first table entry becomes an empty table entry. The detection unit sends a second request message to the storage controller, where the second request message is used to request the storage controller to mark at least one memory block as unoccupied. And the storage controller marks the at least one memory block in the storage unit corresponding to the storage controller as unoccupied according to the second request message. Optionally, the detecting unit may also delete the first data block stored in the at least one memory block. It should be understood that the second indication information may be a v signal in the table entry, and when v is equal to 0, it may indicate that the corresponding memory chunk is not occupied.

Based on the scheme, the memory blocks in the storage unit can be multiplexed along with the change of the three-dimensional scene data.

An embodiment of the present application provides a data processing apparatus, and as shown in fig. 7, a schematic block diagram of a data processing apparatus 700 according to an embodiment of the present application is shown, where the data processing apparatus 700 may implement the method of data processing in fig. 6. The data processing apparatus 700 includes: a detection unit 710 for:

determining a first table entry used for indexing a first data block in a hash table according to a received first message, wherein the first message is used for indicating to store the first data block, the first message comprises an identifier of the first data block and the length of the first data block, and the first table entry is unoccupied;

determining address information of at least one memory block for storing the first data block according to the length of the first data block, where the length of data storable in the at least one memory block is greater than or equal to the length of the first data block, and the length of data storable in each memory block is fixed;

storing the first data block into the at least one memory block according to the address information of the at least one memory block;

writing the address information of the at least one memory block, the identifier of the first data block, and first indication information into the first table entry, so as to obtain the first data block from the at least one memory block, where the first indication information is used to indicate that the at least one memory block is occupied.

Optionally, the detecting unit 710 is specifically configured to: determining a row number of a table entry used for indexing the first data block in the hash table according to the identifier of the first data block; and determining the first table entry which is not occupied from the hash table according to the row number.

Optionally, the detecting unit 710 is specifically configured to: determining the row number based on a hash function and an identification of the first block, the hash function comprising:

f(k)＝kmodm

Optionally, the detecting unit 710 is specifically configured to: receiving first information sent by a storage controller, wherein the first information comprises address information of a plurality of unoccupied memory blocks in a storage unit corresponding to the storage controller; and determining, according to the length of the first data block, address information of the at least one memory block used for storing the first data block from the address information of the plurality of unoccupied memory blocks.

Optionally, the detecting unit 710 is specifically configured to: sending a first request message to a storage controller, where the first request message is used to obtain address information of the at least one memory block storing the first data block, and the first request message includes a length of the first data block, where the storage controller is used to determine, according to the length of the first data block, the address information of the at least one memory block that is not occupied in a storage unit corresponding to the storage controller; receiving a second response message sent by the storage controller, where the second response message includes address information of the at least one unoccupied memory block.

Optionally, the address information of the at least one memory block includes: starting address information of a first memory block of the at least one memory block and the number of the memory blocks, where addresses of each memory block of the at least one memory block are consecutive.

Optionally, the detecting unit 710 is specifically configured to: writing the start address information of a first memory block of the at least one memory block, the number of the memory blocks, the identifier of the first data block, and the first indication information into the first table entry.

Optionally, the detecting unit 710 is further configured to: determining the first table entry used for indexing the first data block in the hash table according to a received second message, wherein the second message is used for indicating to acquire the first data block, and the second message comprises an identifier of the first data block; and acquiring the first data block from the at least one memory block according to the address information of the at least one memory block written in the first table entry.

Optionally, the apparatus further includes a computing unit 720; the detecting unit 710 is further configured to determine, according to a received third message, the first entry used for indexing the first data block in the hash table, where the third message is used to instruct to process the first data block, and the third message includes an identifier of the first data block;

the detecting unit 710 is further configured to obtain the first data block from the at least one memory block according to the address information of the at least one memory block written in the first table entry;

the calculating unit 720 is configured to process the first data block acquired by the detecting unit;

the calculating unit 720 is further configured to replace the first data block in the at least one memory block with the processed first data block.

Optionally, the detecting unit 710 is further configured to: determining the first table entry used for indexing the first data block in the hash table according to a received fourth message, where the fourth message is used for indicating to release or delete the first data block, and the fourth message includes an identifier of the first data block; modifying the first indication information in the first table entry into second indication information, where the second indication information is used to indicate that the at least one memory block is not occupied; sending a second request message to the storage controller, where the second request message is used to request the storage controller to mark the at least one memory block as unoccupied.

An embodiment of the present application provides a data processing apparatus, and as shown in fig. 8, a schematic block diagram of a data processing apparatus 800 according to an embodiment of the present application is shown. The data processing apparatus 800 comprises: a detection unit 810, a calculation unit 820, a structure buffer 830, an address buffer 840, a data buffer 850, and a storage unit 860.

The detecting unit 810 can implement the functions that the detecting unit can implement in the method embodiment of data processing of fig. 7; the computing unit 820 may implement the functions that may be implemented by the computing unit in the method embodiment of data processing of fig. 7; the structure cache region 830 may be used for caching the hash table in the method embodiment of data processing of fig. 7; the address cache region 840 may be configured to cache address information of the memory block obtained from the hash table; the data buffer 850 may be used to buffer data blocks retrieved from the storage unit 860; the storage unit 860 includes a storage controller and a plurality of memory blocks, and is used for storing data blocks to be stored.

It should be understood that the storage unit 860 may be a component module of the data processing apparatus 800, or may be independent of the data processing apparatus 800.

In the embodiment of the present application, another data processing apparatus is provided, and as shown in fig. 9, a schematic block diagram of a data processing apparatus 900 according to the embodiment of the present application is shown. The data processing apparatus 900 includes:

the data processing apparatus 900 includes: at least one processor 910, the processor is connected to a memory 920, the memory 920 is used for storing a computer program, and the processor 910 is used for executing the computer program stored in the memory 920, so as to enable the apparatus to execute the method of data processing in any possible implementation manner in the embodiment of the present application.

The processor 910 may be an integrated circuit chip having signal processing capability. In implementation, the steps of the above method embodiments may be performed by integrated logic circuits of hardware in a processor or instructions in the form of software. The processor may be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off the shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, or discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules may be located in ram, flash, rom, prom, or eprom, registers, etc. as is well known in the art. The storage medium is located in a memory, and a processor reads information in the memory and completes the steps of the method in combination with hardware of the processor.

The embodiment of the present application further provides a data processing chip, which includes a processor, where the processor is connected to a memory, the memory is used for storing a computer program, and the processor is used for executing the computer program stored in the memory, so that the chip executes the method in the foregoing method embodiment.

The embodiment of the present application also provides a computer-readable storage medium on which a computer program for implementing the method in the above method embodiment is stored. When the computer program runs on a computer, the computer is enabled to implement the method in the above-described method embodiments.

Embodiments of the present application further provide a computer program product, which includes computer program code, when the computer program code runs on a computer, the method in the above method embodiments is executed.

In addition, the term "and/or" in the present application is only one kind of association relationship describing the associated object, and means that three kinds of relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter associated objects are in an "or" relationship; the term "in this application, may mean" one "and" two or more ", for example, in a, B and C, may mean: a alone, B alone, C alone, A and B together, A and C together, C and B together, and A and B together and C and B together, seven cases exist.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one type of logical functional division, and other divisions may be realized in practice, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the mutual coupling or direct coupling shown or discussed may be indirect coupling through some interfaces, devices or units, and may be electrical, mechanical or other forms.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a portable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, an optical disk, or other various media capable of storing program codes.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method of data processing, comprising:

2. The method of claim 1, wherein determining a first entry in a hash table for indexing a first data block based on the received first message comprises:

determining a row number of a table entry used for indexing the first data block in the hash table according to the identifier of the first data block;

and determining the first table entry which is not occupied from the hash table according to the row number.

3. The method of claim 2, wherein the determining, according to the identifier of the first data block, a row number of an entry in the hash table for indexing the first data block comprises:

determining the row number based on a hash function and an identification of the first block, the hash function comprising:

f(k)＝kmodm

4. The method according to any one of claims 1 to 3, wherein the determining, according to the length of the first data block, address information of at least one memory block used for storing the first data block includes:

receiving first information sent by a storage controller, wherein the first information comprises address information of a plurality of unoccupied memory blocks in a storage unit corresponding to the storage controller;

according to the length of the first data block, determining address information of the at least one memory block used for storing the first data block from the address information of the plurality of unoccupied memory blocks.

5. The method according to any one of claims 1 to 3, wherein the determining, according to the length of the first data block, address information of at least one memory block used for storing the first data block includes:

sending a first request message to a storage controller, where the first request message is used to obtain address information of the at least one memory block storing the first data block, and the first request message includes a length of the first data block, where the storage controller is used to determine, according to the length of the first data block, the address information of the at least one memory block that is not occupied in a storage unit corresponding to the storage controller;

receiving a second response message sent by the storage controller, where the second response message includes address information of the at least one unoccupied memory block.

6. The method according to any of claims 1 to 5, wherein the address information of the at least one memory chunk comprises:

starting address information of a first memory block of the at least one memory block and the number of the memory blocks, where addresses of each memory block of the at least one memory block are consecutive.

7. The method according to claim 6, wherein the writing the address information of the at least one memory block, the identifier of the first data block, and the first indication information into the first entry includes:

writing the start address information of a first memory block of the at least one memory block, the number of the memory blocks, the identifier of the first data block, and the first indication information into the first table entry.

8. The method according to any one of claims 1 to 7, further comprising:

determining the first table entry used for indexing the first data block in the hash table according to a received second message, wherein the second message is used for indicating to acquire the first data block, and the second message comprises an identifier of the first data block;

and acquiring the first data block from the at least one memory block according to the address information of the at least one memory block written in the first table entry.

9. The method according to any one of claims 1 to 8, further comprising:

determining the first table entry used for indexing the first data block in the hash table according to a received third message, where the third message is used to instruct to process the first data block, and the third message includes an identifier of the first data block;

acquiring the first data block from the at least one memory block according to the address information of the at least one memory block written in the first table entry;

processing the acquired first data block;

replacing the first data block in the at least one memory block with the processed first data block.

10. The method according to any one of claims 4 to 7, further comprising:

determining the first table entry used for indexing the first data block in the hash table according to a received fourth message, where the fourth message is used for indicating to release or delete the first data block, and the fourth message includes an identifier of the first data block;

modifying the first indication information in the first table entry into second indication information, where the second indication information is used to indicate that the at least one memory block is not occupied;

sending a second request message to the storage controller, where the second request message is used to request the storage controller to mark the at least one memory block as unoccupied.

11. A data processing apparatus, comprising: a detection unit to:

12. The apparatus according to claim 11, wherein the detection unit is specifically configured to:

and determining the unoccupied first table entry from the hash table according to the row number.

13. The apparatus according to claim 12, wherein the detection unit is specifically configured to:

f(k)＝kmodm

14. The device according to any one of claims 11 to 13, characterized in that the detection unit is specifically configured to:

15. The device according to any one of claims 11 to 13, characterized in that the detection unit is specifically configured to:

16. The apparatus according to any of claims 11 to 15, wherein the address information of the at least one memory chunk comprises:

starting address information of a first memory block in the at least one memory block and the number of the memory blocks, where addresses of each memory block in the at least one memory block are consecutive.

17. The apparatus according to claim 16, wherein the detection unit is specifically configured to:

18. The apparatus according to any one of claims 11 to 17, wherein the detection unit is further configured to:

19. The apparatus according to any one of claims 11 to 18, further comprising a computing unit;

the detecting unit is further configured to determine, according to a received third message, the first entry used for indexing the first data block in the hash table, where the third message is used to instruct to process the first data block, and the third message includes an identifier of the first data block;

the detecting unit is further configured to obtain the first data block from the at least one memory block according to the address information of the at least one memory block written in the first table entry;

the calculation unit is configured to process the first data block acquired by the detection unit;

the computing unit is further configured to replace the first data block in the at least one memory block with the processed first data block.

20. The apparatus according to any one of claims 14 to 17, wherein the detection unit is further configured to:

21. A data processing apparatus, comprising: at least one processor configured to perform the method of any one of claims 1 to 10.

22. A data processing chip comprising a processor coupled to a memory for storing a computer program, the processor being configured to execute the computer program stored in the memory to cause the chip to perform the method of any one of claims 1 to 10.

23. A computer-readable storage medium, having stored thereon a computer program which, when executed by data processing apparatus, causes the data processing apparatus to perform the method of any one of claims 1 to 10.

24. A computer program product comprising a computer program which, when executed, causes the method of any one of claims 1 to 10 to be carried out.