CN110941399A - Data processing method and device and electronic equipment - Google Patents
Data processing method and device and electronic equipment Download PDFInfo
- Publication number
- CN110941399A CN110941399A CN201911239158.3A CN201911239158A CN110941399A CN 110941399 A CN110941399 A CN 110941399A CN 201911239158 A CN201911239158 A CN 201911239158A CN 110941399 A CN110941399 A CN 110941399A
- Authority
- CN
- China
- Prior art keywords
- data
- processed
- block
- size
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Design And Manufacture Of Integrated Circuits (AREA)
Abstract
The invention provides a data processing method, a data processing device and electronic equipment, wherein the method comprises the following steps: acquiring data to be processed and the maximum data size which can be supported by hardware equipment for processing the data to be processed; if the size of the data to be processed is larger than the maximum data size, determining a blocking parameter according to the size of the data to be processed and the maximum data size; wherein the blocking parameter comprises: the block direction and the block size; and according to the blocking parameters, carrying out blocking processing on the data to be processed to obtain block data of the data to be processed, and processing the block data through hardware equipment. According to the method and the device, the block size is dynamically determined according to the size of the data to be processed and the maximum data size, namely, the sizes of different data to be processed correspond to different block sizes, and compared with the traditional block dividing mode with a fixed size, the number of blocks obtained by the mode is small, so that the efficiency of subsequent data processing is improved.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a data processing method and apparatus, and an electronic device.
Background
In data processing, if data to be processed exceeds the video memory support range of the hardware device, the data needs to be preprocessed so that the data meets the video memory support range. In the related art, the preprocessing mode is usually to scale data equally, but the mode easily loses the detailed information of the data, and cannot achieve the expected processing effect; based on this, during the preprocessing, the data can be further processed in a blocking mode according to a preset fixed size, so that a plurality of data blocks with fixed sizes are obtained.
Disclosure of Invention
The invention aims to provide a data processing method, a data processing device and electronic equipment, which are used for reducing the number of data blocks so as to improve the subsequent data processing efficiency.
In a first aspect, an embodiment of the present invention provides a data processing method, where the method includes: acquiring data to be processed and the maximum data size which can be supported by hardware equipment for processing the data to be processed; if the size of the data to be processed is larger than the maximum data size, determining a blocking parameter according to the size of the data to be processed and the maximum data size; wherein the blocking parameter comprises: the block direction and the block size; and according to the blocking parameters, carrying out blocking processing on the data to be processed to obtain block data of the data to be processed, and processing the block data through hardware equipment.
In an optional embodiment, the blocking direction includes a height direction of the data to be processed; the block size comprises a block width and a block height; the block width includes the width of the data to be processed.
In an optional embodiment, the step of determining the blocking parameter according to the size of the data to be processed and the maximum data size includes: and determining the height of the block according to the maximum data size and the width of the block.
In an alternative embodiment, the step of determining the partition height according to the maximum data size and the partition width includes: a first ratio of the maximum data size to the tile width is determined as the tile height.
In an optional embodiment, the step of performing block processing on the data to be processed according to the block parameter to obtain block data of the data to be processed includes: determining the block number of the data to be processed according to the block height; and according to the number of the blocks, carrying out block processing on the data to be processed along the height direction of the data to be processed to obtain a plurality of block data of the data to be processed.
In an optional embodiment, the step of determining the number of blocks of the data to be processed according to the block height includes: calculating a second ratio of the height of the data to be processed to the height of the blocks; and rounding the second ratio to obtain the block number of the data to be processed.
In an optional embodiment, after the step of obtaining a plurality of block data of the data to be processed, the method further includes: and respectively filling the multiple block data of the data to be processed based on the receptive field radius of the model file for processing the data to be processed to obtain the multiple block data of the processed data to be processed.
In an optional implementation, after the step of obtaining block data of the data to be processed, the method further includes: inputting each block data into a model file for processing the data to be processed one by one according to the arrangement sequence of the block data of the data to be processed to obtain a processing result corresponding to each block data; and storing the processing result corresponding to each block data according to the arrangement sequence of the block data.
In an optional embodiment, the maximum data size supportable by the hardware device that processes the data to be processed is obtained by: acquiring the total amount of video memory of hardware equipment for processing data to be processed and the consumption of the video memory consumed by the hardware equipment for operating a preset model file; wherein, the model file is used for processing the data to be processed on the hardware equipment; and calculating the difference between the total amount of the display memory and the consumption of the display memory, and determining the maximum data size which can be supported by the hardware equipment according to the difference.
In a second aspect, an embodiment of the present invention provides a data processing apparatus, including: the data acquisition module is used for acquiring data to be processed and the maximum data size which can be supported by hardware equipment for processing the data to be processed; the parameter determining module is used for determining a blocking parameter according to the size of the data to be processed and the maximum data size if the size of the data to be processed is larger than the maximum data size; wherein the blocking parameter comprises: the block direction and the block size; and the data blocking module is used for carrying out blocking processing on the data to be processed according to the blocking parameters to obtain the blocking data of the data to be processed so as to process the blocking data through the hardware equipment.
In a third aspect, an embodiment of the present invention provides an electronic device, which includes a processor and a memory, where the memory stores machine executable instructions capable of being executed by the processor, and the processor executes the machine executable instructions to implement the data processing method described in any one of the foregoing embodiments.
In a fourth aspect, embodiments of the invention provide a machine-readable storage medium storing machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement a data processing method as described in any of the preceding embodiments.
The embodiment of the invention has the following beneficial effects:
the invention provides a data processing method, a data processing device and electronic equipment.A data to be processed is firstly obtained, and the maximum data size which can be supported by hardware equipment for processing the data to be processed is also obtained; if the size of the data to be processed is larger than the maximum data size, determining a blocking parameter according to the size of the data to be processed and the maximum data size; the blocking parameters include: the block direction and the block size; and then, carrying out blocking processing on the data to be processed according to the blocking parameters to obtain block data of the data to be processed, so as to process the block data through hardware equipment. According to the method and the device, the block size is dynamically determined according to the size of the data to be processed and the maximum data size, namely, the sizes of different data to be processed correspond to different block sizes, and compared with the traditional block dividing mode with a fixed size, the number of blocks obtained by the mode is small, so that the efficiency of subsequent data processing is improved.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention as set forth above.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a flow chart of another data processing method according to an embodiment of the present invention;
FIG. 3 is a flow chart of another data processing method according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of block data according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the related art, there are two general ways to pre-process data to be processed that exceed the video memory support range of the hardware device:
and in the mode 1, the data to be processed is scaled according to equal proportion. However, the method easily loses the detailed information of the data, and cannot achieve the expected processing effect.
And 2, performing block processing on the data to be processed according to a preset fixed size to obtain a plurality of data blocks with fixed sizes. For example, the size of the data to be processed is 1920 × 1080, the fixed size of the partition is 800 × 800, and the number of the partitions is 6. The method can meet the range supported by the video memory and retain the detail information of the data, but the number of the data blocks with fixed sizes is large, so that the subsequent data processing efficiency is low.
Based on the above problems, embodiments of the present invention provide a data processing method, an apparatus, and an electronic device, which can be applied to scenes such as image processing and video processing, especially a blocking processing scene of an image. To facilitate understanding of the present embodiment, a detailed description is first provided for a data processing method disclosed in the present embodiment, and as shown in fig. 1, the method includes the following steps:
step S102, acquiring data to be processed and the maximum data size supportable by the hardware equipment processing the data to be processed.
The data to be processed may be data input by a user or acquired data collected by an external device, and the data to be processed may be an image, video data, or data such as a digital table. The hardware device may be a device for processing data to be processed, and the hardware device may be computer hardware, which generally refers to a generic term for various physical devices in a computer system, which are composed of electronic, mechanical, and optoelectronic components. Usually, other execution files are run on the hardware device, and the execution files need to occupy a certain memory capacity, so the maximum data size supportable by the hardware device can be obtained by the occupied capacity of the running execution files and the maximum capacity of the hardware device, and can also be obtained according to an actual test value.
The maximum data size may comprise a product of a maximum width and a maximum height supportable by the hardware device; when the data to be processed is image or video data, the maximum size may also be referred to as resolution, that is, the maximum width corresponds to the number of pixels in the horizontal direction of the resolution, and the maximum height corresponds to the number of pixels in the vertical direction of the resolution.
Step S104, if the size of the data to be processed is larger than the maximum data size, determining a blocking parameter according to the size of the data to be processed and the maximum data size; wherein the blocking parameter comprises: the block direction and the block size.
When it is detected that the size of the data to be processed is larger than the maximum data size supportable by the hardware device, the data to be processed needs to be subjected to blocking processing, where the size of the data to be processed includes a width and a height, for example, when the size of the data to be processed is 1920 × 1080 and the maximum data size of the hardware device is 1280 × 720, the data to be processed needs to be subjected to blocking processing, so that the blocked data meets the maximum data size.
The blocking parameters may be determined according to the size of the data to be processed and the maximum data size, and it may also be understood that, for the same maximum data size and the data to be processed with different sizes, the corresponding blocking parameters are different. The blocking parameters comprise a blocking direction and a blocking size, and during specific implementation, the blocking direction can be used for blocking from the height direction according to the structure of data to be processed, and can also be used for blocking from the width direction; the block size is dynamically variable following the size of the data to be processed, and is not a fixed value, for example, the width of the block size may be selected according to the width of the size of the data to be processed, or the height of the block size may be selected according to the height of the size of the data to be processed.
And step S106, according to the blocking parameters, blocking the data to be processed to obtain block data of the data to be processed, and processing the block data through hardware equipment.
After the blocking parameter is determined, the data to be processed may be blocked according to the blocking parameter to obtain blocked data, that is, the data to be processed may be blocked according to the blocking size along the blocking direction to obtain blocked data. The size of the block data meets the maximum data size of the hardware equipment, and the block data can be correspondingly processed in the hardware equipment, namely, the block data is processed by utilizing an execution file running in the hardware equipment, and the processing can be inference operation, data analysis, data transmission and the like.
The data processing method provided by the embodiment of the invention comprises the steps of firstly obtaining data to be processed and the maximum data size which can be supported by hardware equipment for processing the data to be processed; if the size of the data to be processed is larger than the maximum data size, determining a blocking parameter according to the size of the data to be processed and the maximum data size; the blocking parameters include: the block direction and the block size; and then, carrying out blocking processing on the data to be processed according to the blocking parameters to obtain block data of the data to be processed, so as to process the block data through hardware equipment. According to the method and the device, the block size is dynamically determined according to the size of the data to be processed and the maximum data size, namely, the sizes of different data to be processed correspond to different block sizes, and compared with the traditional block dividing mode with a fixed size, the number of blocks obtained by the mode is small, so that the efficiency of subsequent data processing is improved.
The embodiment of the invention also provides another data processing method, which is realized on the basis of the method of the embodiment; the method mainly describes a specific process of determining a blocking parameter according to the size of the data to be processed and the maximum data size (specifically realized by the following step S204), and a specific process of blocking the data to be processed according to the blocking parameter (specifically realized by the following steps S210-S212); as shown in fig. 2, the method comprises the steps of:
step S202, acquiring data to be processed and a maximum data size supportable by a hardware device processing the data to be processed.
Step S204, judging whether the size of the data to be processed is larger than the maximum data size; if not, executing the following step S206; if so, go to step S208.
Step S206, carrying out subsequent processing on the data to be processed through hardware equipment; and (6) ending.
When the size of the data to be processed is smaller than or equal to the maximum data size, the data to be processed can be directly subjected to subsequent processing through corresponding hardware equipment, and when the size of the data to be processed is larger than the maximum data size, the data to be processed needs to be subjected to blocking processing, so that the blocked data to be processed meets the maximum data size which can be supported by the hardware equipment.
Step S208, determining the blocking direction and the blocking size in the blocking parameters according to the size of the data to be processed and the maximum data size; the block direction comprises the height direction of the data to be processed, the block size comprises a block width and a block height, and the block width comprises the width of the data to be processed.
When determining the blocking parameters, the height direction of the data to be processed may be determined as the blocking direction; the width of the size of the data to be processed can be determined as the block width, namely the block width can be dynamically changed along with the width of the data to be processed; the partition height may be determined according to the maximum data size and the partition width, and in a specific implementation, the first ratio of the maximum data size to the partition width is determined as the partition height.
The first ratio may be a ratio of a maximum data size to a partition width, for example, when the size of the data to be processed is W × H, the width thereof is W, the height thereof is H, and the maximum data size is W × H, the partition width is determined as W, and the first ratio is (W × H)/W, that is, the partition height is (W × H)/W.
And step S210, determining the block number of the data to be processed according to the block height.
According to the height of the data to be processed and the block height, the number of blocks can be determined, and in a specific implementation, the step S210 can be implemented by the following steps 10 to 11:
and step 10, calculating a second ratio of the height of the data to be processed to the height of the blocks. The second ratio is a ratio of the height of the data to be processed to the height of the partitions, for example, if the height of the data to be processed is H and the height of the partitions is Hs, then the second ratio is H/Hs.
And step 11, rounding the second ratio to obtain the block number of the data to be processed.
When the second ratio is an integer, determining the second ratio as the block number; if the second ratio is not an integer, rounding up the second ratio is required, and a rounding up mode can be adopted. For example, when the size of the data to be processed is 1960 × 1080 and the maximum data size is 1280 × 720, the partition width is 1960, the partition height is (1280 × 720)/1960 ═ 480, the second ratio is 1080/480 ═ 2.25, and the number of partitions is 3 after rounding up.
Step S212, according to the number of the blocks, the data to be processed is processed in a blocking mode along the height direction of the data to be processed, and a plurality of block data of the data to be processed are obtained.
The above-mentioned direction along the height of the data to be processed may be understood as performing block processing on the data to be processed from top to bottom or from bottom to top, for example, dividing the data to be processed into block data of the number of blocks according to the block size from top to bottom.
In the process of blocking data to be processed, each block data reads data from the data to be processed line by line, and the blocking width changes along with the width of the data to be processed, namely, after one line of data is read, the next line of data is read (also can be understood as reading another page after reading one page), so that the read addresses can be ensured to be continuous, the continuity of a memory is ensured, and the phenomenon of page skipping when fixed-size blocking is adopted can be prevented. Therefore, the way that the width of the block changes along with the width of the data to be processed can also improve the processing efficiency of the data.
Step S214, processing the block data by a hardware device.
The data processing method determines the width of data to be processed as the width of a block, and determines the height of the block according to the width of the data to be processed and the maximum data size of hardware equipment; then determining the number of the blocks based on the height of the blocks and the height of the data to be processed; and performing blocking processing on the data to be processed along the height direction of the data to be processed based on the block data quantity to obtain a plurality of block data of the data to be processed. The width of the blocks can dynamically change along with the width of the data to be processed in the mode, compared with the traditional block mode with fixed size, the number of the blocks obtained in the mode is small, and the phenomenon of page skipping can be avoided in the data reading process, so that the efficiency of data processing is improved.
The embodiment of the invention also provides another data processing method, which is realized on the basis of the method of the embodiment; the method mainly describes a specific process of acquiring the maximum data size supportable by a hardware device for processing data to be processed (specifically, realized by the following steps S302-S304), and a specific process of performing filling processing on a plurality of block data of the data to be processed (specifically, realized by the following step S312); as shown in fig. 3, the method comprises the steps of:
step S302, acquiring the total amount of video memory of the hardware equipment for processing the data to be processed and the consumption of the video memory consumed by the hardware equipment for operating a preset model file; wherein the model file is used for processing the data to be processed on the hardware device.
The hardware device for processing the data to be processed is usually provided with a display card, and the total amount of the display memory may be the capacity of the display card in the case that the display card is occupied. The model file generally refers to a PB (Protocol Buffer) file frozen after the deep learning network is trained, the PB file generally has language independence, can run independently, has a closed serialized format, can be parsed by any language, and is generally a tensrflow model allowing other languages and deep learning frameworks to read, continue training, and migrate. In specific implementation, the trained deep learning network in the model file can process data to be processed, and the process can also be inference operation.
The video memory consumption is generally the sum of the capacity occupied by running the model file and the capacity occupied by the inference operation. The total amount of memory and the amount of memory consumption can be obtained by inquiring parameters of hardware devices and parameters of model files.
Step S304, calculating a difference between the total amount of the display memory and the consumption amount of the display memory, and determining the maximum data size supportable by the hardware device according to the difference.
In a specific implementation, if the total amount of the memory is M and the memory consumption is M, the maximum data size supportable by the hardware device is M-M.
Step S306, if the size of the acquired data to be processed is larger than the maximum data size, determining the blocking direction, the blocking width and the blocking height according to the size of the data to be processed and the maximum data size; the block direction comprises the height direction of the data to be processed, the block size comprises a block width and a block height, and the block width is the width of the data to be processed.
The block height may be determined according to the maximum data size and the block width, that is, the maximum data size is divided by the block width to obtain the block height.
And step S308, determining the block number of the data to be processed according to the block height.
Step S310, according to the number of the blocks, the data to be processed is processed in a blocking mode along the height direction of the data to be processed, and a plurality of block data of the data to be processed are obtained.
Step S312, based on the radius of the reception field of the model file for processing the data to be processed, filling the multiple block data of the data to be processed, so as to obtain multiple block data of the processed data to be processed.
In order to ensure that the boundary of the block data is correct in the block processing process, padding (padding) data needs to be used to protect the block data, and the width of the padding data may be the radius of the receptive field. The data to be processed is usually an image or a video frame, and the Receptive Field (Receptive Field) may be a region size of a mapping of a pixel point on a feature map (feature map) output by each layer of the deep learning network on an input image. The receptive field radius is usually determined by a deep learning network in the model file, that is, the receptive field radius is determined while the deep learning network determines.
Fig. 4 illustrates the filling process of block data in detail, taking the number of blocks as 4 as an example. In the process of filling the tile data according to the reception field radius, the height of the data to be processed in the tile data may be the height of the tile minus the reception field radius (e.g., the height of the effective area in the first tile data in fig. 4), or may be the height of the tile minus twice the reception field radius (e.g., the height of the effective area in the second and third tile data in fig. 4). The data in the valid area and the padding area (corresponding to the pad area and the invalid area in fig. 4) in fig. 4 are both data in the data to be processed, where the data in the padding area is corresponding to redundant data, and the boundary of the data can be determined to be correct in the data recovery process.
Step S314, according to the arrangement order of the block data of the data to be processed, inputting each block data into the model file for processing the data to be processed one by one, and obtaining a processing result corresponding to each block data.
When the blocking direction is the height direction of the data to be processed, the sequence of the block data is usually arranged from top to bottom to form a first block data, a second block data and the like, and the first block data and the second block data are sequentially input into a model file of the data to be processed for processing to obtain a processing result corresponding to each data block; if the processing result includes the processing result of the filled region, the processing result corresponding to the data of the valid region is usually extracted to obtain the final processing result.
Step S316, storing the processing result corresponding to each block data according to the arrangement sequence of the block data.
In a specific implementation, the result corresponding to each block data may be spliced into data with the same size as the data to be processed according to the arrangement order of the block data, that is, the processing result corresponding to the filling area is removed according to the arrangement order.
According to the data processing method, the boundary of the data to be processed is protected by filling the block data through the receptive field radius, the height of the block is kept consistent with that of the data to be processed, when the data is blocked, not only is the detail information of the data to be processed ensured, but also the block data amount is effectively reduced, so that the time of subsequent data processing is saved, and the data processing efficiency is improved.
Corresponding to the embodiment of the data processing method, an embodiment of the present invention further provides a data processing apparatus, as shown in fig. 5, the apparatus including:
the data obtaining module 50 is configured to obtain data to be processed and a maximum data size supportable by a hardware device that processes the data to be processed.
A parameter determining module 51, configured to determine a blocking parameter according to the size of the data to be processed and the maximum data size if the size of the data to be processed is larger than the maximum data size; wherein the blocking parameter comprises: the block direction and the block size.
And the data blocking module 52 is configured to perform blocking processing on the data to be processed according to the blocking parameters to obtain block data of the data to be processed, so as to perform subsequent processing on the block data through the hardware device.
The data processing device firstly acquires data to be processed and the maximum data size which can be supported by hardware equipment for processing the data to be processed; if the size of the data to be processed is larger than the maximum data size, determining a blocking parameter according to the size of the data to be processed and the maximum data size; the blocking parameters include: the block direction and the block size; and then, carrying out blocking processing on the data to be processed according to the blocking parameters to obtain block data of the data to be processed, so as to process the block data through hardware equipment. In the method, the block size is dynamically determined according to the size of the data to be processed and the maximum data size, that is, the sizes of different data to be processed correspond to different block sizes.
Specifically, the blocking direction includes a height direction of the data to be processed; the block size comprises a block width and a block height; the block width includes the width of the data to be processed.
Further, the parameter determining module 51 is configured to: and determining the block height according to the maximum data size and the block width.
Further, the parameter determining module 51 is further configured to: a first ratio of the maximum data size to the tile width is determined as the tile height.
Further, the data blocking module 52 is configured to determine the number of blocks of the data to be processed according to the height of the blocks; and according to the number of the blocks, carrying out block processing on the data to be processed along the height direction of the data to be processed to obtain a plurality of block data of the data to be processed.
Specifically, the data blocking module 52 is further configured to: calculating a second ratio of the height of the data to be processed to the height of the blocks; and rounding the second ratio to obtain the block number of the data to be processed.
Further, the apparatus further comprises a filling module for: and respectively filling the multiple block data of the data to be processed based on the receptive field radius of the model file for processing the data to be processed to obtain the multiple block data of the processed data to be processed.
Further, the apparatus further includes a sequence processing module configured to: inputting each block data into a model file for processing the data to be processed one by one according to the arrangement sequence of the block data of the data to be processed to obtain a processing result corresponding to each block data; and storing the processing result corresponding to each block data according to the arrangement sequence of the block data.
Specifically, the apparatus further includes a maximum data size obtaining module, configured to: acquiring the total amount of the video memory of the hardware equipment for processing the data to be processed and the consumption of the video memory consumed by the hardware equipment for operating a preset model file; the model file is used for processing data to be processed on hardware equipment; and calculating the difference between the total amount of the display memory and the consumption of the display memory, and determining the maximum data size which can be supported by the hardware equipment according to the difference.
The data processing apparatus provided in the embodiment of the present invention has the same implementation principle and technical effect as the foregoing method embodiments, and for the sake of brief description, reference may be made to the corresponding contents in the foregoing method embodiments for the parts of the apparatus embodiments that are not mentioned.
An embodiment of the present invention further provides an electronic device, which is shown in fig. 6 and includes a processor 101 and a memory 100, where the memory 100 stores machine executable instructions that can be executed by the processor 101, and the processor 101 executes the machine executable instructions to implement the data processing method.
Further, the electronic device shown in fig. 6 further includes a bus 102 and a communication interface 103, and the processor 101, the communication interface 103, and the memory 100 are connected through the bus 102.
The Memory 100 may include a high-speed Random Access Memory (RAM) and may further include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used. The bus 102 may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 6, but that does not indicate only one bus or one type of bus.
The processor 101 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 101. The Processor 101 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component. The various methods, steps and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100, and completes the steps of the method of the foregoing embodiment in combination with the hardware thereof.
The embodiment of the present invention further provides a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to implement the data processing method.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the apparatus and/or the electronic device described above may refer to corresponding processes in the foregoing method embodiments, and are not described herein again.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (12)
1. A method of data processing, the method comprising:
acquiring data to be processed and the maximum data size which can be supported by hardware equipment for processing the data to be processed;
if the size of the data to be processed is larger than the maximum data size, determining a blocking parameter according to the size of the data to be processed and the maximum data size; wherein the blocking parameters include: the block direction and the block size;
and according to the blocking parameters, blocking the data to be processed to obtain block data of the data to be processed, so that the block data is processed through the hardware equipment.
2. The method of claim 1, wherein the blocking direction comprises a height direction of the data to be processed; the block size comprises a block width and a block height; the block width comprises a width of the data to be processed.
3. The method according to claim 2, wherein the step of determining the blocking parameter according to the size of the data to be processed and the maximum data size comprises: and determining the block height according to the maximum data size and the block width.
4. The method of claim 3, wherein the step of determining the tile height based on the maximum data size and the tile width comprises: determining a first ratio of the maximum data size to the tile width as the tile height.
5. The method according to claim 2, wherein the step of performing block processing on the data to be processed according to the block parameter to obtain block data of the data to be processed comprises:
determining the block number of the data to be processed according to the block height;
and according to the block number, carrying out block processing on the data to be processed along the height direction of the data to be processed to obtain a plurality of block data of the data to be processed.
6. The method of claim 5, wherein the step of determining the number of partitions of the data to be processed according to the partition height comprises:
calculating a second ratio of the height of the data to be processed to the height of the blocks;
and rounding the second ratio to obtain the block number of the data to be processed.
7. The method of claim 5, wherein after the step of obtaining the plurality of blocks of data to be processed, the method further comprises:
and filling the multiple block data of the data to be processed respectively based on the radius of the receptive field of the model file for processing the data to be processed to obtain the multiple block data of the processed data to be processed.
8. The method of claim 1, wherein after the step of obtaining the block data of the data to be processed, the method further comprises:
according to the arrangement sequence of the block data of the data to be processed, inputting each block data into a model file for processing the data to be processed one by one to be processed, and obtaining a processing result corresponding to each block data;
and storing the processing result corresponding to each block data according to the arrangement sequence of the block data.
9. The method according to claim 1, wherein the maximum data size supportable by the hardware device processing the data to be processed is obtained by:
acquiring the total amount of the video memory of the hardware equipment for processing the data to be processed and the consumption of the video memory consumed by the hardware equipment for operating a preset model file; wherein the model file is used for processing the data to be processed on the hardware device;
and calculating a difference value between the total amount of the display memory and the consumption amount of the display memory, and determining the maximum data size which can be supported by the hardware equipment according to the difference value.
10. A data processing apparatus, characterized in that the apparatus comprises:
the data acquisition module is used for acquiring data to be processed and the maximum data size which can be supported by hardware equipment for processing the data to be processed;
a parameter determining module, configured to determine a blocking parameter according to the size of the data to be processed and the maximum data size if the size of the data to be processed is larger than the maximum data size; wherein the blocking parameters include: the block direction and the block size;
and the data blocking module is used for carrying out blocking processing on the data to be processed according to the blocking parameters to obtain block data of the data to be processed so as to process the block data through the hardware equipment.
11. An electronic device comprising a processor and a memory, the memory storing machine executable instructions executable by the processor, the processor executing the machine executable instructions to implement the data processing method of any one of claims 1 to 9.
12. A machine-readable storage medium having stored thereon machine-executable instructions which, when invoked and executed by a processor, cause the processor to implement the data processing method of any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911239158.3A CN110941399A (en) | 2019-12-05 | 2019-12-05 | Data processing method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911239158.3A CN110941399A (en) | 2019-12-05 | 2019-12-05 | Data processing method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110941399A true CN110941399A (en) | 2020-03-31 |
Family
ID=69909904
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911239158.3A Pending CN110941399A (en) | 2019-12-05 | 2019-12-05 | Data processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110941399A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113836068A (en) * | 2021-09-30 | 2021-12-24 | 北京金山云网络技术有限公司 | PCIe display card control method and device, server and electronic equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102663759A (en) * | 2012-04-20 | 2012-09-12 | 中国科学院遥感应用研究所 | Remote-sensing image rapid-processing method |
| CN107909026A (en) * | 2016-11-30 | 2018-04-13 | 深圳奥瞳科技有限责任公司 | Age and gender assessment based on the small-scale convolutional neural networks of embedded system |
| CN109886866A (en) * | 2019-01-15 | 2019-06-14 | 珠海亿智电子科技有限公司 | A kind of adaptive affine transformation processing method and processing device |
| CN110390679A (en) * | 2019-07-03 | 2019-10-29 | 上海联影智能医疗科技有限公司 | Image processing method, computer equipment and readable storage medium storing program for executing |
-
2019
- 2019-12-05 CN CN201911239158.3A patent/CN110941399A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102663759A (en) * | 2012-04-20 | 2012-09-12 | 中国科学院遥感应用研究所 | Remote-sensing image rapid-processing method |
| CN107909026A (en) * | 2016-11-30 | 2018-04-13 | 深圳奥瞳科技有限责任公司 | Age and gender assessment based on the small-scale convolutional neural networks of embedded system |
| CN109886866A (en) * | 2019-01-15 | 2019-06-14 | 珠海亿智电子科技有限公司 | A kind of adaptive affine transformation processing method and processing device |
| CN110390679A (en) * | 2019-07-03 | 2019-10-29 | 上海联影智能医疗科技有限公司 | Image processing method, computer equipment and readable storage medium storing program for executing |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113836068A (en) * | 2021-09-30 | 2021-12-24 | 北京金山云网络技术有限公司 | PCIe display card control method and device, server and electronic equipment |
| CN113836068B (en) * | 2021-09-30 | 2024-04-26 | 北京金山云网络技术有限公司 | PCIe display card control method and device, server and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109783181B (en) | Screen adaptive display method, electronic device and computer storage medium | |
| JP6871314B2 (en) | Object detection method, device and storage medium | |
| CN109729298B (en) | Image processing method and image processing apparatus | |
| CN106611401B (en) | Method and device for storing image in texture memory | |
| CN113096125B (en) | Method, device and storage medium for determining concentration of low-density lipoprotein reagent | |
| JP7026165B2 (en) | Text recognition method and text recognition device, electronic equipment, storage medium | |
| CN111859865A (en) | Method, device, terminal and medium for converting PDF document | |
| CN110941399A (en) | Data processing method and device and electronic equipment | |
| CN110187816B (en) | Automatic page turning method for cartoon type electronic book, computing device and storage medium | |
| CN102999885A (en) | Method and device for determining average brightness by Retinex video enhancement algorithm | |
| CN111476858B (en) | WebGL-based 2d engine rendering method, device and equipment | |
| CN110516793B (en) | Pooling processing method and device and storage medium | |
| CN107977923B (en) | Image processing method, image processing device, electronic equipment and computer readable storage medium | |
| CN114998377B (en) | Image edge detection method and device, electronic equipment and storage medium | |
| CN116185545A (en) | Page rendering method and device | |
| CN109388726A (en) | A kind of image display method, device, equipment and computer-readable medium | |
| CN115270710A (en) | Typesetting method and typesetting device | |
| CN112418098A (en) | Training method of video structured model and related equipment | |
| CN103838708B (en) | Electronic typesetting method and electronic typesetting device | |
| CN106610974B (en) | Webpage photo wall generation method and device | |
| CN108629350B (en) | Method and device for identifying similarity relation between pictures | |
| CN110399581B (en) | Page rendering method, equipment, storage medium and device based on machine learning | |
| CN115905135B (en) | Multi-frame 3D file processing method and device | |
| CN111914512B (en) | Display method and reading equipment based on electronic book horizontal screen double-turning mode | |
| CN111526263B (en) | Image processing method, device and computer system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |