WO2022000456A1 - 图像处理方法、集成电路、装置及设备 - Google Patents

图像处理方法、集成电路、装置及设备 Download PDF

Info

Publication number
WO2022000456A1
WO2022000456A1 PCT/CN2020/100105 CN2020100105W WO2022000456A1 WO 2022000456 A1 WO2022000456 A1 WO 2022000456A1 CN 2020100105 W CN2020100105 W CN 2020100105W WO 2022000456 A1 WO2022000456 A1 WO 2022000456A1
Authority
WO
WIPO (PCT)
Prior art keywords
pixels
data
aggregation result
cost aggregation
group
Prior art date
Application number
PCT/CN2020/100105
Other languages
English (en)
French (fr)
Inventor
陈琳
李泽祥
高明明
Original Assignee
深圳市大疆创新科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市大疆创新科技有限公司 filed Critical 深圳市大疆创新科技有限公司
Priority to PCT/CN2020/100105 priority Critical patent/WO2022000456A1/zh
Priority to CN202080005628.4A priority patent/CN112889030A/zh
Publication of WO2022000456A1 publication Critical patent/WO2022000456A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows

Definitions

  • the present application relates to the technical field of image processing, and in particular, to an image processing method, apparatus, and device.
  • the binocular stereo matching technology uses different viewpoints to framing the same scene to match the corresponding points of the two images obtained, so as to obtain the corresponding point parallax, and then the depth information of the pixels can be calculated.
  • This technology can be applied to the fields of UAV and unmanned vehicle obstacle avoidance and 3D reconstruction.
  • the process of calculating depth information includes: matching cost calculation, cost aggregation, disparity calculation and optimization, and disparity refinement. Among them, for the step of cost aggregation, it takes a long time and the processing frame rate is low. Therefore, it is necessary to propose an improved solution.
  • Embodiments of the present application provide an image processing method, apparatus, and device, which are used to solve the problems of long time consumption and low processing frame rate in the prior art.
  • an embodiment of the present application provides an image processing method, in which a target image in two images corresponding to different viewpoints is divided into multiple blocks, and a certain processing order exists between the multiple blocks; the Methods include:
  • first group of computing units and the second group of computing units are two groups of computing units processed in parallel in an integrated circuit.
  • an embodiment of the present application provides an integrated circuit, in which a target image in two images corresponding to different viewpoints is divided into multiple blocks, and a certain processing sequence exists between the multiple blocks; the integrated circuit The circuit includes: a first group of computing units and a second group of computing units processed in parallel;
  • the first group of calculation units are used to calculate and obtain a first cost aggregation result of a plurality of first pixels located in the first block of the plurality of blocks in the first direction;
  • the second group of calculation units is configured to calculate and obtain a second cost aggregation result of a plurality of second pixels located in a second block of the plurality of blocks in a second direction.
  • an embodiment of the present application provides an image processing apparatus, including: the integrated circuit according to any one of the second aspect.
  • an embodiment of the present application provides an unmanned aerial vehicle, wherein the unmanned aerial vehicle includes a body, a power system, a photographing device, and an integrated circuit arranged on the body;
  • the photographing device is used for photographing during the flight of the UAV to obtain two images corresponding to different viewpoints, and the target image in the two images is divided into a plurality of blocks, and the plurality of There is a certain processing order between blocks;
  • the integrated circuit includes: a first group of computing units and a second group of computing units that are processed in parallel;
  • the first group of calculation units are used to calculate and obtain a first cost aggregation result of a plurality of first pixels located in the first block of the plurality of blocks in the first direction;
  • the second group of calculation units is configured to calculate and obtain a second cost aggregation result of a plurality of second pixels located in a second block of the plurality of blocks in a second direction.
  • Embodiments of the present application provide an image processing method, apparatus, and device, which reduce processing time and improve the processing frame rate.
  • FIG. 1 is a schematic diagram of an application scenario of an image processing method provided by an embodiment of the present application.
  • FIG. 2 is a schematic diagram of 16 cost aggregation directions provided by an embodiment of the present application.
  • FIG. 3 is a schematic diagram of eight cost aggregation directions provided by an embodiment of the present application.
  • FIG. 4 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • FIG. 5 is a schematic diagram of block division of a target image according to an embodiment of the present application.
  • FIG. 6 is a schematic diagram of a processing sequence of blocks by the first and second groups of computing units according to an embodiment of the present application
  • FIG. 7 is a schematic flowchart of an image processing method provided by another embodiment of the present application.
  • FIG. 8 is a schematic diagram of reading a first cost aggregation result from an internal memory according to an embodiment of the present application.
  • 9a is a schematic diagram of accessing a first cost aggregation result in multiple consecutive clock cycles according to an embodiment of the present application.
  • FIG. 9b is a schematic diagram of accessing a first cost aggregation result in multiple consecutive clock cycles according to another embodiment of the present application.
  • FIG. 10 is a schematic diagram of accessing a first cost aggregation result in consecutive multiple clock cycles according to another embodiment of the present application.
  • FIG. 11 is a schematic flowchart of an image processing method provided by another embodiment of the present application.
  • FIG. 12 is a schematic diagram of the principle that some pixels include upper edge pixels according to an embodiment of the present application.
  • FIG. 13 is a schematic diagram of the principle that some pixels include left edge pixels according to an embodiment of the present application.
  • FIG. 14 is a schematic diagram of the principle that some pixels include lower edge pixels according to an embodiment of the present application.
  • 15 is a schematic diagram of the principle that some pixels include right edge pixels according to an embodiment of the present application.
  • 16 is a schematic diagram of some pixels of a block provided by an embodiment of the present application.
  • 17 is a schematic diagram of restoring the unidirectional cost aggregation results of other pixels based on the unidirectional cost aggregation results of some pixels according to an embodiment of the present application;
  • FIG. 18 is a schematic diagram of some pixels of each block in a target image provided by an embodiment of the present application.
  • 19 is a schematic diagram of the processing sequence of blocks by the first and second groups of computing units provided by another embodiment of the present application.
  • FIG. 21 is a schematic structural diagram of an unmanned aerial vehicle provided by an embodiment of the present application.
  • the image processing method provided in this embodiment of the present application can be applied to the image processing system 10 shown in FIG. 1 , where the image processing system 10 includes a photographing device 11 and an image processing device 12 .
  • the photographing device 11 can capture two images corresponding to different viewpoints.
  • the photographing device 11 can be, for example, a binocular camera.
  • the photographing device 11 can also be other types of devices (for example, a device including multiple cameras). ), which is not limited in this application.
  • the image processing device 12 may obtain two images corresponding to different viewpoints from the photographing device 11, and use the image processing method provided in the embodiment of the present application to process the target image in the two images.
  • the image processing apparatus 12 may include an integrated circuit (Integrated Circuit, IC), and the integrated circuit may be, for example, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC).
  • the photographing device 11 and the image processing device 12 can be integrated into the same device, and the device can be, for example, an unmanned aerial vehicle, a smart phone, etc.
  • the device can also be in other forms. Not limited.
  • the photographing device 11 and the image processing device 12 may be installed in different devices.
  • the photographing device 11 may be installed in a drone, and the image processing device 12 may be installed in a control terminal for controlling the drone.
  • the photographing device 11 and the image processing device 12 may also be respectively disposed in other equipment, which is not limited in this application.
  • the image processing method provided in the embodiment of the present application is applied to the cost aggregation stage in which the depth information of the image is calculated by using the stereo matching technology, and the target image is the image for which the depth information needs to be calculated among the two images.
  • the main process of calculating image depth information includes:
  • the matching cost is also called the matching cost value.
  • the matching cost represents the similarity between two pixels.
  • Cost aggregation is used to use all pixels in the image corresponding to the entire scene to influence each single pixel to obtain a more accurate result than the matching cost.
  • Parallax calculation and optimization and parallax refinement are used to verify the calculation results and perform sub-pixel processing, etc.
  • the depth information of the target image can be calculated, and then the distance between an object and the photographing device can be further obtained according to the principle of triangulation.
  • the embodiments of the present application specifically improve the method for performing cost aggregation, and the mathematical principle of cost aggregation will be introduced below.
  • C(p, D p ) is the matching cost of a certain pixel, and p is any pixel in the image.
  • the disparity difference is 1, and the range of Np is the adjacent area of pixel p.
  • ] is a penalty term when the disparity variation range in adjacent pixels is large (ie, the disparity difference value is greater than 1).
  • the computational complexity involved in the cost aggregation process is relatively high, and it is a non-deterministic polynomial (NP) problem with polynomial complexity.
  • cost aggregation can be performed from 16 directions, cost aggregation can also be performed from 8 directions, or cost aggregation can also be performed from other number of directions, which is not limited in this embodiment of the present application.
  • the cost aggregation in a certain direction can be achieved by the following formula:
  • the first term C(p, d) in the polynomial is the matching cost of pixel p.
  • the second item is the minimum value among the 4 data including L r (pr,d), L r (pr,d-1)+P 1 , and L r (pr,d+1)+P 1 , L r (pr,d+1)+P 1 and min i L r (pr,i)+P 2 .
  • L r (pr, d) represents the cost aggregation result of the previous pixel of pixel p in the r direction under the disparity d.
  • L r (pr, d-1) represents the cost aggregation result of the previous pixel of pixel p in the r direction under the disparity d-1.
  • L r (pr, d+1) represents the cost aggregation result of the previous pixel of pixel p in the r direction under disparity d+1.
  • min i L r (pr,i) represents the minimum cost aggregation result of the previous pixel of pixel p in the r direction under the disparity i, where the value of i ranges from 0 to the maximum disparity value. .
  • P 1 and P 2 are penalty coefficients, and the farther the distance from the pixel p, the larger the penalty coefficient.
  • the third action min k L r (pr, k ) is to prevent L r (p, d) saturated, i.e. as to prevent the iteration L r (p, d) growing too large.
  • the cost aggregation result of a certain pixel p in a certain direction can be calculated, and by using this formula repeatedly, the cost aggregation results corresponding to a certain pixel p in all directions can be obtained, and then a certain pixel p
  • the cost aggregation results corresponding to all directions are added together to obtain the final cost aggregation result of a certain pixel p.
  • the disparity d corresponding to the minimum final cost aggregation result is the final disparity value.
  • the target image is divided into multiple blocks, and there is a certain processing sequence among the multiple blocks, and the first group of computing units is used to calculate the number of blocks located in the multiple blocks.
  • the cost aggregation result of multiple pixels in a block in the first direction using the second set of calculation units to calculate the cost aggregation result of multiple pixels located in the second block of multiple blocks in the second direction , wherein the first group of computing units and the second group of computing units are two groups of computing units that are processed in parallel in the integrated circuit, which realizes the calculation of the first cost aggregation result of the pixels located in the first block in the first direction.
  • performing the second cost aggregation result of the pixels located in the second block in the second direction reduces the processing time and improves the processing frame rate.
  • FIG. 4 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • the execution subject of this embodiment may be the image processing apparatus 12 .
  • the method of this embodiment may include:
  • Step 41 using the first group of calculation units to calculate and obtain the first cost aggregation result of the plurality of first pixels located in the first block of the plurality of blocks in the first direction.
  • the multiple blocks are obtained by dividing the target image, and there is a certain processing sequence among the multiple blocks.
  • the processing order may include a first processing order for the first group of computing units. Taking the target image divided into 4 blocks shown in FIG. 5 , the multiple blocks can be specifically block 1 to block 4 in FIG. 5 as an example, and the first processing sequence between the 4 blocks can be, for example, block 1 ⁇ block 2 ⁇ block 3 ⁇ block 4, or block 4 ⁇ block 3 ⁇ block 2 ⁇ block 1, of course, in other embodiments, the first The processing sequence may also be in other forms, which are not limited in this application.
  • a small square in FIG. 5 represents a pixel in the target image.
  • the cost aggregation result of multiple pixels located in block 4 in the first direction can be calculated first, and then the result of cost aggregation in block 3 can be calculated.
  • the cost aggregation results of multiple pixels in the first direction are calculated, and then the cost aggregation results of multiple pixels located in block 2 in the first direction are calculated, and finally the multiple pixels located in block 1 are calculated in the first direction.
  • the first direction refers to a part of all directions in which cost aggregation needs to be performed, for example, it may be four directions from L0_pd to L3_pd in FIG. 3 , or, for example, it may be four directions from L4_pd to L7_pd in FIG. 3 , of course. , in other embodiments, the first direction may also be in other forms, which are not limited in this application.
  • the first group of calculation units is a group of calculation units in an integrated circuit, and the first group of calculation units can be used to calculate a plurality of pixels in the first block of the plurality of blocks (for the convenience of description, hereinafter referred to as the first pixel) in the first direction of cost aggregation result (for convenience of description, hereinafter referred to as the first cost aggregation result).
  • the first cost aggregation result refers to the result of cost aggregation of pixel matching costs in the first direction. It can be understood that, in the case where the first direction is multiple directions, the first cost aggregation result may be the sum of the cost aggregation results in each direction in the multiple first directions.
  • the first cost aggregation result of the plurality of first pixels located in the first block in the first direction may be calculated based on the foregoing formula 2.
  • the first block may be block 4, block 3, block 2, and block 1 in FIG. 5 in sequence.
  • Calculate the first cost aggregation results of multiple first pixels located in block 4 based on formula 2 then calculate the first cost aggregation results of multiple first pixels located in block 3 based on formula 2, and then calculate based on formula 2.
  • the first cost aggregation results of the plurality of first pixels in block 2 are finally calculated based on formula 2.
  • the first cost aggregation results of the plurality of first pixels in block 1 are calculated.
  • the order of block 3 and block 2 can be reversed.
  • Step 42 using the second set of calculation units to calculate and obtain the second cost aggregation result of the plurality of second pixels located in the second block of the plurality of blocks in the second direction.
  • the processing sequence among the multiple blocks may further include a second processing sequence for the second group of computing units.
  • the second processing sequence may be the same as the first processing sequence. Consistent, in another embodiment the second processing order may be inconsistent with the first processing order.
  • the second processing sequence between the 4 blocks can be, for example, block 1 ⁇ block 2 ⁇ block 3 ⁇ block 4, Or block 4 ⁇ block 3 ⁇ block 2 ⁇ block 1, of course, in other embodiments, the second processing sequence among the multiple blocks may also be in other forms, which is not limited in this application.
  • the cost aggregation result of multiple pixels located in block 1 in the second direction can be calculated first, and then the result of cost aggregation in block 2 can be calculated.
  • the cost aggregation results of multiple pixels in the second direction are calculated, and then the cost aggregation results of multiple pixels located in block 3 in the second direction are calculated, and finally the multiple pixels located in block 4 are calculated in the first Cost aggregation results in both directions.
  • the second direction refers to a part of the directions that are different from the first direction among all the directions for which cost aggregation needs to be performed.
  • the first direction and the second direction may be opposite to each other.
  • the first direction is the four directions from L0_pd to L3_pd in FIG. 3
  • the second direction may be the four directions from L4_pd to L7_pd in FIG. 3
  • the first direction may be four directions from L4_pd to L7_pd in FIG. 3
  • the second direction may be four directions from L0_pd to L3_pd in FIG. 3 .
  • the first direction may also be in other forms, which are not limited in this application.
  • the second group of computing units is another group of computing units in the integrated circuit, and the second group of computing units and the first group of computing units are two groups of computing units processed in parallel in the integrated circuit, namely the first computing unit and the second computing unit.
  • Units can perform computations in parallel, which is beneficial to reduce computation time compared with serial computations.
  • the second set of calculation units can be used to calculate the cost aggregation result in the second direction of multiple pixels in the second block of the multiple blocks (for the convenience of description, hereinafter referred to as the second pixel) in the second direction (for the convenience of description, the following Recorded as the second cost aggregation result).
  • the second cost aggregation result refers to the result of cost aggregation of pixel matching costs in the second direction. It can be understood that, in the case where the second direction is multiple directions, the second cost aggregation result may be the sum of the cost aggregation results in each direction in the multiple second directions.
  • the second cost aggregation result of the plurality of second pixels located in the second block in the second direction may be calculated based on the aforementioned formula 2.
  • the second block may be block 1, block 2, block 3 and block 4 in FIG. 5 in sequence.
  • Calculate the second cost aggregation result of multiple second pixels located in block 1 based on Equation 2 then calculate the second cost aggregation result of multiple second pixels located in block 2 based on Equation 2, and then calculate based on Equation 2.
  • the second cost aggregation results of the plurality of second pixels in block 3 are finally calculated based on formula 2, and the second cost aggregation results of the plurality of second pixels located in block 4 are calculated.
  • the order of block 2 and block 3 can be reversed.
  • the target image is divided into multiple blocks, and there is a certain processing sequence among the multiple blocks, and the first group of computing units is used to calculate the number of blocks located in the multiple blocks.
  • the cost aggregation result of multiple pixels in a block in the first direction using the second set of calculation units to calculate the cost aggregation result of multiple pixels located in the second block of multiple blocks in the second direction , wherein the first group of computing units and the second group of computing units are two groups of computing units that are processed in parallel in the integrated circuit, which realizes the calculation of the first cost aggregation result of the pixels located in the first block in the first direction.
  • the second cost aggregation result of the pixels located in the second block in the second direction is calculated, and the matching cost of the pixels in the image needs to be aggregated in some directions first, and then the matching cost of the pixels in the image needs to be aggregated.
  • the calculation time can be reduced by almost half, and the frame rate can be nearly doubled, thereby reducing the processing time and improving the processing frame rate.
  • both the first cost aggregation result and the second cost aggregation result are the intermediate results of cost aggregation.
  • the first cost aggregation of the pixel p also needs to be aggregated.
  • the result is added to the second cost aggregation result for that pixel p. Therefore, in the case where the first processing order and the second processing order are inconsistent, it is necessary to store the intermediate result of cost aggregation of multiple blocks.
  • the first group of calculation units calculates and obtains the first cost aggregation result a1 of multiple pixels 4 located in block 4, and the second group of calculation units calculates and obtains the second cost aggregation result b1 of multiple pixels 1 located in block 1, which needs to be stored The first cost aggregation result a1 and the second cost aggregation result b1.
  • the first group of calculation units calculates and obtains the first cost aggregation result a2 of the plurality of pixels 3 located in the block 3, and the second group of calculation units calculates and obtains the second cost of the plurality of pixels 2 located in the block 2.
  • the cost aggregation result b2 needs to store the first cost aggregation result a2 and the second cost aggregation result b2.
  • the first group of calculation units calculates and obtains the first cost aggregation result a3 of the plurality of pixels 2 located in the block 2
  • the second group of calculation units calculates and obtains the second cost of the plurality of pixels 3 located in the block 3.
  • the second cost aggregation result b2 of the plurality of pixels 2 in block 2 has been obtained in the previous time period, after the first cost aggregation result a3 of the plurality of pixels 2 in block 2 is calculated in time period 3, it can be directly based on The second cost aggregation results b2 of the plurality of pixels 2 of block 2 are calculated in period 2 and the first cost aggregation results a3 of the plurality of pixels 2 of block 2 are calculated in period 3 to obtain the plurality of The final cost aggregation result for pixel 2.
  • the final cost aggregation result of the plurality of pixels so that there is no need to store the first cost aggregation result a3 of the plurality of pixels 2 of block 2 and the second cost aggregation result b3 of the plurality of pixels 3 of block 3 in memory .
  • the first group of calculation units calculates and obtains the first cost aggregation result a4 of the plurality of pixels 1 located in block 1
  • the second group of calculation units calculates and obtains the plurality of pixels 4 located in block 4.
  • the second cost aggregation results b1 of the plurality of pixels 1 in block 1 have been obtained in the previous period, after the first cost aggregation results a4 of the plurality of pixels 1 in block 1 are calculated in period 4, it can be Calculated directly based on the second cost aggregation result b1 of the plurality of pixels 1 of the block 1 and the first cost aggregation result a4 of the plurality of pixels 1 of the block 1 calculated within the period 1 to obtain the plurality of pixels 1 .
  • the final cost aggregation result since the second cost aggregation results b1 of the plurality of pixels 1 of the block 1 have been obtained in the previous period, after the first cost aggregation results a4 of the plurality of pixels 1 in block 1 are calculated in period 4, it can be Calculated directly based on the second cost aggregation result b1 of the plurality of pixels 1 of the block 1 and the first cost aggregation result a4 of the plurality of pixels 1 of the block 1 calculated within the period 1 to obtain
  • the second cost aggregation result b4 of the plurality of pixels 4 in block 4 is calculated in period 4, it can be Calculated directly based on the first cost aggregation result a1 of the plurality of pixels 4 of the block 4 and the second cost aggregation result b4 of the plurality of pixels 4 in the block 4 calculated within the period 1 to obtain the plurality of pixels 4 .
  • the final cost aggregation result so that there is no need to store the first cost aggregation result a4 of the plurality of pixels 1 of block 1 and the second cost aggregation result b4 of the plurality of pixels 4 of block 4 in the memory.
  • time period 1 to the time period 4 are four adjacent time periods from early to late. Pixel 1 to Pixel 4 are used to distinguish pixels in different blocks in Block 1 to Block 4.
  • the first direction and the second direction are multiple directions, and the directions included in the first direction and the second direction are different.
  • the first processing sequence and the second processing sequence may be consistent, the first block and the second block may be adjacent, and the second block may be located before the first block in the processing sequence.
  • both the first processing sequence and the second processing sequence may be block 1 ⁇ block 2 ⁇ block 3 ⁇ block 4.
  • the group calculation unit calculates and obtains the first cost aggregation result a1' of the pixels located in the block 1. At this time, the first cost aggregation result a1' needs to be stored.
  • the first group of calculation units calculates and obtains the first cost aggregation result a2' of multiple pixels located in block 2
  • the second group of calculation units calculates and obtains the first cost aggregation result a2' located in block 1
  • the final cost aggregation result of the pixels in block 1 can be directly determined during this period, There is no need to store the second cost aggregation result b1' and the first cost aggregation result a1'.
  • the first cost aggregation result of block 2 cannot be obtained, but only the first cost aggregation result a2' of block 2 is obtained. Therefore, the first cost aggregation result a2' needs to be stored.
  • the first group of calculation units calculates and obtains the first cost aggregation result a3' of multiple pixels located in block 3
  • the second group of calculation units calculates and obtains the first cost aggregation result a3' located in block 2
  • the final cost aggregation result of the pixels in block 2 can be directly determined during this period, There is no need to store the second cost aggregation result b2'.
  • the first cost aggregation result of block 3 cannot be obtained, but only the first cost aggregation result a3' of block 3 is obtained. Therefore, the first cost aggregation result a3' needs to be stored.
  • the first group of calculation units calculates and obtains the first cost aggregation result a4' of multiple pixels located in block 4
  • the second group of calculation units calculates and obtains the first cost aggregation result a4' located in block 3
  • the final cost aggregation result of the pixels in block 3 can be directly determined during this period, That is, there is no need to store the second cost aggregation result b3'.
  • the second group of computing units calculates and obtains the second cost aggregation result b4' of multiple pixels in block 4. Since the first cost aggregation result a4' of block 4 It has been obtained in the previous period, so the final cost aggregation result of the pixels in block 4 can be directly determined in this period, without storing the second cost aggregation result b4'.
  • the number of storage units with a specific size of intermediate results that need to be simultaneously stored in the storage device is less than 4. Therefore, the capacity requirement for storage space is small. For example, two storage units of a specific size may be required to store the intermediate results at the same time. Specifically, in the period from t1 seconds to t2 seconds, the first cost aggregation result a1' and the first cost aggregation result a2' may be stored.
  • the bit width of the cost aggregation result is 16 bits (bit) and the maximum disparity (disparity) is equal to 128, the method of storing two intermediate results of a specific size at the same time requires a storage space of 4Mbit .
  • FIG. 7 is a schematic flowchart of an image processing method provided by another embodiment of the present application. Based on the embodiment shown in FIG. 4 , this embodiment mainly describes a method for determining the final cost aggregation result of a certain pixel p (that is, as follows An optional implementation of the third cost aggregation result). As shown in FIG. 7 , the method of this embodiment may include:
  • Step 71 Using the first set of calculation units, calculate and obtain the first cost aggregation result of the plurality of first pixels located in the first block of the plurality of blocks in the first direction.
  • the calculated first cost aggregation result of the plurality of first pixels located in the first block in the first direction may be stored in an internal memory of the integrated circuit or an external memory of the integrated circuit.
  • the internal memory of the integrated circuit can be, for example, a static random-access memory (Static Random-Access Memory, SRAM)
  • the external memory of the integrated circuit can be, for example, a double-rate synchronous dynamic random access memory (Double Data Rate Synchronous Dynamic Random Access Memory, DDR SDRAM) (abbreviated as DDR).
  • step 41 for the specific method for obtaining the first cost aggregation result of the plurality of pixels located in the first block by using the first group of calculation units, reference may be made to the specific description of step 41, and details are not repeated here.
  • Step 72 using a second group of calculation units to calculate and obtain a second cost aggregation result of a plurality of second pixels located in the second block of the plurality of blocks in the second direction, and based on the plurality of first
  • the first cost aggregation result of two pixels in the first direction and the second cost aggregation result of the plurality of second pixels in the second direction are calculated to obtain the third cost aggregation result of the plurality of second pixels .
  • the second block is adjacent to the first block, and the second block is located before the first block in the processing sequence.
  • the first cost aggregation result of the plurality of second pixels located in the second block in the first direction has been calculated.
  • the first cost aggregation result of a plurality of second pixels located in the second block in the first direction may be calculated by the first group of calculation units, or may be obtained by the first group of The calculation units other than the calculation unit obtain the first cost aggregation result of the plurality of second pixels located in the second block in the first direction.
  • the first cost aggregation result of the plurality of second pixels located in the second block in the first direction that has been calculated before it needs to be stored in the internal storage device of the integrated circuit or outside the integrated circuit. in the storage device.
  • both the first cost aggregation result of the first pixel located in the first block and the first cost aggregation result of the second pixel located in the second block are stored in the integrated circuit in the internal memory.
  • the first cost aggregation results of multiple first pixels located in the first block and the first cost aggregation results of multiple second pixels located in the second block may share the The same storage space in the internal memory.
  • a storage space in the internal memory may store the first cost aggregation result of the second pixel located in the second block, after the period of time and adjacent to the period of time
  • the storage space in the internal memory may store the first cost aggregation result of the first pixel located in the first block.
  • a specific size is beneficial to reduce the occupation of the internal memory by the intermediate results of further aggregation, save the storage resources of the internal storage space, reduce the capacity requirement for the internal storage space, and save the cost.
  • the bit width of the cost aggregation result is 16 bits (bit) and the maximum disparity (disparity) is equal to 128, the method of storing an intermediate result of a specific size at the same time requires a storage space of 2Mbit .
  • a single calculation of the first set of calculation units and the second set of calculation units each occupies a single clock cycle.
  • a single calculation may be performed for a single pixel at all parallaxes, or a single calculation may be performed for a single pixel at a partial parallax, or a single calculation may be performed for multiple pixels under all parallaxes.
  • the following mode 1 or mode 2 can be used to realize the first pixel located in the first block.
  • the first cost aggregation result and the first cost aggregation result of the second pixel located in the second block share the same storage space in the internal memory.
  • the first data calculated by the first group of calculation units is stored in the internal memory in a storage space corresponding to the second data used by the second group of calculation units for calculation within the clock cycle.
  • the first data is data of the first cost aggregation result of the first pixel
  • the second data is the data of the first cost aggregation result of the second pixel.
  • the size of the data is related to the range targeted for a single calculation.
  • the size of the data may be the size of the first cost aggregation result of a single pixel under all parallaxes;
  • the size of the data may be the size of the first cost aggregation result of a single pixel under partial parallax; in the case where a single calculation may be performed for multiple pixels under all parallax , the size of the data may be the size of the first cost aggregation result of multiple pixels under all parallaxes.
  • the internal memory may include a first internal memory.
  • the first internal memory may be a single-port internal memory.
  • the first internal memory may be other types of internal memory (eg, dual-port internal memory) used as a single-port internal memory.
  • the method of this embodiment may further include: within a first clock cycle, reading the second data and third data adjacent to the second data from the target storage space of the first internal memory, and The first data calculated by the first group of calculation units is cached.
  • the second data is used by the second group of calculation units to calculate the third cost aggregation result within the first clock cycle;
  • the third data is the data of the first cost aggregation result of the second pixel , and the third data is used by the second group of calculation units to calculate a third cost aggregation result in a second clock cycle;
  • the second clock cycle is the next clock cycle of the first clock cycle.
  • the second group of computing units can calculate the third cost in the adjacent first clock cycle and the second clock cycle.
  • the aggregation result of the first cost of the aggregation result that is, reading from the first internal memory every other clock cycle is beneficial to reduce the number of reads for the first internal memory and improve the reading efficiency.
  • data K1 for the first cost aggregation result of the second pixel calculated by the second group of calculation units in clock cycle 1 may be read from the first internal memory K1 , and the data K2 for the first cost aggregation result of the second pixel calculated by the second group of calculation units in clock cycle 2; after that, in clock cycle 3, can be read from the first internal memory for the first cost aggregation
  • the data K3 of the first cost aggregation result of the second pixel calculated by the two sets of calculation units in clock cycle 3 and the data K3 of the second pixel in the second block used by the second set of calculation units in clock cycle 4
  • the data K4 of the aggregated result of the first cost ..., and so on.
  • the second pixel is the pixel in the second block
  • clock cycle 1 and clock cycle 3 are the first clock cycle
  • clock cycle 2 and clock cycle 4 are the second clock cycle
  • data K1 and data K3 are the aforementioned
  • the second data, data K2 and data K4 are the aforementioned third data.
  • the method of this embodiment may further include: in the second clock cycle, writing the first data and the fourth data into the target storage space of the first internal memory; the first data Four data is the data of the first cost aggregation result of the first pixel calculated by the first group of calculation units in the second clock cycle. Therefore, by writing the first internal memory once in the second clock cycle, the first cost aggregation result calculated by the first group of computing units in the first clock cycle and the second clock cycle can be written into The first internal memory, that is, writing into the first internal memory every other clock cycle is beneficial to reduce the number of times of writing to the first internal memory and improve the reading efficiency.
  • the first cost for the second pixel calculated by the second group of calculation units in clock cycle 1 may be read from the first internal memory
  • the data K1 of the aggregation result, and the data K2 of the first cost aggregation result of the second pixel calculated by the first group of calculation units in clock cycle 2 the first group of calculation units calculated the first cost aggregation of the first pixel
  • the data K5 is cached, and the second group of calculation units calculates the third cost result based on the data K1.
  • the first group of calculation units calculates and obtains the data K6 of the first cost aggregation result of the first pixel, writes the data K5 and K6 into the target storage space originally occupied by the data K1 and the data K2, and,
  • the second group of calculation units calculates the third aggregate cost result based on the data K2; after that, in clock cycle 3, the third set of calculation units used for the calculation in clock cycle 3 of the second group of calculation units can be read from the first internal memory.
  • the first group of calculation units calculates and obtains the data K8 of the first cost aggregation result of the first pixel, writes the data K7 and K8 into the target storage space originally occupied by the data K3 and the data K4, and,
  • the second group of calculation units calculates the third cost result based on the data K4; , and so on.
  • the first pixel is the pixel in the first block
  • the second pixel is the pixel in the second block
  • the data K5 and the data K7 are the aforementioned first data
  • the data K1 and the data K3 are the aforementioned first data.
  • the second data, the data K2 and the data K4 are the aforementioned third data
  • the data K6 and the data K8 are the aforementioned fourth data.
  • the internal memory may include a first internal memory
  • the first internal memory may be a dual-port internal memory.
  • data K1 for the first cost aggregation result of the second pixel calculated by the second group of calculation units in clock cycle 1 may be read from the first internal memory K1 , and write the data K5 of the first cost aggregation result of the first pixel calculated by the first group of calculation units in the clock cycle 1 into the target storage space originally occupied by the data K1 in the first internal memory; after that, in the clock cycle 2 , the data K2 for the first cost aggregation result of the second pixel calculated by the second group of calculation units in clock cycle 2 can be read from the first internal memory, and the first group of calculation units in clock cycle 2.
  • the data K6 of the first cost aggregation result of the first pixel obtained by the internal calculation is written into the target storage space originally occupied by the data K2 in the first internal memory;
  • the data K7 is written into the target storage space originally occupied by the data K3 in the first internal memory; after that, in clock cycle 4, it can be read from the first internal memory for the second group of calculation units to perform in clock cycle 4.
  • the first pixel is the pixel in the first block
  • the second pixel is the pixel in the second block
  • clock cycle 1 and clock cycle 3 are the first clock cycle
  • clock cycle 2 and clock cycle 4 are for the second clock cycle.
  • the data K1 and the data K3 are the second data
  • the data K2 and the data K4 are the third data
  • the data K5 and the data K7 are the first data
  • the data K6 and the data K8 are the fourth data.
  • the first cost The bit width of the aggregated result is 16 bits, then the size of the aggregated result of the first cost of the second pixel calculated by the second group of computing units in a single clock cycle is Dper ⁇ 16 (that is, the size of the second data is Dper ⁇ 16), The size of the first cost aggregation result of the first pixel calculated by a group of computing units is also Dper ⁇ 16 (that is, the size of the first data is also Dper ⁇ 16), so the bit width of the first internal memory can be set to Dper ⁇ 16 ⁇ 2, the bit width is doubled, the depth is reduced by 50%, and the capacity is unchanged.
  • the first data calculated by the first group of calculation units in the second clock cycle is stored in the internal memory in the storage space corresponding to the second data used by the second group of calculation units for calculation in the first clock cycle;
  • the first data is the data of the first cost aggregation result of the first pixel
  • the second data is the data of the first cost aggregation result of the second pixel
  • the second clock cycle is the the next clock cycle of the first clock cycle.
  • the internal memory may include a first internal memory and a second internal memory.
  • the first internal memory and the second internal memory may be single-port internal memory, and the first internal memory and the second internal memory are both single-port internal memory, which is conducive to saving costs;
  • the internal memory and the second internal memory may be other types of internal memories used as single-port internal memories, and the method in this embodiment may further include: within a first clock cycle, The first data is written into the first internal memory, and the second data is read from the target storage space of the second internal memory; the second data is used for the second group of computing units in the The third cost aggregation result is calculated within the first clock cycle.
  • the method of this embodiment may further include: within the second clock cycle, reading third data from the first internal memory, and writing fourth data calculated by the second group of computing units into in the target storage space of the second internal memory.
  • the third data is the data of the first cost aggregation result of the second pixel, and the third data is used for the third cost result calculated by the second group of calculation units in the second clock cycle;
  • the fourth data is the data of the first cost aggregation result of the first pixel calculated by the first group of calculation units in the second clock cycle;
  • the second clock cycle is the first clock the next clock cycle of the cycle.
  • data K1 for the first cost aggregation result of the second pixel calculated by the second group of calculation units in clock cycle 1 may be read from the second internal memory K1 , and write the data K5 of the first cost aggregation result of the first pixel calculated by the first group of calculation units in the clock cycle 1 into the first internal memory; after that, in the clock cycle 2, can be retrieved from the first internal memory.
  • Read the data K2 for the first cost aggregation result of the second pixel calculated by the second group of calculation units in clock cycle 2 and calculate the first group of calculation units in clock cycle 2 to obtain the first pixel of the first pixel.
  • the data K6 of the cost aggregation result is written into the target storage space originally occupied by the data K1 in the second internal memory; after that, in the clock cycle 3, it can be read from the second internal memory for the second group of calculation units in the clock cycle.
  • the data K3 of the first cost aggregation result of the second pixel that is calculated within 3 and the data K7 of the first cost aggregation result of the first pixel calculated by the first group of calculation units in clock cycle 3 is written into the first internal memory.
  • the first pixel is the pixel in the first block
  • the second pixel is the pixel in the second block
  • clock cycle 1 and clock cycle 3 are the first clock cycle
  • clock cycle 2 and clock cycle 4 are
  • data K1 and data K3 are second data
  • data K2 and data K4 are third data
  • data K5 and data K7 are first data
  • data K6 and data K8 are fourth data.
  • the first cost aggregation result, the second cost aggregation result, and the third cost aggregation result may specifically include aggregation results of pixels under each parallax within the parallax range.
  • the first cost aggregation result of a pixel p under disparity d can be expressed as L first direction (p, d)
  • the second cost aggregation result of pixel p under disparity d can be expressed as L second direction (p, d)
  • the third cost aggregation result of pixel p under disparity d can be expressed as L first direction + second direction (p, d).
  • the third cost aggregation result of the plurality of second pixels may specifically include: for each pixel in the plurality of second pixels, comparing the first cost aggregation result of the pixel under each parallax with the pixel in each parallax.
  • the second cost aggregation results under the parallax are correspondingly added to obtain the third cost aggregation results of the pixels under each of the parallaxes.
  • the first cost aggregation result and the second cost aggregation result are intermediate results of cost aggregation
  • the third cost aggregation result is the final cost aggregation result.
  • the first direction is L0_pd-L3_pd
  • the second direction is L4_pd-L7_pd
  • Two directions (p,d) L L0_pd (p,d)+L L1_pd (p,d)+
  • the storage manner of the third cost aggregation result can be flexibly implemented according to requirements.
  • the third cost aggregation result can be stored in an external memory such as DDR of the integrated circuit.
  • the final disparity may also be determined based on the third cost aggregate result.
  • the method of this embodiment may further include: taking the disparity with the smallest aggregation result of the pixels within the disparity range as the final disparity of the pixels. For example, if d is equal to 10 from 0 to 127, and the value of L in the first direction + the second direction (p, d) is the smallest, then 10 is the final disparity of pixel p.
  • the second set of calculation units may be used to calculate the final disparity of the pixels.
  • the cost aggregation result of the plurality of first pixels in the first block located in the plurality of blocks in the first direction is calculated and obtained, and the second a set of calculation units, which calculate and obtain a cost aggregation result of a plurality of second pixels located in the second block of the plurality of blocks in the second direction, and a first cost aggregation based on the plurality of second pixels in the first direction
  • the result and the second cost aggregation result of the plurality of second pixels in the second direction, and the third cost aggregation result of the plurality of second pixels is calculated, wherein the first block and the second block are adjacent and the second The block is located before the first block, which realizes that while calculating the first cost aggregation result of the first pixel located in the first block in the first direction, the second pixel located in the second block is processed in the second block.
  • the calculation of the second cost aggregation result in the direction and the final cost aggregation result of the second pixel is beneficial to reduce the processing time, improve the processing frame rate, reduce the intermediate data that needs to be stored, and reduce the capacity requirements for the memory, which is beneficial to cut costs.
  • FIG. 11 is a schematic flowchart of an image processing method provided by another embodiment of the present application.
  • this embodiment mainly describes an optional implementation manner in which the first cost aggregation result is obtained by calculating by the first group of computing units.
  • the method of this embodiment may include:
  • Step 111 Using the first group of computing units, read from the memory the unidirectional cost aggregation results of some pixels in the plurality of first pixels in at least one of the plurality of first directions, and based on the The first cost aggregation result of the part of the pixels is calculated to obtain the first cost aggregation result of other pixels in the plurality of first pixels; the part of the pixels in at least one of the plurality of first directions in the first cost.
  • Unidirectional cost aggregation results in one direction are obtained from previous processing and stored in the memory.
  • the unidirectional cost aggregation result is one of L0_pd to L7_pd.
  • Step 112 using a second set of calculation units to calculate and obtain a second cost aggregation result of a plurality of second pixels located in a second block of the plurality of blocks in a second direction.
  • step 112 reference may be made to the relevant descriptions of the foregoing embodiments, and details are not described herein again.
  • the unidirectional cost aggregation result refers to the cost aggregation result in a single direction.
  • the cost aggregation results of some pixels in the L0_pd direction of the plurality of first pixels may be read from the memory, and the plurality of first pixels may be read from the memory.
  • the cost aggregation result of some pixels in a pixel in the L1_pd direction may be read from the memory.
  • the previous processing can be done based on the aforementioned Equation 2.
  • the first direction includes the 4 directions in the upper left part in FIG. 3 , that is, L0_pd to L3_pd
  • a certain pixel p in the r direction can be calculated based on the formula 2 given above. Calculate the unidirectional cost aggregation results of all pixels in the target image in the L0_pd to L3_pd directions respectively, and store the unidirectional cost aggregation results of at least one direction in L0_pd to L3_pd of some pixels in the memory. .
  • the unidirectional cost aggregation results of all pixels in the first block in each first direction can be calculated and obtained, but the unidirectional cost aggregation results of some pixels in at least one first direction are selected.
  • the cost aggregation result is saved.
  • the unidirectional cost aggregation results of other pixels in each first direction can be recovered to obtain the unidirectional cost aggregation results of other pixels in each first direction, so as to be based on the other pixels.
  • the unidirectional cost aggregation results in each first direction are calculated to obtain the first cost aggregation results of other pixels.
  • the memory in order to reduce the occupation of the internal memory of the integrated circuit as much as possible, the memory may be an external memory of the integrated circuit.
  • the memory may also be other memory, which is not limited in this application.
  • the first cost aggregation of other pixels in the first pixels in the first block can be calculated based on the first cost aggregation results of some pixels in the first pixels in the first block. The theoretical basis for the results.
  • Equation 2 it can be seen from Equation 2 that for the cost aggregation result of any pixel in the image in a single direction, it is determined by two parts. Specifically, the cost aggregation result includes the matching cost of the pixel and the cost aggregation result of the previous pixel in the direction. Based on this, taking the direction L0_pd as an example, Equation 2 can be rewritten as Equation 3 to describe the process of cost aggregation:
  • i is the current pixel in the image
  • C i represents the matching cost of the current pixel i
  • L0_pd i-1 represents the cost aggregation result of the previous pixel i-1 in the L0_pd direction
  • L0_pd i represents the current pixel i in the L0_pd direction
  • the cost aggregation result i is the current pixel in the image
  • C i represents the matching cost of the current pixel i
  • L0_pd i-1 represents the cost aggregation result of the previous pixel i-1 in the L0_pd direction
  • L0_pd i represents the current pixel i in the L0_pd direction
  • the cost aggregation result is the current pixel in the image
  • C i represents the matching cost of the current pixel i
  • L0_pd i-1 represents the cost aggregation result of the previous pixel i-1 in the L0_pd direction
  • L0_pd i represents
  • the cost aggregation result of a certain pixel p in a certain direction can be determined based on the cost aggregation result of the previous pixel p in this direction.
  • the first cost aggregation result of some pixels in the first pixel is calculated to obtain the first cost aggregation result of other pixels in the plurality of first pixels located in the first block.
  • the first cost aggregation result of pixels in the first direction can be expressed as:
  • the first direction includes a downward direction
  • the partial pixels include pixels located on an upper edge of the first block among the plurality of first pixels.
  • the downward direction may include a vertical downward direction and an oblique downward direction. Taking the eight directions shown in FIG. 3 as an example, the vertical downward direction may be, for example, the L1_pd direction in FIG. 3 , and the oblique downward direction may be, for example, the L0_pd direction and the L2_pd direction in FIG. 3 .
  • the cost aggregation result of a certain pixel A in the L1_pd direction can be determined based on the cost aggregation result of the previous pixel of the pixel A in the L1_pd direction, that is, the pixel A3 in the L1_pd direction.
  • the cost aggregation result of pixel A3 in the L1_pd direction can be determined based on the cost aggregation result of pixel A3 in the L1_pd direction, that is, the cost aggregation result of pixel A2 in the L1_pd direction, and the cost aggregation result of pixel A2 in the L1_pd direction can be based on pixel A1
  • the cost aggregation result in the L1_pd direction is determined, and the cost aggregation result of the pixel A1 in the L1_pd direction may be determined based on the cost aggregation result of the pixel A00 on the upper edge of the block in the L1_pd direction.
  • the cost aggregation results of the pixels on the upper edge of the first block in the vertical downward direction can be used to determine the cost aggregation results of other pixels in the vertical downward direction. Therefore, in one embodiment, the cost aggregation result of the pixels in the upper edge row of the first block (for example, the row where the pixel A00 is located) in the L1_pd direction is stored for determining the cost aggregation results of other pixels in the L1_pd direction . Among them, other pixels are located inside the first block. That is, other pixels are not on the edge of the block.
  • the cost aggregation result of a certain pixel A in the L2_pd direction may be based on the cost aggregation result of the previous pixel of the pixel A in the L2_pd direction, that is, the pixel A6 in the L2_pd direction
  • the cost aggregation result of pixel A6 in the L2_pd direction can be determined based on the cost aggregation result of pixel A6 in the L2_pd direction, that is, the cost aggregation result of pixel A5 in the L2_pd direction
  • the cost aggregation result of pixel A5 in the L2_pd direction can be based on the pixel
  • the cost aggregation result of A4 in the L2_pd direction is determined, and the cost aggregation result of the pixel A4 in the L2_pd direction can be determined based on the cost aggregation result of the
  • the cost aggregation results of the pixels on the upper edge of the first block in the oblique downward direction can be used to determine the cost aggregation results of other pixels in the oblique downward direction. Therefore, in one embodiment, the cost aggregation result of the pixels in the upper edge row of the first block (eg, the row where the pixel A01 is located) in the L2_pd direction is stored, so as to be used to determine the cost aggregation results of other pixels in the L2_pd direction . Among them, other pixels are located inside the first block. That is, other pixels are not on the edge of the block.
  • the cost aggregation result of the pixels of the upper edge row of the first block in the L0_pd direction is stored, so as to be used to determine the cost aggregation of other pixels in the L0_pd direction result.
  • other pixels are located inside the first block. That is, other pixels are not on the edge of the block.
  • the first direction includes a rightward direction
  • the partial pixels include pixels located at the left edge of the first block among the plurality of first pixels.
  • the rightward direction may include a horizontal rightward direction and an oblique rightward direction.
  • the horizontal right direction may be, for example, the L3_pd direction in FIG. 3
  • the oblique right direction may be, for example, the L2_pd direction and the L4_pd direction in FIG. 3 .
  • the cost aggregation result of a certain pixel B in the L3_pd direction can be determined based on the cost aggregation result of the previous pixel of the pixel B in the L3_pd direction, that is, the pixel B3 in the L3_pd direction.
  • the cost aggregation result of pixel B3 in the L3_pd direction can be determined based on the cost aggregation result of pixel B3 in the L3_pd direction, that is, the cost aggregation result of pixel B2 in the L3_pd direction, and the cost aggregation result of pixel B2 in the L3_pd direction can be based on pixel B1
  • the cost aggregation result in the L3_pd direction is determined, and the cost aggregation result of the pixel B1 in the L3_pd direction can be determined based on the cost aggregation result of the pixel B00 on the upper edge of the block in the L3_pd direction.
  • the cost aggregation result of the pixels on the left edge of the first block in the horizontal right direction can be used to determine the cost aggregation results of other pixels in the horizontal right direction. Therefore, in one embodiment, the cost aggregation result of the pixels in the left edge row of the first block (eg, the row where pixel B00 is located) in the L3_pd direction is stored for determining the cost aggregation results of other pixels in the L3_pd direction . Among them, other pixels are located inside the first block. That is, other pixels are not on the edge of the block.
  • the cost aggregation result of a certain pixel B in the L2_pd direction may be based on the cost aggregation result of the previous pixel of the pixel B in the L2_pd direction, that is, the pixel B6 in the L2_pd direction
  • the cost aggregation result of pixel B6 in the L2_pd direction can be determined based on the cost aggregation result of pixel B6 in the L2_pd direction, that is, the cost aggregation result of pixel B5 in the L2_pd direction
  • the cost aggregation result of pixel B5 in the L2_pd direction can be based on the pixel B5.
  • the cost aggregation result of B4 in the L2_pd direction is determined, and the cost aggregation result of the pixel B4 in the L2_pd direction may be determined based on the cost aggregation result of the pixel B01 at the left edge of the block in the L2_pd direction. It can be seen that the cost aggregation result of the pixels on the left edge of the first block in the diagonal right direction can be used to determine the cost aggregation results of other pixels in the diagonal right direction.
  • the cost aggregation result of the pixels in the left edge row of the first block (eg, the row where pixel B01 is located) in the L2_pd direction is stored for determining the cost aggregation results of other pixels in the L2_pd direction .
  • other pixels are located inside the first block. That is, other pixels are not on the edge of the block.
  • the cost aggregation result of the pixels in the left edge row of the first block in the L4_pd direction is stored for determining the cost aggregation of other pixels in the L4_pd direction result.
  • other pixels are located inside the first block. That is, other pixels are not on the edge of the block.
  • the first direction includes an upward direction
  • the partial pixels include pixels located at the lower edge of the first block among the plurality of first pixels.
  • the upward direction may include a vertical upward direction and an oblique upward direction. Taking the eight directions shown in FIG. 3 as an example, the vertical upward direction may be, for example, the L5_pd direction in FIG. 3 , and the oblique upward direction may be, for example, the L4_pd direction and the L6_pd direction in FIG. 3 .
  • the cost aggregation result of a certain pixel C in the L5_pd direction can be determined based on the cost aggregation result of the previous pixel C3 in the L5_pd direction, that is, the pixel C3 in the L5_pd direction.
  • the cost aggregation result of pixel C3 in the L5_pd direction can be determined based on the cost aggregation result of pixel C3 in the L5_pd direction, that is, the cost aggregation result of pixel C2 in the L5_pd direction, and the cost aggregation result of pixel C2 in the L5_pd direction can be based on the pixel C1.
  • the cost aggregation result in the L5_pd direction is determined, and the cost aggregation result of the pixel C1 in the L5_pd direction can be determined based on the cost aggregation result of the pixel C00 on the upper edge of the block in the L5_pd direction.
  • the cost aggregation results of the pixels on the lower edge of the first block in the vertical upward direction can be used to determine the cost aggregation results of other pixels in the vertical upward direction. Therefore, in one embodiment, the cost aggregation results of the pixels in the lower edge row of the first block (eg, the row where the pixel C00 is located) in the L5_pd direction is stored to be used to determine the cost aggregation results of other pixels in the L5_pd direction . Among them, other pixels are located inside the first block. That is, other pixels are not on the edge of the block.
  • the cost aggregation result of a certain pixel C in the L4_pd direction may be based on the cost aggregation result of the previous pixel of the pixel C in the L4_pd direction, that is, the pixel C6 in the L4_pd direction Determine
  • the cost aggregation result of the pixel C6 in the L4_pd direction can be determined based on the cost aggregation result of the pixel C6 in the L4_pd direction, that is, the cost aggregation result of the pixel C5 in the L4_pd direction
  • the cost aggregation result of the pixel C5 in the L4_pd direction can be based on the pixel C5.
  • the cost aggregation result of C4 in the L4_pd direction is determined, and the cost aggregation result of the pixel C4 in the L4_pd direction can be determined based on the cost aggregation result of the pixel C00 on the upper edge of the block in the L4_pd direction. It can be seen that the cost aggregation results of the pixels on the lower edge of the first block in the oblique upward direction can be used to determine the cost aggregation results of other pixels in the oblique upward direction.
  • the cost aggregation result of the pixels in the lower edge row of the first block (eg, the row where the pixel C01 is located) in the L4_pd direction is stored for determining the cost aggregation results of other pixels in the L4_pd direction .
  • other pixels are located inside the first block. That is, other pixels are not on the edge of the block.
  • the cost aggregation result of the pixels in the lower edge row of the first block in the L6_pd direction is stored, so as to be used to determine the cost aggregation of other pixels in the L6_pd direction result.
  • other pixels are located inside the first block. That is, other pixels are not on the edge of the block.
  • the first direction includes a leftward direction
  • the partial pixels include pixels located at the right edge of the first block among the plurality of first pixels.
  • the leftward direction may include a horizontal leftward direction and an oblique leftward direction. Taking the eight directions shown in FIG. 3 as an example, the horizontal left direction may be, for example, the L7_pd direction in FIG. 3 , and the oblique left direction may be, for example, the L0_pd direction and the L6_pd direction in FIG. 3 .
  • the cost aggregation result of a certain pixel D in the L7_pd direction can be determined based on the cost aggregation result of the previous pixel of the pixel D in the L7_pd direction, that is, the pixel D3 in the L7_pd direction.
  • the cost aggregation result of pixel D3 in the L7_pd direction can be determined based on the cost aggregation result of pixel D3 in the L7_pd direction, that is, the cost aggregation result of pixel D2 in the L7_pd direction, and the cost aggregation result of pixel D2 in the L7_pd direction can be based on the pixel D1.
  • the cost aggregation result in the L7_pd direction is determined, and the cost aggregation result of the pixel D1 in the L7_pd direction may be determined based on the cost aggregation result of the pixel D00 on the upper edge of the block in the L7_pd direction.
  • the cost aggregation result of the pixels on the right edge of the first block in the horizontal left direction can be used to determine the cost aggregation results of other pixels in the horizontal left direction. Therefore, in one embodiment, the cost aggregation result of the pixels in the right edge row of the first block (eg, the row where pixel D00 is located) in the L7_pd direction is stored for determining the cost aggregation results of other pixels in the L7_pd direction . Among them, other pixels are located inside the first block. That is, other pixels are not on the edge of the block.
  • the cost aggregation result of a certain pixel D in the L6_pd direction may be based on the cost aggregation result of the previous pixel D6 in the L6_pd direction, that is, the pixel D6 in the L6_pd direction.
  • the cost aggregation result of pixel D6 in the L6_pd direction can be determined based on the cost aggregation result of pixel D6 in the L6_pd direction, that is, the cost aggregation result of pixel D5 in the L6_pd direction, and the cost aggregation result of pixel D5 in the L6_pd direction can be based on the pixel D5.
  • the cost aggregation result of D4 in the L6_pd direction is determined, and the cost aggregation result of the pixel D4 in the L6_pd direction can be determined based on the cost aggregation result of the pixel D01 at the right edge of the block in the L6_pd direction.
  • the cost aggregation result of the pixels on the right edge of the first block in the oblique left direction can be used to determine the cost aggregation results of other pixels in the oblique left direction. Therefore, in one embodiment, the cost aggregation result of the pixels in the right edge row of the first block (eg, the row where pixel D01 is located) in the L6_pd direction is stored for determining the cost aggregation results of other pixels in the L6_pd direction . Among them, other pixels are located inside the first block. That is, other pixels are not on the edge of the block.
  • the cost aggregation result of the pixels in the right edge row of the first block in the L0_pd direction is stored, so as to be used to determine the cost aggregation of other pixels in the L0_pd direction result.
  • other pixels are located inside the first block. That is, other pixels are not on the edge of the block.
  • FIGS. 12 to 15 Based on the analysis of FIGS. 12 to 15, it can be seen that, taking the first direction as L0_pd to L3_pd in the upper left part of FIG. 3 as an example, as shown in FIG. Pixels at the top edge of the first block, and pixels at the left edge of the first block indicated by the gray box. Other pixels in the first block may specifically be pixels represented by white boxes. It should be noted that the different filling colors of the pixels in FIG. 16 indicate that the directions corresponding to the cost aggregation results that need to be stored in the memory are different.
  • the storage device For the pixel at the black frame position in Figure 16, that is, the pixel at the upper edge of the first block, the storage device needs to store the cost aggregation results in the three directions of L0_pd, L1_pd, and L2_pd respectively; for the pixel at the gray frame position in Figure 16, namely For the pixels on the left edge of the first block, the storage device needs to store the cost aggregation results in the L2_pd and L3_pd directions respectively. For the pixel at the overlapping position of the gray box and the black box in FIG. 16 , the storage device needs to store the cost aggregation results stored in the four directions of L0_pd, L1_pd, L2_pd, and L3_pd respectively.
  • the unidirectional cost aggregation result in each first direction of other pixels in the first block may be determined in a row-by-row or point-by-point manner. It should be noted that, in one embodiment, when the second direction is L0_pd ⁇ L7_pd, a method similar to the above-mentioned method may be used to obtain the cost aggregation result of the pixels located in the block in the second direction. For the sake of brevity, no further description is given.
  • the first block when the row-by-row method is adopted, can be determined in a row-by-row and column-by-row manner according to the unidirectional cost aggregation results of the pixels on the upper edge and the left edge of the first block in at least one first direction. Unidirectional cost aggregation results of other pixels in a block in each first direction.
  • the unidirectional cost matching results of the pixels in the first row in the L0_pd, L1_pd, and L2_pd directions and the matching cost of the pixels in the second row can be determined.
  • the unidirectional cost aggregation result of the pixels in the second column in the L3_pd direction can also be calculated from the unidirectional cost aggregation results of the pixels in the 1st column in the L3_pd direction and the matching cost of the pixels in the 2nd column.
  • the calculation processes of the unidirectional cost aggregation results in different directions from L0_pd to L3_pd are independent of each other, so the unidirectional cost aggregation results in different directions can be calculated in parallel.
  • the recovery of the unidirectional cost aggregation results of the pixels in the second row in the L0_pd, L1_pd, and L2_pd directions and the unidirectional cost aggregation results of the pixels in the second column in the L3_pd direction can be completed within one clock cycle.
  • For the first block shown in FIG. 17 only 16 cycles can complete the recovery calculation of the unidirectional cost aggregation results of other pixels in the first block in each first direction.
  • the calculation speed is faster.
  • the first block can be determined in a point-by-point manner according to the unidirectional cost aggregation results of the pixels of the upper edge and the left edge in the first block in at least one first direction.
  • the position of the pixel in the upper left corner of the first block is (0,0) in the L3_pd direction.
  • the unidirectional cost matching result of the pixel at position (1,1) in the L3_pd direction the position (1,1) is calculated by the unidirectional cost matching result of the pixel at position (0,1) in the L0_pd, L1_pd, and L2_pd directions respectively. ) of the pixels in the L0_pd, L1_pd, and L2_pd directions of the unidirectional cost matching results.
  • the addition result can be stored in the internal memory of the integrated circuit.
  • the addition result of the unidirectional cost matching results ie the first cost aggregation result
  • the first cost can be aggregated
  • the result is stored in memory.
  • the point-by-point restoration method can directly restore the unidirectional cost matching results of a single first pixel in the L0_pd to L3_pd directions, so The unidirectional cost matching results in the four directions can be directly added to obtain the first cost aggregation result of a single first pixel, and the first cost aggregation result is stored in the internal memory of the integrated circuit.
  • the line-by-line restoration method it is impossible to restore all the unidirectional cost matching results of a single first pixel in the L0_pd to L3_pd directions at the same time.
  • the cost aggregation result of the pixels on the upper edge of the first block in the downward direction is the default maximum value, so it can be No need to store in a storage device.
  • the cost aggregation result of the pixels on the left edge of the first block in the right direction is the default maximum value, so there is no need to store in the storage in the device.
  • the cost aggregation result of the pixels of the lower edge of the first block in the upward direction is the default maximum value.
  • the cost aggregation result of the pixels of the right edge of the first block in the left direction is the default maximum value.
  • the above-mentioned default maximum value may not be stored in the same storage manner as the storage manner of the calculated cost aggregation result. For example, there is no need to store the default maximum value in the storage device corresponding to the calculated cost aggregation result.
  • the first Part of the pixels in the block may be the pixels in the black frame position and the pixels in the gray frame position in FIG. 18 .
  • the first direction includes L0-pd to L3-pd, theoretically, some of the pixels in the first block are the pixels on the upper edge and the left edge of the first block, and some of the pixels should be located at L0- Unidirectional cost aggregation results in the pd to L3-pd direction are kept in memory.
  • the pixels on the top and left edges of block 4 are both the pixels on the edge of the target image, the pixels on the top and left edges of block 4 are unidirectional costs in the L0-pd to L3-pd directions, respectively.
  • the aggregation result can directly take the default value.
  • the pixels on the top edge of block 3 are all pixels on the top edge of the target image, the unidirectional cost aggregation results of the pixels on the top edge of block 3 in the L0-pd to L3-pd directions can directly take the default value. .
  • the unidirectional cost aggregation results of the pixels on the left edge of block 2 in the L0-pd to L3-pd directions can directly take the default value.
  • the pixels on the left edge and the top edge of block 1 are not the pixels on the edge of the target image, the pixels on the top edge and the left edge of block 2 are unidirectional costs in the L0-pd to L3-pd directions, respectively.
  • the aggregate result cannot directly take the default value.
  • the first aggregation result of the plurality of pixels in the target block in the plurality of blocks in the first direction may be obtained during the previous processing, the The target block is the block first processed by the second set of computing units among the plurality of blocks.
  • the first cost aggregation result of multiple pixels in block 1 in the first direction can be obtained in advance in the previous processing process , with reference to Figure 19:
  • the first group of calculation units can calculate and obtain the first cost aggregation results of multiple pixels located in block 2 in the first direction
  • the second group of The computing unit may calculate and obtain the second cost aggregation result of the plurality of pixels located in the block 1 in the second direction, and based on the second cost aggregation result of the plurality of pixels located in the block 1 in the first direction obtained in the previous processing process.
  • a cost aggregation result is calculated to obtain a third cost aggregation result of a plurality of pixels located in block 1.
  • the first group of calculation units can calculate and obtain the first cost aggregation result of multiple pixels located in block 3 in the first direction
  • the second group of calculation units can calculate and obtain The second cost aggregation result of the plurality of pixels located in the block 2 in the second direction, and based on the calculation of the first group of calculation units in the period of t0 seconds to t1 seconds, the plurality of pixels in the block 2 are in the first direction.
  • the upward first cost aggregation result is calculated to obtain the third cost aggregation result of a plurality of pixels located in block 2.
  • the first group of calculation units can calculate and obtain the first cost aggregation result of multiple pixels located in block 4 in the first direction
  • the second group of calculation units can calculate and obtain The aggregated result of the second cost of the plurality of pixels located in the block 3 in the second direction, and based on the calculation of the first group of calculation units in the period of t1 seconds to t2 seconds, the plurality of pixels in the block 3 are in the first direction.
  • the upward first cost aggregation result is calculated to obtain the third cost aggregation result of a plurality of pixels located in block 3.
  • the second group of calculation units can calculate and obtain the second cost aggregation result of the plurality of pixels located in block 4 in the second direction, and based on the first group of calculation units in the second cost aggregation result
  • the first cost aggregation result of the plurality of pixels in block 4 in the first direction is calculated and obtained during the period from t2 seconds to t3 seconds, and the third cost aggregation result of the plurality of pixels located in block 4 is obtained by calculation. Comparing Fig. 19 and Fig. 6, it can be seen that the first aggregation result of the plurality of pixels in the block 1 in the first direction obtained in the previous processing process is more than that in the block 1 calculated by the first group of computing units.
  • the first aggregation result of the pixels in the first direction can shorten the processing time for the entire image, that is, shorten the time consumption from t5 seconds to t4 seconds.
  • the unidirectional cost aggregation result of some pixels in at least one first direction among the plurality of first pixels is read from the memory, and based on the results of the partial pixels
  • the first cost aggregation result is calculated to obtain the first cost aggregation result of other pixels in the plurality of first pixels, which realizes the unidirectional cost aggregation result based on some pixels in a block in at least one direction, and the calculation results in the block are obtained.
  • the method of the first cost aggregation results of other pixels compared with the calculation of the first cost aggregation results of pixels based on Equation 2, expands the range of methods for calculating the first cost aggregation results of pixels and increases the first cost of calculating pixels. Flexibility of aggregated results.
  • FIG. 20 is a schematic structural diagram of an integrated circuit provided by an embodiment of the application. As shown in FIG. 20 , the integrated circuit 200 may include: a first group of computing units 21 and a second group of computing units 22 that are processed in parallel.
  • the first group of calculation unit 21 is used to calculate and obtain the first cost aggregation result of the plurality of first pixels located in the first block of the plurality of blocks in the first direction;
  • the second group of calculation units 22 is configured to calculate and obtain a second cost aggregation result of a plurality of second pixels located in a second block of the plurality of blocks in a second direction.
  • the integrated circuit provided in this embodiment can be used to implement the technical solutions of the foregoing method embodiments, and the implementation principles and technical effects thereof are similar to those of the method embodiments, which will not be repeated here.
  • An embodiment of the present application further provides an image processing apparatus, and the image processing apparatus may include an integrated circuit as shown in FIG. 20 .
  • FIG. 21 is a schematic structural diagram of an unmanned aerial vehicle provided by an embodiment of the application.
  • the unmanned aerial vehicle 210 includes a fuselage 211 , a power system 212 disposed on the fuselage, a photographing device 213 and an integrated circuit 214;
  • the power system 212 is used to provide power for the unmanned aerial vehicle
  • the photographing device 213 is used for photographing during the flight of the UAV to obtain two images corresponding to different viewpoints.
  • the target image in the two images is divided into multiple blocks, and the multiple There is a certain processing order between blocks;
  • the integrated circuit 214 includes: a first group of computing units and a second group of computing units that are processed in parallel;
  • the first group of calculation units are used to calculate and obtain a first cost aggregation result of a plurality of first pixels located in the first block of the plurality of blocks in the first direction;
  • the second group of calculation units is configured to calculate and obtain a second cost aggregation result of a plurality of second pixels located in a second block of the plurality of blocks in a second direction.
  • the drone 210 may further include a gimbal 215 , and the photographing device 213 may be disposed on the fuselage 211 through the gimbal 215 .
  • the drone may also include other elements or devices, which are not listed here.
  • the embodiments of the present application provide an image processing method, device, and device.
  • a target image is divided into multiple blocks, and a certain processing sequence exists between multiple blocks.
  • the calculation results in multiple blocks are calculated.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Processing (AREA)

Abstract

一种图像处理方法、装置及设备，该方法包括：采用第一组计算单元，计算得到位于多个区块中第一区块中的多个第一像素在第一方向上的第一代价聚合结果；采用第二组计算单元，计算得到位于多个区块中第二区块中的多个第二像素在第二方向上的第二代价聚合结果；其中，第一组计算单元和第二组计算单元是集成电路中并行处理的两组计算单元。本申请减少了处理耗时，提高了处理帧率。

Description

图像处理方法、集成电路、装置及设备 技术领域
本申请涉及图像处理技术领域，尤其涉及一种图像处理方法、装置及设备。
背景技术
双目立体匹配技术利用不同视点对同一场景进行取景后所获得的两幅图像进行对应点匹配，从而获得对应点视差，进而可以计算得到像素的深度信息。此技术可以应用于无人机、无人车的避障以及三维重建等领域。一般来说，计算深度信息的过程包括：匹配代价计算、代价聚合、视差计算和优化、视差精细化。其中，对于代价聚合的步骤来说，耗时较长，处理帧率较低。因此，有必要提出一种改进的方案。
发明内容
本申请实施例提供一种图像处理方法、装置及设备，用以解决现有技术中耗时较长，处理帧率较低的问题。
第一方面，本申请实施例提供一种图像处理方法，对应不同视点的两幅图像中的目标图像被划分为多个区块，所述多个区块之间存在一定的处理顺序；所述方法包括：
采用第一组计算单元，计算得到位于所述多个区块中第一区块中的多个第一像素在第一方向上的第一代价聚合结果；
采用第二组计算单元，计算得到位于所述多个区块中第二区块中的多个第二像素在第二方向上的第二代价聚合结果；
其中，所述第一组计算单元和所述第二组计算单元是集成电路中并行处 理的两组计算单元。
第二方面，本申请实施例提供一种集成电路，对应不同视点的两幅图像中的目标图像被划分为多个区块，所述多个区块之间存在一定的处理顺序；所述集成电路包括：并行处理的第一组计算单元和第二组计算单元；
所述第一组计算单元，用于计算得到位于所述多个区块中第一区块中的多个第一像素在第一方向上的第一代价聚合结果；
所述第二组计算单元，用于计算得到位于所述多个区块中第二区块中的多个第二像素在第二方向上的第二代价聚合结果。
第三方面，本申请实施例提供一种图像处理装置，包括：第二方面任一项所述的集成电路。
第四方面，本申请实施例提供一种无人机，所述无人机包括机身、设置于所述机身上的动力系统、拍摄装置和集成电路；
所述动力系统，用于为所述无人机提供动力；
所述拍摄装置，用于在所述无人机飞行过程中进行拍摄，以获得对应不同视点的两幅图像，所述两幅图像中的目标图像被划分为多个区块，所述多个区块之间存在一定的处理顺序；
所述集成电路包括：并行处理的第一组计算单元和第二组计算单元；
所述第一组计算单元，用于计算得到位于所述多个区块中第一区块中的多个第一像素在第一方向上的第一代价聚合结果；
所述第二组计算单元，用于计算得到位于所述多个区块中第二区块中的多个第二像素在第二方向上的第二代价聚合结果。
本申请实施例提供一种图像处理方法、装置及设备，减少了处理耗时，提高了处理帧率。
附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作一简单地介绍，显而易见地，下面描述中的附图是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本申请实施例提供的图像处理方法的应用场景示意图；
图2为本申请一实施例提供的16个代价聚合方向示意图；
图3为本申请一实施例提供的8个代价聚合方向示意图；
图4为本申请一实施例提供的图像处理方法的流程示意图；
图5为本申请实施例提供的目标图像的区块划分的示意图；
图6为本申请一实施例提供的第一、第二组计算单元对于区块的处理顺序示意图；
图7为本申请另一实施例提供的图像处理方法的流程示意图；
图8为本申请一实施例提供的从内部存储器中读取第一代价聚合结果的示意图；
图9a为本申请一实施例提供的连续多个时钟周期内第一代价聚合结果的存取示意图；
图9b为本申请另一实施例提供的连续多个时钟周期内第一代价聚合结果的存取示意图；
图10为本申请又一实施例提供的连续多个时钟周期内第一代价聚合结果的存取示意图；
图11为本申请又一实施例提供的图像处理方法的流程示意图；
图12为本申请一实施例提供的部分像素包括上边缘像素的原理的示意图；
图13为本申请一实施例提供的部分像素包括左边缘像素的原理的示意图；
图14为本申请一实施例提供的部分像素包括下边缘像素的原理的示意图；
图15为本申请一实施例提供的部分像素包括右边缘像素的原理的示意图；
图16为本申请一实施例提供的区块的部分像素的示意图；
图17为本申请一实施例提供的基于部分像素的单方向代价聚合结果恢复其他像素的单方向代价聚合结果的示意图；
图18为本申请一实施例提供的目标图像中各区块的部分像素的示意图；
图19为本申请另一实施例提供的第一、第二组计算单元对于区块的处理顺序示意图；
图20为本申请一实施例提供的集成电路的结构示意图；
图21为本申请一实施例提供的无人机的结构示意图。
具体实施方式
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本申请实施例提供的图像处理方法可以应用于如图1所示的图像处理系统10，图像处理系统10包括拍摄装置11和图像处理装置12。
拍摄装置11可以拍摄得到对应不同视点的两幅图像，拍摄装置11例如可以为双目摄像头，当然，在其他实施例中，拍摄装置11还可以为其他类型装置(例如，包含多个摄像头的装置)，本申请对此不做限定。
图像处理装置12可以从拍摄装置11获得对应不同视点的两幅图像，并针对两幅图像中的目标图像采用本申请实施例提供的图像处理方法进行处理。其中，图像处理装置12可以包括集成电路(Integrated Circuit，IC)，该集成电路例如可以为专用集成电路(Application Specific Integrated Circuit，ASIC)。
可选的，拍摄装置11和图像处理装置12可以集成于同一设备，该设备例如可以为无人机、智能手机等，当然，在其他实施例中该设备还可以为其他形式，本申请对此不做限定。
或者，可选的，拍摄装置11和图像处理装置12可以设置于不同的设备，例如，拍摄装置11可以设置于无人机中，图像处理装置12可以设置于用于控制无人机的控制终端中，当然，在其他实施例中，拍摄装置11和图像处理装置12还可以分别设置于其他设备中，本申请对此不做限定。
本申请实施例提供的图像处理方法是应用于利用立体匹配技术计算图像深度信息的代价聚合阶段，所述目标图像即为两幅图像中需要计算深度信息的图像。其中，计算图像深度信息的主要过程包括：
1、计算匹配代价(matching cost)。其中，匹配代价也称为匹配代价值。
2、代价聚合(cost aggregation)。
3、视差计算和优化(disparity computation/optimization)。
4、视差精细化(disparity refinement)。
其中，匹配代价表示两个像素之间的相似度。代价聚合是用于利用整个场景对应的图像中的所有像素对每一个单一像素施加影响，来获得一个较匹配代价更加精确的结果。视差计算和优化和视差精细化用于对计算结果进行校验以及进行亚像素处理等。通过上面列举的图像深度信息的计算过程，可以计算出目标图像的深度信息，然后根据三角测量原理可以进一步得到某一物体到拍摄装置之间的距离。本申请实施例具体针对进行代价聚合的方法进行改进，下面将介绍代价聚合的数学原理。
在数学上，代价聚合的过程可以看作是求取一个全局能量最小值：
Figure PCTCN2020100105-appb-000001
其中，C(p,D p)为某一像素的匹配代价，p为图像中的任一像素。P 1T[|D p-D q|＝1]为相邻像素中视差变化范围较小时的惩罚项，T[|D p-D q|＝1]表示相邻像素D p和D q之间的视差差值为1，Np范围为像素p的相邻区域。P 2T[|D p-D q＞1|]为相邻像素中视差变化范围较大(即，视差差值大于1)时的惩罚项。代价聚合过程涉及的计算复杂度较高，是一个多项式复杂程度的非确定性问题(Non-deterministic Polynomial，NP)问题。
为了降低代价聚合过程的计算复杂度，可以仅使用某些方向上的像素进行代价聚合。如图2和图3所示，依次是16个方向和8个方向示意图，图中的p为图像中某一像素。除了可以从16个方向进行代价聚合之外，还可以从8个方向进行代价聚合，或者也可以从其他数目的方向进行代价聚合，对此本申请实施例不做限定。
对于图像中的某一像素p，在某一方向上的代价聚合可以通过下面的公式实现：
L r(p,d)＝C(p,d)+min(L r(p-r,d),L r(p-r,d-1)+P 1,
L r(p-r,d+1)+P 1,min iL r(p-r,i)+P 2)      (公式2)
-min kL r(p-r,k)
其中，多项式中的第一项C(p,d)为像素p的匹配代价。第二项为4个数据中的最小值，该4个数据包括L r(p-r,d)、L r(p-r,d-1)+P 1、L r(p-r,d+1)+P 1、L r(p-r,d+1)+P 1和min iL r(p-r,i)+P 2。L r(p-r,d)表示像素p在r方向上的上一个像素在视差d下的代价聚合结果。L r(p-r,d-1)表示像素p在r方向上的上一个像素在视差d-1下的代价聚合结果。L r(p-r,d+1)表示像素p在r方向上的上 一个像素在视差d+1下的代价聚合结果。min iL r(p-r,i)表示像素p在r方向上的上一个像素在视差i下的最小代价聚合结果，其中，i的取值范围为0到最大视差值之间的任一数值。P 1和P 2为惩罚系数，与像素p的距离越远，惩罚系数越大。第三项min kL r(p-r,k)的作用是防止L r(p,d)饱和，即防止随着迭代L r(p,d)的增长过大。
通过上面的公式2，可以计算某一像素p在某一方向上的代价聚合结果，重复使用该公式，可以得到某一像素p在所有方向上分别对应的代价聚合结果，进而将某一像素p在所有方向上分别对应的代价聚合结果进行相加，即可得到某一像素p的最终代价聚合结果。对于某一像素p来说，最小的最终代价聚合结果对应的视差d为最终的视差取值。
本申请实施例提供的图像处理方法，通过目标图像被划分为多个区块，多个区块之间存在一定的处理顺序，采用第一组计算单元，计算得到位于多个区块中的第一区块的多个像素在第一方向上的代价聚合结果，采用第二组计算单元，计算得到位于多个区块的第二区块中的多个像素在第二方向上的代价聚合结果，其中，第一组计算单元和第二组计算单元是集成电路中并行处理的两组计算单元，实现了在计算位于第一区块中的像素在第一方向上的第一代价聚合结果的同时，进行位于第二区块中的像素在第二方向上的第二代价聚合结果的，减少了处理耗时，提高了处理帧率。
下面结合附图，对本申请的一些实施方式作详细说明。在不冲突的情况下，下述的实施例及实施例中的特征可以相互组合。
图4为本申请一实施例提供的图像处理方法的流程示意图，本实施例的执行主体可以为图像处理装置12。如图4所示，本实施例的方法可以包括：
步骤41，采用第一组计算单元，计算得到位于多个区块中第一区块中的多个第一像素在第一方向上的第一代价聚合结果。
本步骤中，所述多个区块是通过对目标图像划分得到，多个区块之间存在一定的处理顺序。所述处理顺序可以包括针对第一组计算单元的第一处理顺序。以将目标图像划分为图5所示的4个区块，多个区块具体可以为图5中的区块1至区块4为例，4个区块之间第一处理顺序例如可以为区块1→区块2→区块3→区块4，或者区块4→区块3→区块2→区块1，当然，在其他实施例中，多个区块之间的第一处理顺序还可以为其他形式，本申请对此不做限定。
需要说明的是，图5中一个小方格表示目标图像中的一个像素。
需要说明的是，图5中给出的区块划分方式仅为举例。
以区块4→区块3→区块2→区块1为例，可以先计算得到位于区块4中的多个像素在第一方向上的代价聚合结果，再计算得到位于区块3中的多个像素在第一方向上的代价聚合结果，然后计算得到位于区块2中的多个像素在第一方向上的代价聚合结果，最后计算得到位于区块1中的多个像素在第一方向上的代价聚合结果。
所述第一方向是指需要进行代价聚合的所有方向中的部分方向，例如可以为图3中的L0_pd至L3_pd四个方向，或者，例如可以为图3中的L4_pd至L7_pd四个方向，当然，在其他实施例中，第一方向还可以为其他形式，本申请对此不做限定。
所述第一组计算单元是集成电路中的一组计算单元，第一组计算单元可以用于计算多个区块中第一区块中的多个像素(为了便于描述，以下记为第一像素)在第一方向上的代价聚合结果(为了便于描述，以下记为第一代价聚合结果)。
第一代价聚合结果是指像素的匹配代价在第一方向上代价聚合的结果。可以理解的是，在第一方向为多个方向情况下，第一代价聚合结果可以为多个第一方向中各方向上代价聚合的结果之和。
在一个实施例中，可以基于前述公式2计算位于第一区块中的多个第一像素在第一方向上的第一代价聚合结果。以第一方向为图3中的L0_pd至L3_pd四个方向为例，第一区块依次可以为图5中的区块4、区块3、区块2和区块1，具体的，可以先基于公式2计算位于区块4中的多个第一像素的第一代价聚合结果，然后基于公式2计算位于区块3中多个第一像素的第一代价聚合结果，之后基于公式2计算位于区块2中多个第一像素的第一代价聚合结果，最后基于公式2计算位于区块1中多个第一像素的第一代价聚合结果。其中，区块3和区块2的顺序可以对调。
步骤42，采用第二组计算单元，计算得到位于所述多个区块中第二区块中的多个第二像素在第二方向上的第二代价聚合结果。
本步骤中，所述多个区块之间的处理顺序还可以包括针对第二组计算单元的第二处理顺序，需要说明的是，在一个实施例中第二处理顺序可以与第一处理顺序一致，在另一个实施例中第二处理顺序可以与第一处理顺序不一致。同样，以将目标图像划分为如图5所示的4个区块为例，4个区块之间第二 处理顺序例如可以为区块1→区块2→区块3→区块4，或者区块4→区块3→区块2→区块1，当然，在其他实施例中，多个区块之间的第二处理顺序还可以为其他形式，本申请对此不做限定。
以区块1→区块2→区块3→区块4为例，可以先计算得到位于区块1中的多个像素在第二方向上的代价聚合结果，再计算得到位于区块2中的多个像素在第二方向上的代价聚合结果，然后计算得到位于区块3中的多个像素在第二方向上的代价聚合结果，最后计算得到位于区块4中的多个像素在第二方向上的代价聚合结果。
所述第二方向是指需要进行代价聚合的所有方向中与第一方向不同的部分方向。可选的，为了简化实现，第一方向和第二方向可以对应相反，例如第一方向为图3中的L0_pd至L3_pd四个方向，第二方向可以为图3中的L4_pd至L7_pd四个方向，又例如，第一方向可以为图3中的L4_pd至L7_pd四个方向，第二方向可以为图3中L0_pd至L3_pd四个方向。当然，在其他实施例中，第一方向还可以为其他形式，本申请对此不做限定。
所述第二组计算单元是集成电路中的另一组计算单元，第二组计算单元和第一组计算单元是集成电路中并行处理的两组计算单元，即第一计算单元和第二计算单元可以并行进行计算，与串行计算相比，有利于减少计算耗时。第二组计算单元可以用于计算多个区块中第二区块中的多个像素(为了便于描述，以下记为第二像素)在第二方向上的代价聚合结果(为了便于描述，以下记为第二代价聚合结果)。
第二代价聚合结果是指像素的匹配代价在第二方向上代价聚合的结果。可以理解的是，在第二方向为多个方向情况下，第二代价聚合结果可以为多个第二方向中各方向上代价聚合的结果之和。
在一个实施例中，可以基于前述公式2计算位于第二区块中的多个第二像素在第二方向上的第二代价聚合结果。以第二方向为图3中的L4_pd至L7_pd四个方向为例，第二区块依次可以为图5中的区块1、区块2、区块3和区块4，具体的，可以先基于公式2计算位于区块1中的多个第二像素的第二代价聚合结果，然后基于公式2计算位于区块2中多个第二像素的第二代价聚合结果，之后基于公式2计算位于区块3中多个第二像素的第二代价聚合结果，最后基于公式2计算位于区块4中多个第二像素的第二代价聚合结果。其中，区块2和区块3的顺序可以对调。
本申请实施例提供的图像处理方法，通过目标图像被划分为多个区块，多个区块之间存在一定的处理顺序，采用第一组计算单元，计算得到位于多个区块中的第一区块的多个像素在第一方向上的代价聚合结果，采用第二组计算单元，计算得到位于多个区块的第二区块中的多个像素在第二方向上的代价聚合结果，其中，第一组计算单元和第二组计算单元是集成电路中并行处理的两组计算单元，实现了在计算位于第一区块中的像素在第一方向上的第一代价聚合结果的同时，对位于第二区块中的像素在第二方向上的第二代价聚合结果进行计算，与需要先对图像中像素的匹配代价在部分方向上进行聚合，再对图像中像素的匹配代价在另一部分方向上进行聚合的串行计算方式相比，可减少几乎一半的计算时间，提升近一倍的帧率，从而减少了处理耗时，提高了处理帧率。
可以理解的是，第一代价聚合结果和第二代价聚合结果均是代价聚合的中间结果，为了获得目标图像中某一像素p的最终代价聚合结果，还需要将该像素p的第一代价聚合结果和该像素p的第二代价聚合结果进行相加。因此，在第一处理顺序和第二处理顺序不一致的情况下，需要存储多个区块的代价聚合的中间结果。
以第一处理顺序为区块4→区块3→区块2→区块1，第二处理顺序为区块1→区块2→区块3→区块4为例：在时段1内，第一组计算单元计算得到位于区块4中多个像素4的第一代价聚合结果a1，第二组计算单元计算得到位于区块1的多个像素1的第二代价聚合结果b1，需要存储第一代价聚合结果a1和第二代价聚合结果b1。之后，在时段2内，第一组计算单元计算得到位于区块3中多个像素3的第一代价聚合结果a2，第二组计算单元计算得到位于区块2的多个像素2的第二代价聚合结果b2，需要存储第一代价聚合结果a2和第二代价聚合结果b2。然后，在时段3内，第一组计算单元计算得到位于区块2中多个像素2的第一代价聚合结果a3，第二组计算单元计算得到位于区块3的多个像素3的第二代价聚合结果b3。另外，由于区块2的多个像素2第二代价聚合结果b2在之前时段已获得，因此时段3内在计算得到区块2的该多个像素2的第一代价聚合结果a3之后，可以直接基于在时段2内计算得到的区块2的该多个像素2的第二代价聚合结果b2和时段3内在计算得到区块2的该多个像素2的第一代价聚合结果a3计算得到该多个像素2的最终代价聚合结果。类似的，由于区块3的该多个像素3的第一代价聚合结果a2在之前时段已获得，因此在时段3计算得 到区块3的该多个像素3的第二代价聚合结果b3之后，可以直接基于在时段2内计算得到的区块3的该多个像素3的第一代价聚合结果a2和时段3内在计算得到区块3的该多个像素3的第二代价聚合结果b3计算得到该多个像素3的最终代价聚合结果，从而无需在存储器中存储区块2的该多个像素2的第一代价聚合结果a3和区块3的该多个像素3的第二代价聚合结果b3。最后，在时段4内，第一组计算单元计算得到位于区块1中该多个像素1的第一代价聚合结果a4，第二组计算单元计算得到位于区块4的该多个像素4的第二代价聚合结果b4。另外，由于区块1的该多个像素1的第二代价聚合结果b1在之前时段已获得，因此时段4内在计算得到区块1的该多个像素1的第一代价聚合结果a4之后，可以直接基于在时段1内计算得到的区块1的该多个像素1的第二代价聚合结果b1和区块1的该多个像素1的第一代价聚合结果a4计算得到该多个像素1的最终代价聚合结果。类似的，由于区块4的该多个像素4第一代价聚合结果a1在之前时段已获得，因此在时段4计算得到区块4的该多个像素4的第二代价聚合结果b4之后，可以直接基于在时段1内计算得到的区块4的该多个像素4的第一代价聚合结果a1和区块4的该多个像素4的第二代价聚合结果b4计算得到该多个像素4的最终代价聚合结果，从而无需在存储器中存储区块1的该多个像素1的第一代价聚合结果a4和区块4的该多个像素4的第二代价聚合结果b4。
需要说明的是，时段1至时段4是相邻且时间由早至晚的4个时段。像素1至像素4是为了区分区块1-区块4中不同区块的像素。
可以看出，在时段1至时段4的处理过程中，存在存储设备中需要同时存储多个像素4的第一代价聚合结果a1、多个像素3的第一代价聚合结果a2、多个像素1的第二代价聚合结果b1和多个像素2的第二代价聚合结果b2。因此，一共需要用于存储4个中间结果的、具有特定大小的存储单元。其中，一个特定大小的存储单元的容量等于一个区块中一像素在第一方向下的第一代价聚合结果的大小之和，或者等于一个区块中一像素在第二方向下的第二代价聚合结果的大小之和。因此，对存储空间的容量要求较高。即，在第一处理顺序和第二处理顺序不一致的情况下，需要同时存储较多的中间结果，对存储空间的容量要求较高。其中，第一方向和第二方向为多个方向，并且第一方向和第二方向包括的方向不相同。
为了降低对于存储空间的容量要求，第一处理顺序和第二处理顺序可以一致，第一区块和第二区块可以相邻，且处理顺序中第二区块可以位于第一 区块之前。例如，第一处理顺序和第二处理顺序均可以为区块1→区块2→区块3→区块4，参考图6：在时间t由t0秒至t1秒的一段时间内，第一组计算单元计算得到位于区块1中多个像素的第一代价聚合结果a1’，此时，第一代价聚合结果a1’需要被存储。在时间t由t1秒至t2秒的一段时间内，第一组计算单元计算得到位于区块2中多个像素的第一代价聚合结果a2’，第二组计算单元计算得到位于区块1中多个像素的第二代价聚合结果b1’，由于区块1的第一代价聚合结果a1’在之前时段已获得，因此在该时段内可以直接确定出区块1中像素的最终代价聚合结果，无需存储对第二代价聚合结果b1’和第一代价聚合结果a1’。另外，由于在该时间段内，不能得到区块2的最终代价聚合结果，而仅得到区块2的第一代价聚合结果a2’。因此，需要存储第一代价聚合结果a2’。在时间t由t2秒至t3秒的一段时间内，第一组计算单元计算得到位于区块3中多个像素的第一代价聚合结果a3’，第二组计算单元计算得到位于区块2中多个像素的第二代价聚合结果b2’，由于区块2的第一代价聚合结果a2’在之前时段已获得，因此在该时段内可以直接确定出区块2中像素的最终代价聚合结果，无需存储第二代价聚合结果b2’。另外，由于在该时间段内，不能得到区块3的最终代价聚合结果，而仅得到区块3的第一代价聚合结果a3’。因此，需要存储第一代价聚合结果a3’。在时间t由t3秒至t4秒的一段时间内，第一组计算单元计算得到位于区块4中多个像素的第一代价聚合结果a4’，第二组计算单元计算得到位于区块3中多个像素的第二代价聚合结果b3’，由于区块3的第一代价聚合结果a3’在之前时段已获得，因此在该时段内可以直接确定出区块3中像素的最终代价聚合结果，即无需存储第二代价聚合结果b3’。另外，由于在该时间段内，不能得到区块4的最终代价聚合结果，而仅得到区块4的第一代价聚合结果a4’。因此，需要存储第一代价聚合结果a4’。在时间t由t4秒至t5秒的一段时间内，第二组计算单元计算得到位于区块4中多个像素的第二代价聚合结果b4’，由于区块4的第一代价聚合结果a4’在之前时段已获得，因此在该时段内可以直接确定出区块4中像素的最终代价聚合结果，无需存储第二代价聚合结果b4’。
可以看出，在t0秒至t5秒的处理过程中，存储设备中需要同时存储的中间结果的具有特定大小的存储单元的个数小于4。因此，对存储空间的容量要求较小。例如，可以需要同时存储中间结果的2个特定大小的存储单元，具体的，在由t1秒到t2秒的时段，可以存储第一代价聚合结果a1’和第一代价聚合结果 a2’。在由t2秒到t3秒的时段，由于第一代价聚合结果a1’已用于计算最终代价聚合结果，因此只需要同时存储第一代价聚合结果a2’和第一代价聚合结果a3’。在由t3秒到t4秒的时段，由于第一代价聚合结果a2’已用于计算最终代价聚合结果，因此只需要同时存储第一代价聚合结果a3’和第一代价结果a4’。假设单个区块的大小为32×32，代价聚合结果的位宽为16比特(bit)且最大视差(disparity)等于128，则同时存储2个特定大小的中间结果的方式，需要4Mbit的存储空间。
图7为本申请另一实施例提供的图像处理方法的流程示意图，本实施例在图4所示实施例的基础上，主要描述了一种确定某一像素p的最终代价聚合结果(即如下的第三代价聚合结果)的一种可选实现方式。如图7所示，本实施例的方法可以包括：
步骤71，采用第一组计算单元，计算得到位于多个区块中第一区块中的多个第一像素在第一方向上的第一代价聚合结果。
本步骤中，计算得到的位于第一区块中多个第一像素在第一方向上的第一代价聚合结果，可以存储在所述集成电路的内部存储器或所述集成电路的外部存储器中。其中，集成电路的内部存储器例如可以为静态随机存取存储器(Static Random-Access Memory，SRAM)，集成电路的外部存储器例如可以为双倍速率同步动态随机存储器(Double Data Rate Synchronous Dynamic Random Access Memory，DDR SDRAM)(简称为DDR)。
需要说明的是，关于采用第一组计算单元计算得到位于第一区块中多个像素的第一代价聚合结果的具体方式，可以参见步骤41的具体描述，在此不再赘述。
步骤72，采用第二组计算单元，计算得到位于所述多个区块中第二区块中的多个第二像素在第二方向上的第二代价聚合结果，以及基于所述多个第二像素在第一方向上的第一代价聚合结果和所述多个第二像素在所述第二方向上的第二代价聚合结果，计算得到所述多个第二像素的第三代价聚合结果。
本步骤中，所述第二区块与所述第一区块相邻，且处理顺序中所述第二区块位于所述第一区块之前。关于采用第二组计算单元计算得到位于第二区块中多个像素的第二代价聚合结果的具体方式，可以参见步骤42的具体描述，在此不再赘述。
需要说明的是，在执行步骤72之前，已计算得到位于所述第二区块中多 个第二像素在第一方向上的第一代价聚合结果。可选的，可以是由所述第一组计算单元计算得到位于所述第二区块中多个第二像素在第一方向上的第一代价聚合结果，或者，可以由所述第一组计算单元之外的其他计算单元计算得到位于所述第二区块中多个第二像素在第一方向上的第一代价聚合结果。
对于之前已计算得到的位于所述第二区块中多个第二像素在第一方向上的第一代价聚合结果，需要先存储在所述集成电路的内部存储设备或所述集成电路的外部存储设备中。
在一个实施例中，位于所述第一区块中的第一像素的第一代价聚合结果和位于所述第二区块中的第二像素的第一代价聚合结果均存储在所述集成电路的内部存储器中。通过将位于第一区块中多个第一像素的第一代价聚合结果和位于第二区块中的多个第二像素的第一代价聚合结果均存储在内部存储器中，与存储在外部存储器相比，有利于提高第一代价聚合结果的存取效率，减少对于集成电路与外部存储器之间带宽的占用。
可选的，位于所述第一区块中的多个第一像素的第一代价聚合结果和位于所述第二区块中的多个第二像素的第一代价聚合结果，可以共用所述内部存储器中的同一块存储空间。具体的，在一段时间内，内部存储器中的一存储空间中可以存储位于所述第二区块中的第二像素的第一代价聚合结果，在与该段时间之后且与该段时间相邻的另一段时间内，内部存储器中的该存储空间中可以存储位于所述第一区块中的第一像素的第一代价聚合结果。通过位于第一区块中像素的第一代价聚合结果和位于第二区块中像素的第一代价聚合结果，共用内部存储器中的同一块存储空间，能够实现需要同时存储的中间结果可以等于1个特定大小，有利于减小进一步聚合的中间结果对于内部存储器的占用，节省内部存储空间的存储资源，降低对于内部存储空间的容量要求，节省成本。假设单个区块的大小为32×32，代价聚合结果的位宽为16比特(bit)且最大视差(disparity)等于128，则同时存储1个特定大小的中间结果的方式，需要2Mbit的存储空间。
在一个实施例中，所述第一组计算单元和所述第二组计算单元的单次计算均占用单个时钟周期。示例性的，单次计算可以针对单个像素在全部视差进行计算，或者，单次计算可以针对单个像素在部分视差进行计算，或者，单次计算可以针对多个像素在全部视差下进行计算。
可选的，在第一组计算单元和第二组计算单元的单次计算均占用单个时 钟周期情况下，可以采用如下方式一或方式二实现位于所述第一区块中的第一像素的第一代价聚合结果和位于所述第二区块中的第二像素的第一代价聚合结果，共用所述内部存储器中的同一块存储空间。
方式一
一个时钟周期内，所述第一组计算单元计算得到的第一数据，被存入所述内部存储器中该时钟周期内所述第二组计算单元计算使用的第二数据对应的存储空间。其中，所述第一数据为所述第一像素的第一代价聚合结果的数据，所述第二数据为所述第二像素的第一代价聚合结果的数据。由此，可以实现一个时钟周期内，计算得到的第一像素的第一代价聚合结果，占用该时钟周期内计算使用的第二像素的第一代价聚合结果。
需要说明的是，所述数据的大小与单次计算针对的范围相关。示例性的，在单次计算可以针对单个像素在全部视差进行计算的情况下，所述数据的大小可以为单个像素在全部视差下的第一代价聚合结果的大小；在单次计算可以针对单个像素在部分视差进行计算的情况下，所述数据的大小可以为单个像素在部分视差下的第一代价聚合结果的大小；在单次计算可以针对多个像素在全部视差下进行计算的情况下，所述数据的大小可以为多个像素在全部视差下的第一代价聚合结果的大小。
在一个实施例中，所述内部存储器可以包括第一内部存储器。所述第一内部存储器可以为单端口内部存储器。对于所述第一内部存储器来说，采用单端口内部存储器有利于节省成本。或者，第一内部存储器可以为作为单端口内部存储器使用的其他类型内部存储器(例如，双端口内部存储器)。本实施例的方法还可以包括：在第一时钟周期内，从所述第一内部存储器的目标存储空间中读取所述第二数据以及与所述第二数据相邻的第三数据，并缓存所述第一组计算单元计算得到的所述第一数据。其中，所述第二数据用于所述第二组计算单元在所述第一时钟周期内计算第三代价聚合结果；所述第三数据为所述第二像素的第一代价聚合结果的数据，所述第三数据用于所述第二组计算单元在第二时钟周期内计算第三代价聚合结果；所述第二时钟周期为所述第一时钟周期的下一个时钟周期。
基于此，可以实现通过在第一时钟周期内单次读取第一内部存储器，可以读取到用于第二组计算单元在相邻的第一时钟周期和第二时钟周期内计算第三代价聚合结果的第一代价聚合结果，即每隔一个时钟周期从第一内部存 储器中进行一次读取，有利于减少针对第一内部存储器的读取次数，提高读取效率。
例如，如图8所示，在时钟周期1内，可以从第一内部存储器中读取用于第二组计算单元在时钟周期1内进行计算的第二像素的第一代价聚合结果的数据K1，以及用于第二组计算单元在时钟周期2内进行计算的第二像素的第一代价聚合结果的数据K2；之后，在时钟周期3内，可以从第一内部存储器中读取用于第二组计算单元在时钟周期3内进行计算的第二像素的第一代价聚合结果的数据K3，以及用于第二组计算单元在时钟周期4内进行计算的第二区块中第二像素的第一代价聚合结果的数据K4；……，依次类推。其中，第二像素即为第二区块中的像素，时钟周期1、时钟周期3即为第一时钟周期，时钟周期2、时钟周期4即为第二时钟周期，数据K1和数据K3为前述的第二数据，数据K2和数据K4为前述的第三数据。
进一步的，本实施例的方法还可以包括：在所述第二时钟周期内，将所述第一数据和第四数据写入所述第一内部存储器的所述目标存储空间中；所述第四数据为在所述第二时钟周期内所述第一组计算单元计算得到的所述第一像素的第一代价聚合结果的数据。由此，可以实现通过在第二时钟周期内单次写入第一内部存储器，可以将第一组计算单元在第一时钟周期内和第二时钟周期内计算得到的第一代价聚合结果写入第一内部存储器，即每隔一个时钟周期向第一内部存储器中进行一次写入，有利于减少针对第一内部存储器的写入次数，提高读取效率。另外，通过上述在第一时钟周期内读取两个数据，在第二时钟周期内写入两个数据的方式，使得无需占用用于计算单元计算代价聚合结果之外的其他时钟周期进行数据读写，确保第一组计算单元和第二组计算单元在每个时钟周期均能够进行计算，有利于提高处理效率，减少处理时长。
例如，参考图9a，在图8的基础上，在时钟周期1内，可以从第一内部存储器中读取用于第二组计算单元在时钟周期1内进行计算的第二像素的第一代价聚合结果的数据K1，以及用于第一组计算单元在时钟周期2内进行计算的第二像素的第一代价聚合结果的数据K2，第一组计算单元计算得到第一像素的第一代价聚合结果的数据K5，将数据K5进行缓存，并且，第二组计算单元基于数据K1进行第三代价结果的计算。之后，在时钟周期2内，第一组计算单元计算得到第一像素的第一代价聚合结果的数据K6，将数据K5和K6写入原本 由数据K1和数据K2占用的目标存储空间，并且，第二组计算单元基于数据K2进行第三聚合代价结果的计算；之后，在时钟周期3内，可以从第一内部存储器中读取用于第二组计算单元在时钟周期3内进行计算的第二像素的第一代价聚合结果的数据K3，以及用于第一组计算单元在时钟周期4内进行计算的第二像素的第一代价聚合结果的数据K4，第一组计算单元计算得到第一像素的第一代价聚合结果的数据K7，将数据K7进行缓存，并且，第二组计算单元基于数据K3进行第三代价结果的计算。之后，在时钟周期4内，第一组计算单元计算得到第一像素的第一代价聚合结果的数据K8，将数据K7和K8写入原本由数据K3和数据K4占用的目标存储空间，并且，第二组计算单元基于数据K4进行第三代价结果的计算；……，依次类推。其中，第一像素即为第一区块中的像素，第二像素即为第二区块中的像素，数据K5和数据K7即为前述的第一数据，数据K1和数据K3即为前述的第二数据，数据K2和数据K4即为前述的第三数据，数据K6和数据K8即为前述的第四数据。
然而，本申请并非限于此。在另一实施方式中，所述内部存储器可以包括第一内部存储器，且所述第一内部存储器可以为双端口内部存储器。例如，如图9b所示，在时钟周期1内，可以从第一内部存储器中读取用于第二组计算单元在时钟周期1内进行计算的第二像素的第一代价聚合结果的数据K1，并将第一组计算单元在时钟周期1内计算得到第一像素的第一代价聚合结果的数据K5写入第一内部存储器中原本由数据K1占用的目标存储空间；之后，在时钟周期2内，可以从第一内部存储器中读取用于第二组计算单元在时钟周期2内进行计算的第二像素的第一代价聚合结果的数据K2，并将第一组计算单元在时钟周期2内计算得到第一像素的第一代价聚合结果的数据K6写入第一内部存储器中原本由数据K2占用的目标存储空间；之后，在时钟周期3内，可以从第一内部存储器中读取用于第二组计算单元在时钟周期3内进行计算的第二像素的第一代价聚合结果的数据K3，并将第一组计算单元在时钟周期3内计算得到第一像素的第一代价聚合结果的数据K7写入第一内部存储器中原本由数据K3占用的目标存储空间；之后，在时钟周期4内，可以从第一内部存储器中读取用于第二组计算单元在时钟周期4内进行计算的第二像素的第一代价聚合结果的数据K4，并将第一组计算单元在时钟周期4内计算得到第一像素的第一代价聚合结果的数据K8写入第一内部存储器中原本由数据K4占用的目标存储空间；……，依次类推。其中，第一像素即为第一区块中的像素，第 二像素即为第二区块中的像素，时钟周期1、时钟周期3即为第一时钟周期，时钟周期2、时钟周期4即为第二时钟周期。数据K1和数据K3即为第二数据，数据K2和数据K4即为第三数据，数据K5和数据K7即为第一数据，数据K6和数据K8即为第四数据。
需要说明的是，为了便于表示在同一时钟周期内读取和写入的数据占第一内部存储器的同一存储空间，图9b中第一内部存储器中存储的灰色线条填充表示的数据表示在对应时钟周期需要读取的数据。
在单次计算可以针对单个像素在部分视差进行计算的情况下，假设单个区块的大小为32×32，最大视差等于128，单个时钟周期计算的视差数量为Dper，Dper小于128，第一代价聚合结果的位宽为16bit，则单个时钟周期内第二组计算单元计算消耗的第二像素的第一代价聚合结果的大小为Dper×16(即第二数据的大小为Dper×16)，第一组计算单元计算得到的第一像素的第一代价聚合结果的大小也为Dper×16(即第一数据的大小也为Dper×16)，由此可以将第一内部存储器的位宽设置为Dper×16×2，位宽增加一倍，深度减小50％，容量不变。
方式二
第二时钟周期内所述第一组计算单元计算得到的第一数据，被存入所述内部存储器中第一时钟周期内所述第二组计算单元计算使用的第二数据对应的存储空间；其中，所述第一数据为所述第一像素的第一代价聚合结果的数据，所述第二数据为所述第二像素的第一代价聚合结果的数据；所述第二时钟周期为所述第一时钟周期的下一个时钟周期。由此，可以实现一时钟周期内计算得到的第一像素的第一代价聚合结果，占用相邻时钟周期内计算使用的第二像素的第一代价聚合结果。需要说明的是，关于数据的具体说明可以参见前述方式一中的相关描述，在此不再赘述。
在一个实施例中，所述内部存储器可以包括第一内部存储器和第二内部存储器。所述第一内部存储器和所述第二内部存储器可以为单端口内部存储器，通过所述第一内部存储器和所述第二内部存储器均是单端口内部存储器，有利于节省成本；或者，第一内部存储器和第二内部存储器可以为作为单端口内部存储器使用的其他类型内部存储器，本实施例的方法还可以包括：在第一时钟周期内，将所述第一组计算单元计算得到的所述第一数据写入所述第一内部存储器中，并从所述第二内部存储器的目标存储空间中读取所述第 二数据；所述第二数据用于所述第二组计算单元在所述第一时钟周期内计算第三代价聚合结果。
基于此，可以实现在一个时钟周期内从一个内部存储器中读取到用于第二组计算单元在该周期内进行计算的第二像素的第一代价聚合结果，并将第一计算单元在该周期计算得到的第一像素的第一代价聚合结果存储至另一内部存储器中，即在同一时钟周期内分别针对不同的内部存储器进行读或写，有利于降低对于内部存储器的端口数量要求，降低成本。另外，能够使得针对第一代价聚合结果的读或写，无需占用用于计算单元计算代价聚合结果之外的其他时钟周期进行数据读写，确保第一组计算单元和第二组计算单元在每个时钟周期均能够进行计算，有利于提高处理效率，减少处理时长。
进一步的，本实施例的方法还可以包括：在第二时钟周期内，从所述第一内部存储器中读取第三数据，并将所述第二组计算单元计算得到的第四数据写入所述第二内部存储器的所述目标存储空间中。其中，所述第三数据为所述第二像素的第一代价聚合结果的数据，所述第三数据用于所述第二组计算单元在第二时钟周期内计算得到的第三代价结果；所述第四数据为在所述第二时钟周期内所述第一组计算单元计算得到的所述第一像素的第一代价聚合结果的数据；所述第二时钟周期为所述第一时钟周期的下一个时钟周期。由此，可以实现第一像素的第一代价聚合结果和第二像素的第一代价聚合结果，共用两个内部存储器的存储空间。
例如，如图10所示，在时钟周期1内，可以从第二内部存储器中读取用于第二组计算单元在时钟周期1内进行计算的第二像素的第一代价聚合结果的数据K1，并将第一组计算单元在时钟周期1内计算得到第一像素的第一代价聚合结果的数据K5写入第一内部存储器中；之后，在时钟周期2内，可以从第一内部存储器中读取用于第二组计算单元在时钟周期2内进行计算的第二像素的第一代价聚合结果的数据K2，并将第一组计算单元在时钟周期2内计算得到第一像素的第一代价聚合结果的数据K6写入第二内部存储器中原本由数据K1占用的目标存储空间；之后，在时钟周期3内，可以从第二内部存储器中读取用于第二组计算单元在时钟周期3内进行计算的第二像素的第一代价聚合结果的数据K3，并将第一组计算单元在时钟周期3内计算得到第一像素的第一代价聚合结果的数据K7写入第一内部存储器中原本由数据K2占用的目标存储空间；之后，在时钟周期4内，可以从第一内部存储器中读取用于第二组计 算单元在时钟周期4内进行计算的第二像素的第一代价聚合结果的数据K4，并将第一组计算单元在时钟周期4内计算得到第一像素的第一代价聚合结果的数据K8写入第二内部存储器中原本由数据K3占用的目标存储空间；……，依次类推。其中，第一像素即为第一区块中的像素，第二像素即为第二区块中的像素，时钟周期1、时钟周期3即为第一时钟周期，时钟周期2、时钟周期4即为第二时钟周期，数据K1和数据K3即为第二数据，数据K2和数据K4即为第三数据，数据K5和数据K7即为第一数据，数据K6和数据K8即为第四数据。
本申请实施例中，所述第一代价聚合结果、所述第二代价聚合结果以及所述第三代价聚合结果具体可以包括像素在视差范围内各视差下的聚合结果。示例性的，某一像素p在视差d下的第一代价聚合结果可以表示为L 第一方向(p,d)，像素p在视差d下的第二代价聚合结果可以表示为L 第二方向(p,d)，像素p在视差d下的第三代价聚合结果可以表示为L 第一方向+第二方向(p,d)。
基于此，所述基于所述多个第二像素在第一方向上的第一代价聚合结果和所述多个第二像素在所述第二方向上的第二代价聚合结果，计算得到所述多个第二像素的第三代价聚合结果，具体可以包括：针对于所述多个第二像素中的各像素，将所述像素在各视差下的第一代价聚合结果与所述像素在各所述视差下的第二代价聚合结果对应相加，得到所述像素在各所述视差下的第三代价聚合结果。需要说明的是，其中的第一代价聚合结果和第二代价聚合结果均为代价聚合的中间结果，第三代价聚合结果为最终代价聚合结果。
以视差范围为0到127，第一方向为L0_pd-L3_pd，第二方向为L4_pd-L7_pd为例，某一像素p在视差d下的第一代价聚合结果L 第一方向(p,d)＝L L0_pd(p,d)+L L1_pd(p,d)+L L2_pd(p,d)+L L3_pd(p,d)，像素p在视差d下的第二代价聚合结果L 第二方向(＝L L4_pd(p,dd)+L L5_pd(p,d)+L L6_pd(p,d)+L L7_pd(p,d)，像素p在视差d下的第三代价聚合结果L 第一方向+第二方向(p,d)＝L L0_pd(p,d)+L L1_pd(p,d)+L L2_pd(p,d)+L L3_pd(p,d)+L L4_pd(p,d)+L L5_pd(p,d)+L L6_pd(p,d)+L L7_pd(p,d)。其中，L Li_pd(p,d)，i等于0至7，可以表示像素p在方向Li_pd上的代价聚合。
需要说明的是，对于第三代价聚合结果的存储方式，可以根据需求灵活实现，在一个实施例中，可以将第三代价聚合结果存储在集成电路的外部存储器例如DDR中。
可选的，在计算获得第三聚合代价结果之后，还可以基于第三代价聚合结果确定最终视差。基于此，本实施例的方法还可以包括：将所述像素在所述视差范围内聚合结果最小的视差，作为所述像素的最终视差。例如，假设d等于0至127中10时，L 第一方向+第二方向(p,d)的值最小，则10即为像素p的最终视差。在一个实施例中，可以采用所述第二组计算单元计算像素的最终视差。
本申请实施例提供的图像处理方法，通过采用第一组计算单元，计算得到位于多个区块中的第一区块的多个第一像素在第一方向上的代价聚合结果，采用第二组计算单元，计算得到位于多个区块的第二区块中的多个第二像素在第二方向上的代价聚合结果，以及基于多个第二像素在第一方向上的第一代价聚合结果和多个第二像素在第二方向上的第二代价聚合结果，计算得到多个第二像素的第三代价聚合结果，其中，第一区块和第二区块相邻且第二区块位于第一区块之前，实现了在计算位于第一区块中的第一像素在第一方向上的第一代价聚合结果的同时，进行位于第二区块中的第二像素在第二方向上的第二代价聚合结果以及第二像素的最终代价聚合结果的计算，有利于减少处理耗时，提高处理帧率，并且能够减少需要存储的中间数据，降低对于存储器的容量要求，有利于节省成本。
图11为本申请又一实施例提供的图像处理方法的流程示意图。本实施例在前述实施例的基础上，主要描述了采用第一组计算单元计算得到第一代价聚合结果的一种可选实现方式。如图11所示，本实施例的方法可以包括：
步骤111，采用第一组计算单元，从存储器中读取所述多个第一像素中部分像素在多个第一方向中的至少一个所述第一方向上的单方向代价聚合结果，并基于所述部分像素的所述第一代价聚合结果计算得到所述多个第一像素中其他像素的第一代价聚合结果；所述部分像素在所述多个第一方向中的至少一个所述第一方向上的单方向代价聚合结果是通过先前处理获得并存储在所述存储器中。在一个实施方式中，所述单方向代价聚合结果为L0_pd至L7_pd中的一个。
步骤112，采用第二组计算单元，计算得到位于所述多个区块中第二区块中的多个第二像素在第二方向上的第二代价聚合结果。
需要说明的是，步骤112的具体内容可以参见前述实施例的相关描述，在此不再赘述。
在步骤111中，所述单方向代价聚合结果是指在单个方向上的代价聚合结 果。以至少一个所述第一方向为L0_pd方向和L1_pd方向为例，可以从存储器中读取多个第一像素中部分像素在L0_pd方向上的代价聚合结果，以及，从存储器中读取多个第一像素中部分像素在L1_pd方向上的代价聚合结果。
所述先前处理可以基于前述公式2完成。以图3所示的8个方向为例，假设第一方向包括图3中的左上部分的4个方向，即L0_pd至L3_pd，可以基于前述公式2给出的计算某一像素p在r方向上的单方向代价聚合结果，计算目标图像中所有像素分别在L0_pd至L3_pd方向上分别的单方向代价聚合结果，并将部分像素的L0_pd至L3_pd中至少一个方向的单方向代价聚合结果存储在存储器中。
可以看出，在先前处理过程中，第一区块中所有像素在各第一方向上的单方向代价聚合结果均能够计算获得，只是选取了其中部分像素在至少一个第一方向上的单方向代价聚合结果进行保存。之后，可以基于存储器中保存的第一区块中部分像素在至少一个第一方向上的单方向代价聚合结果，恢复得到其他像素在各第一方向上的单方向代价聚合结果，以基于其他像素在各第一方向上的单方向代价聚合结果计算得到其他像素的第一代价聚合结果。
在一个实施例中，为了尽可能的减少对于集成电路的内部存储器的占用，所述存储器可以为所述集成电路的外部存储器。当然，在其他实施例中，所述存储器还可以为其他存储器，本申请对此不做限定。
这里，介绍一下可以基于位于第一区块中的多个第一像素中部分像素的第一代价聚合结果，计算得到位于第一区块中的多个第一像素中其他像素的第一代价聚合结果的理论依据。
通过公式2可以看出，对于图像中任一像素在单一方向上的代价聚合结果来说，其由两部分来决定。具体来说，该代价聚合结果包括该像素的匹配代价以及在该方向上的上一个像素的代价聚合结果。基于此，以方向L0_pd方向为例，可以将公式2重写为公式3来描述代价聚合的过程：
L0_pd i＝f(C i,L0_pd i-1)  (公式3)
其中，i为图像中的当前像素，C i表示当前像素i的匹配代价，L0_pd i-1表示在L0_pd方向上的上一个像素i-1的代价聚合结果，L0_pd i表示L0_pd方向的当前像素i的代价聚合结果。
通过公式3可以看出，某一像素p在某一方向上的代价聚合结果可以基于该像素p在该方向上的上一个像素的代价聚合结果确定，由此能够基于位 于第一区块中的多个第一像素中部分像素的第一代价聚合结果，计算得到位于第一区块中的多个第一像素中其他像素的第一代价聚合结果。
以第一方向为图3中左上部分的L0_pd至L3_pd，共4个方向为例，像素在第一方向上的第一代价聚合结果可以表示为：
Figure PCTCN2020100105-appb-000002
在一个实施例中，所述第一方向包括向下方向，所述部分像素包括所述多个第一像素中位于所述第一区块上边缘的像素。其中，所述向下方向可以包括垂直向下方向和斜向下方向。以图3所示的8个方向为例，垂直向下方向例如可以为图3中的L1_pd方向，斜向下方向例如可以为图3中的L0_pd方向和L2_pd方向。
参考图12，在第一方向包括L1_pd方向情况下，某一像素A在L1_pd方向上的代价聚合结果可以基于像素A在L1_pd方向上的上一像素即像素A3在L1_pd方向上的代价聚合结果确定，像素A3在L1_pd方向上的代价聚合结果可以基于像素A3在L1_pd方向上的上一像素即像素A2在L1_pd方向上的代价聚合结果确定，像素A2在L1_pd方向上的代价聚合结果可以基于像素A1在L1_pd向上的代价聚合结果确定，而像素A1在L1_pd方向上的代价聚合结果可以基于区块上边缘的像素A00在L1_pd方向上的代价聚合结果确定。可以看出，第一区块上边缘的像素在垂直向下方向上的代价聚合结果可以用于确定其他像素在垂直向下方向上的代价聚合结果。因此，在一实施方式中，存储第一区块上边缘行(例如，像素A00所在的行)的像素在L1_pd方向上的代价聚合结果，以用于确定其他像素在L1_pd方向上的代价聚合结果。其中，其他像素位于第一区块的内部。也就是说，其他像素不位于区块的边缘。
继续参考图12，在第一方向包括L2_pd方向情况下，某一像素A在L2_pd方向上的代价聚合结果可以基于像素A在L2_pd方向上的上一像素即像素A6在L2_pd方向上的代价聚合结果确定，像素A6在L2_pd方向上的代价聚合结果可以基于像素A6在L2_pd方向上的上一像素即像素A5在L2_pd方向上的代价聚合结果确定，像素A5在L2_pd方向上的代价聚合结果可以基于像素A4在L2_pd向上的代价聚合结果确定，而像素A4在L2_pd方向上的代价聚合结果可以基于区块上边缘的像素A01在L2_pd方向上的代价聚合结果确定。可以看出，第一区块上边缘的像素在斜向下方向上的代价聚合结果可以用于确定其他像 素在斜向下方向上的代价聚合结果。因此，在一实施方式中，存储第一区块上边缘行(例如，像素A01所在的行)的像素在L2_pd方向上的代价聚合结果，以用于确定其他像素在L2_pd方向上的代价聚合结果。其中，其他像素位于第一区块的内部。也就是说，其他像素不位于区块的边缘。
基于与L2_pd方向类似的原理，针对L0_pd方向，在一实施方式中，存储第一区块上边缘行的像素在L0_pd方向上的代价聚合结果，以用于确定其他像素在L0_pd方向上的代价聚合结果。其中，其他像素位于第一区块的内部。也就是说，其他像素不位于区块的边缘。
在另一个实施例中，所述第一方向包括向右方向，所述部分像素包括所述多个第一像素中位于所述第一区块左边缘的像素。其中，所述向右方向可以包括水平向右方向和斜向右方向。以图3所示的8个方向为例，水平向右方向例如可以为图3中的L3_pd方向，斜向右方向例如可以为图3中的L2_pd方向和L4_pd方向。
参考图13，在第一方向包括L3_pd方向情况下，某一像素B在L3_pd方向上的代价聚合结果可以基于像素B在L3_pd方向上的上一像素即像素B3在L3_pd方向上的代价聚合结果确定，像素B3在L3_pd方向上的代价聚合结果可以基于像素B3在L3_pd方向上的上一像素即像素B2在L3_pd方向上的代价聚合结果确定，像素B2在L3_pd方向上的代价聚合结果可以基于像素B1在L3_pd向上的代价聚合结果确定，而像素B1在L3_pd方向上的代价聚合结果可以基于区块上边缘的像素B00在L3_pd方向上的代价聚合结果确定。可以看出，第一区块左边缘的像素在水平向右方向上的代价聚合结果可以用于确定其他像素在水平向右方向上的代价聚合结果。因此，在一实施方式中，存储第一区块左边缘行(例如，像素B00所在的行)的像素在L3_pd方向上的代价聚合结果，以用于确定其他像素在L3_pd方向上的代价聚合结果。其中，其他像素位于第一区块的内部。也就是说，其他像素不位于区块的边缘。
继续参考图13，在第一方向包括L2_pd方向情况下，某一像素B在L2_pd方向上的代价聚合结果可以基于像素B在L2_pd方向上的上一像素即像素B6在L2_pd方向上的代价聚合结果确定，像素B6在L2_pd方向上的代价聚合结果可以基于像素B6在L2_pd方向上的上一像素即像素B5在L2_pd方向上的代价聚合结果确定，像素B5在L2_pd方向上的代价聚合结果可以基于像素B4在L2_pd向上的代价聚合结果确定，而像素B4在L2_pd方向上的代价聚合结果可 以基于区块左边缘的像素B01在L2_pd方向上的代价聚合结果确定。可以看出，第一区块左边缘的像素在斜向右方向上的代价聚合结果可以用于确定其他像素在斜向右方向上的代价聚合结果。因此，在一实施方式中，存储第一区块左边缘行(例如，像素B01所在的行)的像素在L2_pd方向上的代价聚合结果，以用于确定其他像素在L2_pd方向上的代价聚合结果。其中，其他像素位于第一区块的内部。也就是说，其他像素不位于区块的边缘。
基于与L2_pd方向类似的原理，针对L4_pd方向，在一实施方式中，存储第一区块左边缘行的像素在L4_pd方向上的代价聚合结果，以用于确定其他像素在L4_pd方向上的代价聚合结果。其中，其他像素位于第一区块的内部。也就是说，其他像素不位于区块的边缘。
在又一个实施例中，所述第一方向包括向上方向，所述部分像素包括所述多个第一像素中位于所述第一区块下边缘的像素。其中，所述向上方向可以包括垂直向上方向和斜向上方向。以图3所示的8个方向为例，垂直向上方向例如可以为图3中的L5_pd方向，斜向上方向例如可以为图3中的L4_pd方向和L6_pd方向。
参考图14，在第一方向包括L5_pd方向情况下，某一像素C在L5_pd方向上的代价聚合结果可以基于像素C在L5_pd方向上的上一像素即像素C3在L5_pd方向上的代价聚合结果确定，像素C3在L5_pd方向上的代价聚合结果可以基于像素C3在L5_pd方向上的上一像素即像素C2在L5_pd方向上的代价聚合结果确定，像素C2在L5_pd方向上的代价聚合结果可以基于像素C1在L5_pd向上的代价聚合结果确定，而像素C1在L5_pd方向上的代价聚合结果可以基于区块上边缘的像素C00在L5_pd方向上的代价聚合结果确定。可以看出，第一区块下边缘的像素在垂直向上方向上的代价聚合结果可以用于确定其他像素在垂直向上方向上的代价聚合结果。因此，在一实施方式中，存储第一区块下边缘行(例如，像素C00所在的行)的像素在L5_pd方向上的代价聚合结果，以用于确定其他像素在L5_pd方向上的代价聚合结果。其中，其他像素位于第一区块的内部。也就是说，其他像素不位于区块的边缘。
继续参考图14，在第一方向包括L4_pd方向情况下，某一像素C在L4_pd方向上的代价聚合结果可以基于像素C在L4_pd方向上的上一像素即像素C6在L4_pd方向上的代价聚合结果确定，像素C6在L4_pd方向上的代价聚合结果可以基于像素C6在L4_pd方向上的上一像素即像素C5在L4_pd方向上的 代价聚合结果确定，像素C5在L4_pd方向上的代价聚合结果可以基于像素C4在L4_pd向上的代价聚合结果确定，而像素C4在L4_pd方向上的代价聚合结果可以基于区块上边缘的像素C00在L4_pd方向上的代价聚合结果确定。可以看出，第一区块下边缘的像素在斜向上方向上的代价聚合结果可以用于确定其他像素在斜向上方向上的代价聚合结果。因此，在一实施方式中，存储第一区块下边缘行(例如，像素C01所在的行)的像素在L4_pd方向上的代价聚合结果，以用于确定其他像素在L4_pd方向上的代价聚合结果。其中，其他像素位于第一区块的内部。也就是说，其他像素不位于区块的边缘。
基于与L4_pd方向类似的原理，针对L6_pd方向，在一实施方式中，存储第一区块下边缘行的像素在L6_pd方向上的代价聚合结果，以用于确定其他像素在L6_pd方向上的代价聚合结果。其中，其他像素位于第一区块的内部。也就是说，其他像素不位于区块的边缘。
在又一个实施例中，所述第一方向包括向左方向，所述部分像素包括所述多个第一像素中位于所述第一区块右边缘的像素。其中，所述向左方向可以包括水平向左方向和斜向左方向。以图3所示的8个方向为例，水平向左方向例如可以为图3中的L7_pd方向，斜向左方向例如可以为图3中的L0_pd方向和L6_pd方向。
参考图15，在第一方向包括L7_pd方向情况下，某一像素D在L7_pd方向上的代价聚合结果可以基于像素D在L7_pd方向上的上一像素即像素D3在L7_pd方向上的代价聚合结果确定，像素D3在L7_pd方向上的代价聚合结果可以基于像素D3在L7_pd方向上的上一像素即像素D2在L7_pd方向上的代价聚合结果确定，像素D2在L7_pd方向上的代价聚合结果可以基于像素D1在L7_pd向上的代价聚合结果确定，而像素D1在L7_pd方向上的代价聚合结果可以基于区块上边缘的像素D00在L7_pd方向上的代价聚合结果确定。可以看出，第一区块右边缘的像素在水平向左方向上的代价聚合结果可以用于确定其他像素在水平向左方向上的代价聚合结果。因此，在一实施方式中，存储第一区块右边缘行(例如，像素D00所在的行)的像素在L7_pd方向上的代价聚合结果，以用于确定其他像素在L7_pd方向上的代价聚合结果。其中，其他像素位于第一区块的内部。也就是说，其他像素不位于区块的边缘。
继续参考图15，在第一方向包括L6_pd方向情况下，某一像素D在L6_pd方向上的代价聚合结果可以基于像素D在L6_pd方向上的上一像素即像素D6 在L6_pd方向上的代价聚合结果确定，像素D6在L6_pd方向上的代价聚合结果可以基于像素D6在L6_pd方向上的上一像素即像素D5在L6_pd方向上的代价聚合结果确定，像素D5在L6_pd方向上的代价聚合结果可以基于像素D4在L6_pd向上的代价聚合结果确定，而像素D4在L6_pd方向上的代价聚合结果可以基于区块右边缘的像素D01在L6_pd方向上的代价聚合结果确定。可以看出，第一区块右边缘的像素在斜向左方向上的代价聚合结果可以用于确定其他像素在斜向左方向上的代价聚合结果。因此，在一实施方式中，存储第一区块右边缘行(例如，像素D01所在的行)的像素在L6_pd方向上的代价聚合结果，以用于确定其他像素在L6_pd方向上的代价聚合结果。其中，其他像素位于第一区块的内部。也就是说，其他像素不位于区块的边缘。
基于与L7_pd方向类似的原理，针对L0_pd方向，在一实施方式中，存储第一区块右边缘行的像素在L0_pd方向上的代价聚合结果，以用于确定其他像素在L0_pd方向上的代价聚合结果。其中，其他像素位于第一区块的内部。也就是说，其他像素不位于区块的边缘。
基于前述图12至图15的分析可知，以第一方向为图3中左上部分的L0_pd至L3_pd为例，如图16所示，第一区块中的部分像素具体可以包括黑色框表示的位于第一区块上边缘的像素，以及灰色框表示的位于第一区块左边缘的像素。第一区块中的其他像素具体可以为白色框表示的像素。需要说明的是，图16中像素点不同的填充色，表示在存储器中需要存储的代价聚合结果对应的方向不同。针对图16中黑色框位置的像素即第一区块上边缘的像素，存储设备中需要存储分别在L0_pd、L1_pd、L2_pd三个方向上的代价聚合结果；针对图16中灰色框位置的像素即第一区块左边缘的像素，存储设备中需要存储分别在L2_pd、L3_pd两个方向上的代价聚合结果。针对图16中灰色框与黑色框交叠位置的像素，存储设备中需要存储分别存储在L0_pd、L1_pd、L2_pd和L3_pd四个方向上的代价聚合结果。
本申请实施例中，可以采用逐行或者逐点的方式来确定第一区块中其他像素在各第一方向上的单方向代价聚合结果。需要说明的是，在一个实施方式中，当第二方向为L0_pd～L7_pd时，可以采用与上述方式相似的方式来获得位于区块内部的像素在第二方向上的代价聚合结果。为求简洁，不再赘述。
对应于图16，在采用逐行的方式时可以根据第一区块中上边缘和左边缘的像素在至少一个第一方向上的单方向代价聚合结果，并行按照逐行和列的 方式确定第一区块中其他像素在各第一方向上的单方向代价聚合结果。
在实际应用中，参考图17，可以由第1行中的像素在L0_pd、L1_pd、L2_pd方向上分别的单方向代价匹配结果和第2行中像素的匹配代价(也即前文所述的matching cost)，分别计算第2行中的像素在L0_pd、L1_pd、L2_pd方向上分别的单方向匹配代价结果。还可以通过第1列中的像素在L3_pd方向上的单方向代价聚合结果和第2列中像素的匹配代价，计算第2列中像素在L3_pd方向上的单方向代价聚合结果。
L0_pd至L3_pd不同方向上的单方向代价聚合结果的计算过程之间是相互独立的，因此可以并行计算不同方向上的单方向代价聚合结果。例如，一个时钟周期内可以完成第2行中的像素在L0_pd、L1_pd、L2_pd方向上分别的单方向代价聚合结果以及第2列中的像素在L3_pd方向上的单方向代价聚合结果的恢复，对于图17所示的第一区块来说，仅16个周期即可完成第一区块中其他像素在各第一方向上的单方向代价聚合结果的恢复计算。对于逐行恢复其他像素的单方向代价聚合结果的方式来说，计算速度较快。
值得说明的是，以图17所示的第一区块为例，假设每个单方向代价匹配结果的数据量为16比特，那么可以在集成电路的内部存储器中开辟4块16×16×16比特的存储空间，用于存储第一区块中第一像素在L0_pd-L3_pd方向上的单方向代价匹配结果，其中4对应L0_pd-L3_pd共4个方向。需要说明的是，由于需要用到一个第一区块中多个第一像素分别在L0_pd-L3_pd方向上的单方向代价匹配结果，这其中包括第一区块中的多个第一像素中的其他像素，同时也包括第一区块中的多个第一像素中的部分像素。
对应于图16，在采用逐点的方式时可以根据第一区块中上边缘和左边缘的像素在至少一个第一方向上的单方向代价聚合结果，按照逐点的方式确定第一区块中其他像素在各第一方向上的单方向代价聚合结果。
在实际应用中，参考图17，假设第一区块左上角的像素的位置为(0,0)，可以先由位置(1,0)的像素在L3_pd方向上的单方向代价匹配结果计算出位置(1,1)的像素在L3_pd方向上的单方向代价匹配结果，通过位置(0,1)的像素分别在L0_pd、L1_pd、L2_pd方向上的单方向代价匹配结果计算出位置(1,1)的像素分别在L0_pd、L1_pd、L2_pd方向上的单方向代价匹配结果。在计算出位置(1,1)的像素分别在L0_pd至L3_pd方向上的单方向代价匹配结果之后，可以将它们相加。
可以理解的是，在计算出位置(1,1)的像素在L0_pd至L3_pd方向上的单方向代价匹配结果的相加结果之后，可以将该相加结果存储到集成电路的内部存储器中。通过这样的方式，最后能够计算出第一区块中部分像素和其他像素在L0_pd至L3_pd方向上的单方向代价匹配结果的相加结果(即第一代价聚合结果)，并将第一代价聚合结果存储到存储器中。仍以图17所示的第一区块为例，假设每个单方向代价匹配结果的数据量为16比特，那么可以在集成电路的内部存储器中开辟1块16×16×16比特的存储空间，用于存储第一区块中第一像素的第一代价聚合结果。相比于逐行恢复第一区块的单方向代价聚合结果的方式来说，采用逐点恢复的方式可以直接恢复出单个第一像素分别在L0_pd至L3_pd方向上的单方向代价匹配结果，因此可以直接将这4个方向上的单方向代价匹配结果相加得到单个第一像素的第一代价聚合结果，将第一代价聚合结果存储到集成电路的内部存储器上。而采用逐行恢复的方式，无法同时将单个第一像素分别在L0_pd至L3_pd方向上的单方向代价匹配结果都恢复出，因此需要将单个第一像素分别在L0_pd至L3_pd方向上的单方向代价匹配结果都存储在内部存储器中。从而，对于逐点恢复的方式来说，需要占用的存储空间较小。
需要说明的是，在第一区块的上边缘的像素是目标图像的上边缘的像素情况下，第一区块的上边缘的像素在向下方向上的代价聚合结果是默认最大值，因此可以无需存储在存储设备中。在第一区块的左边缘的像素是目标图像的左边缘的像素情况下，第一区块的左边缘的像素在向右方向上的代价聚合结果是默认最大值，因此可以无需存储在存储设备中。在第一区块的下边缘的像素是目标图像的下边缘的像素情况下，第一区块的下边缘的像素在向上方向上的代价聚合结果是默认最大值。在第一区块的右边缘的像素是目标图像的右边缘的像素情况下，第一区块的右边缘的像素在向左方向上的代价聚合结果是默认最大值。在一个实施方式中，上述默认最大值可以不采用与计算得到的代价聚合结果的存储方式相同的存储方式进行存储。例如，无需将默认最大值存储在计算得到的代价聚合结果对应的存储设备中。
基于此，针对于图5所示的4个区块，假设图5所示的4个区块组成了一张完整的目标图像，且第一方向包括L0-pd至L3-pd，则第一区块中的部分像素可以为图18中黑色框位置的像素和灰色框位置的像素。参考图18，由于第一方向包括L0-pd至L3-pd，因此理论上第一区块的部分像素为第一区块的上边缘和 左边缘的像素，且应将部分像素分别在L0-pd至L3-pd方向上的单方向代价聚合结果保存在存储器中。然而，由于区块4的上边缘和左边缘的像素均是目标图像的边缘的像素，因此区块4的上边缘和左边缘的像素分别在L0-pd至L3-pd方向上的单方向代价聚合结果可以直接取默认值。由于区块3的上边缘的像素均是目标图像的上边缘的像素，因此区块3的上边缘的像素分别在L0-pd至L3-pd方向上的单方向代价聚合结果可以直接取默认值。由于区块2的左边缘的像素均是目标图像的左边缘的像素，因此区块2的左边缘的像素分别在L0-pd至L3-pd方向上的单方向代价聚合结果可以直接取默认值。另外，由于区块1的左边缘和上边缘的像素均不是目标图像的边缘的像素，因此区块2的上边缘和左边缘的像素分别在L0-pd至L3-pd方向上的单方向代价聚合结果不可以直接取默认值。
本申请实施例中，可选的，所述多个区块中目标区块中的多个像素在第一方向上的第一聚合结果，可以是在所述先前处理的过程中获得，所述目标区块为所述多个区块中经所述第二组计算单元第一个处理的区块。通过在所述先前处理的过程中计算获得所述目标区块中红的多个像素在第一方向上的第一聚合结果，可以减少第一计算单元的需要处理的区块的数量，有利于减少处理时长，提高处理帧率。
假设处理顺序为前述的区块1→区块2→区块3→区块4，在先前处理的过程中可以预先获得区块1中的多个像素在第一方向上的第一代价聚合结果，参考图19：在时间t由t0秒至t1秒的一段时间内，第一组计算单元可以计算得到位于区块2中多个像素在第一方向上的第一代价聚合结果，第二组计算单元可以计算得到位于区块1中多个像素在第二方向上的第二代价聚合结果，并基于在先前处理的过程中获得的位于区块1中多个像素在第一方向上的第一代价聚合结果，计算得到位于区块1中的多个像素的第三代价聚合结果。在时间t由t1秒至t2秒的一段时间内，第一组计算单元可以计算得到位于区块3中多个像素在第一方向上的第一代价聚合结果，第二组计算单元可以计算得到位于区块2中多个像素在第二方向上的第二代价聚合结果，并基于第一组计算单元在t0秒至t1秒的时段内计算获得的区块2中多个像素在第一方向上的第一代价聚合结果，计算得到位于区块2中的多个像素的第三代价聚合结果。在时间t由t2秒至t3秒的一段时间内，第一组计算单元可以计算得到位于区块4中多个像素在第一方向上的第一代价聚合结果，第二组计算单元可以计算得到位于区块3中 多个像素在第二方向上的第二代价聚合结果，并基于第一组计算单元在t1秒至t2秒的时段内计算获得的区块3中多个像素在第一方向上的第一代价聚合结果，计算得到位于区块3中的多个像素的第三代价聚合结果。在时间t由t3秒至t4秒的一段时间内，第二组计算单元可以计算得到位于区块4中多个像素在第二方向上的第二代价聚合结果，并基于第一组计算单元在t2秒至t3秒的时段内计算获得的区块4中多个像素在第一方向上的第一代价聚合结果，计算得到位于区块4中的多个像素的第三代价聚合结果。对比图19和图6可以看出，在先前处理过程中获得区块1中的多个像素在第一方向上的第一聚合结果，较由第一组计算单元计算得到区块1中的多个像素在第一方向上的第一聚合结果，可以缩短针对整个图像进行处理的耗时，即将耗时由t5秒缩短至t4秒。
本申请实施例提供的图像处理方法，通过采用第一组计算单元，从存储器中读取多个第一像素中部分像素在至少一个第一方向上的单方向代价聚合结果，并基于部分像素的第一代价聚合结果计算得到多个第一像素中其他像素的第一代价聚合结果，实现了基于一区块中的部分像素在至少一个方向上的单方向代价聚合结果，计算得到该区块中其他像素的第一代价聚合结果的方式，与只是基于公式2计算得到像素的第一代价聚合结果相比，扩大了计算像素的第一代价聚合结果的方法范围，提高了计算像素的第一代价聚合结果的灵活性。
图20为本申请一实施例提供的集成电路的结构示意图，如图20所示，该集成电路200可以包括：并行处理的第一组计算单元21和第二组计算单元22。
其中，所述第一组计算单元21，用于计算得到位于所述多个区块中第一区块中的多个第一像素在第一方向上的第一代价聚合结果；
所述第二组计算单元22，用于计算得到位于所述多个区块中第二区块中的多个第二像素在第二方向上的第二代价聚合结果。
本实施例提供的集成电路，可以用于执行前述方法实施例的技术方案，其实现原理和技术效果与方法实施例类似，在此不再赘述。
本申请实施例还提供一种图像处理装置，该图像处理装置可以包括如图20所示的集成电路。
图21为本申请一实施例提供的无人机的结构示意图，如图21所示，该无人机210包括机身211、设置于所述机身上的动力系统212、拍摄装置213和集成电路214；
所述动力系统212，用于为所述无人机提供动力；
所述拍摄装置213，用于在所述无人机飞行过程中进行拍摄，以获得对应不同视点的两幅图像，所述两幅图像中的目标图像被划分为多个区块，所述多个区块之间存在一定的处理顺序；
所述集成电路214包括：并行处理的第一组计算单元和第二组计算单元；
所述第一组计算单元，用于计算得到位于所述多个区块中第一区块中的多个第一像素在第一方向上的第一代价聚合结果；
所述第二组计算单元，用于计算得到位于所述多个区块中第二区块中的多个第二像素在第二方向上的第二代价聚合结果。
需要说明的是，集成电路214的具体内容参见如图20所示的实施例，在此不再赘述。
可选的，无人机210还可以包括云台215，拍摄装置213可以通过云台215设置在机身211上。当然，无人机除上述列出装置外，还可以包括其他元件或装置，这里不一一例举。
本申请实施例提供一种图像处理方法、装置及设备，通过目标图像被划分为多个区块，多个区块之间存在一定的处理顺序，采用第一组计算单元，计算得到位于多个区块中的第一区块的多个像素在第一方向上的代价聚合结果，采用第二组计算单元，计算得到位于多个区块的第二区块中的多个像素在第二方向上的代价聚合结果，其中，第一组计算单元和第二组计算单元是集成电路中并行处理的两组计算单元，实现了在计算位于第一区块中的像素在第一方向上的第一代价聚合结果的同时，进行位于第二区块中的像素在第二方向上的第二代价聚合结果的计算，与需要先对图像中像素的匹配代价在部分方向上进行聚合，再对图像中像素的匹配代价在另一部分方向上进行聚合的串行计算方式相比，可减少几乎一半的计算时间，提升近一倍的帧率，从而减少了处理耗时，提高了处理帧率。
本领域普通技术人员可以理解：实现上述各方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成。前述的程序可以存储于一计算机可读取存储介质中。该程序在执行时，执行包括上述各方法实施例的步骤；而前述的存储介质包括：ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
最后应说明的是：以上各实施例仅用以说明本申请的技术方案，而非对 其限制；尽管参照前述各实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的范围。

Claims (67)

  1. 一种图像处理方法，其特征在于，对应不同视点的两幅图像中的目标图像被划分为多个区块，所述多个区块之间存在一定的处理顺序；所述方法包括：
    采用第一组计算单元，计算得到位于所述多个区块中第一区块中的多个第一像素在第一方向上的第一代价聚合结果；
    采用第二组计算单元，计算得到位于所述多个区块中第二区块中的多个第二像素在第二方向上的第二代价聚合结果；
    其中，所述第一组计算单元和所述第二组计算单元是集成电路中并行处理的两组计算单元。
  2. 根据权利要求1所述的方法，其特征在于，所述第一区块和所述第二区块相邻，且所述处理顺序中所述第二区块位于所述第一区块之前。
  3. 根据权利要求2所述的方法，其特征在于，所述方法还包括：
    采用所述第二组计算单元，基于所述多个第二像素在第一方向上的第一代价聚合结果和所述多个第二像素在所述第二方向上的第二代价聚合结果，计算得到所述多个第二像素的第三代价聚合结果。
  4. 根据权利要求3所述的方法，其特征在于，位于所述第一区块中的第一像素的第一代价聚合结果和位于所述第二区块中的第二像素的第一代价聚合结果均存储在所述集成电路的内部存储器中。
  5. 根据权利要求4所述的方法，其特征在于，位于所述第一区块中的第一像素的第一代价聚合结果和位于所述第二区块中的第二像素的第一代价聚合结果，共用所述内部存储器中的同一块存储空间。
  6. 根据权利要求5所述的方法，其特征在于，所述第一组计算单元和所述第二组计算单元的单次计算均占用单个时钟周期；
    一时钟周期内所述第一组计算单元计算得到的第一数据，被存入所述内部存储器中该时钟周期内所述第二组计算单元计算使用的第二数据对应的存储空间；其中，所述第一数据为所述第一像素的第一代价聚合结果的数据，所述第二数据为所述第二像素的第一代价聚合结果的数据。
  7. 根据权利要求6所述的方法，其特征在于，所述内部存储器包括第一内部存储器；所述方法还包括：
    在第一时钟周期内，从所述第一内部存储器的目标存储空间中读取所述 第二数据以及与所述第二数据相邻的第三数据，并缓存所述第一组计算单元计算得到的所述第一数据；
    其中，所述第二数据用于所述第二组计算单元在所述第一时钟周期内计算第三代价聚合结果；所述第三数据为所述第二像素的第一代价聚合结果的数据，所述第三数据用于所述第二组计算单元在第二时钟周期内计算第三代价聚合结果；所述第二时钟周期为所述第一时钟周期的下一个时钟周期。
  8. 根据权利要求7所述的方法，其特征在于，所述方法还包括：
    在所述第二时钟周期内，将所述第一数据和第四数据写入所述第一内部存储器的所述目标存储空间中；所述第四数据为在所述第二时钟周期内所述第一组计算单元计算得到的所述第一像素的第一代价聚合结果的数据。
  9. 根据权利要求5所述的方法，其特征在于，所述第一组计算单元和所述第二组计算单元的单次计算均占用单个时钟周期；
    第二时钟周期内所述第一组计算单元计算得到的第一数据，被存入所述内部存储器中第一时钟周期内所述第二组计算单元计算使用的第二数据对应的存储空间；其中，所述第一数据为所述第一像素的第一代价聚合结果的数据，所述第二数据为所述第二像素的第一代价聚合结果的数据；所述第二时钟周期为所述第一时钟周期的下一个时钟周期。
  10. 根据权利要求9所述的方法，其特征在于，所述内部存储器包括第一内部存储器和第二内部存储器；所述方法还包括：
    在第一时钟周期内，将所述第一组计算单元计算得到的所述第一数据写入所述第一内部存储器中，并从所述第二内部存储器的目标存储空间中读取所述第二数据；所述第二数据用于所述第二组计算单元在所述第一时钟周期内计算第三代价聚合结果。
  11. 根据权利要求10所述的方法，其特征在于，所述方法还包括：
    在第二时钟周期内，从所述第一内部存储器中读取第三数据，并将所述第二组计算单元计算得到的第四数据写入所述第二内部存储器的所述目标存储空间中；
    其中，所述第三数据为所述第二像素的第一代价聚合结果的数据，所述第三数据用于所述第二组计算单元在第二时钟周期内计算得到的第三代价结果；所述第四数据为在所述第二时钟周期内所述第一组计算单元计算得到的所述第一像素的第一代价聚合结果的数据。
  12. 根据权利要求6-11任一项所述的方法，其特征在于，所述数据的大小包括下述中的任意一种：
    单个像素在全部视差下的第一代价聚合结果的大小，单个像素在部分视差下的第一代价聚合结果的大小，多个像素在全部视差下的第一代价聚合结果的大小。
  13. 根据权利要求1-12任一项所述的方法，其特征在于，所述第一方向为多个方向；
    所述采用第一组计算单元，计算得到位于所述第一区块中的多个第一像素在第一方向上的第一代价聚合结果，包括：
    采用第一组计算单元，从存储器中读取所述多个第一像素中部分像素在多个第一方向中的至少一个所述第一方向上的单方向代价聚合结果，并基于所述部分像素的所述单方向代价聚合结果计算得到所述多个第一像素中其他像素的第一代价聚合结果，所述部分像素的在所述多个第一方向中的至少一个所述第一方向上的单方向代价聚合结果是通过先前处理获得并存储在所述存储器中。
  14. 根据权利要求13所述的方法，其特征在于，所述多个第一方向包括向下方向，所述部分像素包括所述多个第一像素中位于所述第一区块上边缘的像素。
  15. 根据权利要求13所述的方法，其特征在于，所述多个第一方向包括向右方向，所述部分像素包括所述多个第一像素中位于所述第一区块左边缘的像素。
  16. 根据权利要求13所述的方法，其特征在于，所述多个第一方向包括向上方向，所述部分像素包括所述多个第一像素中位于所述第一区块下边缘的像素。
  17. 根据权利要求13所述的方法，其特征在于，所述多个第一方向包括向左方向，所述部分像素包括所述多个第一像素中位于所述第一区块右边缘的像素。
  18. 根据权利要求13-17任一项所述的方法，其特征在于，所述多个区块中目标区块中的多个像素在所述多个第一方向上的第一聚合结果，是在所述先前处理的过程中获得，所述目标区块为所述多个区块中经所述第二组计算单元第一个处理的区块。
  19. 根据权利要求3-18任一项所述的方法，其特征在于，所述第一代价聚合结果、所述第二代价聚合结果以及所述第三代价聚合结果包括像素在视差范围内各视差下的聚合结果。
  20. 根据权利要求19所述的方法，其特征在于，所述基于所述多个第二像素在第一方向上的第一代价聚合结果和所述多个第二像素在所述第二方向上的第二代价聚合结果，计算得到所述多个第二像素的第三代价聚合结果，包括：
    针对于所述多个第二像素中的各像素，将所述像素在各视差下的第一代价聚合结果与所述像素在各所述视差下的第二代价聚合结果对应相加，得到所述像素在各所述视差下的第三代价聚合结果。
  21. 根据权利要求20所述的方法，其特征在于，所述方法还包括：
    将所述像素在所述视差范围内聚合结果最小的视差，作为所述像素的最终视差。
  22. 根据权利要求1-21任一项所述的方法，其特征在于，所述第一方向和所述第二方向对应相反。
  23. 一种集成电路，其特征在于，对应不同视点的两幅图像中的目标图像被划分为多个区块，所述多个区块之间存在一定的处理顺序；所述集成电路包括：并行处理的第一组计算单元和第二组计算单元；
    所述第一组计算单元，用于计算得到位于所述多个区块中第一区块中的多个第一像素在第一方向上的第一代价聚合结果；
    所述第二组计算单元，用于计算得到位于所述多个区块中第二区块中的多个第二像素在第二方向上的第二代价聚合结果。
  24. 根据权利要求23所述的集成电路，其特征在于，所述第一区块和所述第二区块相邻，且所述处理顺序中所述第二区块位于所述第一区块之前。
  25. 根据权利要求24所述的集成电路，其特征在于，所述第二组计算单元，还用于基于所述多个第二像素在第一方向上的第一代价聚合结果和所述多个第二像素在所述第二方向上的第二代价聚合结果，计算得到所述多个第二像素的第三代价聚合结果。
  26. 根据权利要求25所述的集成电路，其特征在于，位于所述第一区块中的第一像素的第一代价聚合结果和位于所述第二区块中的第二像素的第一代价聚合结果均存储在所述集成电路的内部存储器中。
  27. 根据权利要求26所述的集成电路，其特征在于，位于所述第一区块中的第一像素的第一代价聚合结果和位于所述第二区块中的第二像素的第一代价聚合结果，共用所述内部存储器中的同一块存储空间。
  28. 根据权利要求27所述的集成电路，其特征在于，所述第一组计算单元和所述第二组计算单元的单次计算均占用单个时钟周期；
    一时钟周期内所述第一组计算单元计算得到的第一数据，被存入所述内部存储器中该时钟周期内所述第二组计算单元计算使用的第二数据对应的存储空间；其中，所述第一数据为所述第一像素的第一代价聚合结果的数据，所述第二数据为所述第二像素的第一代价聚合结果的数据。
  29. 根据权利要求28所述的集成电路，其特征在于，所述内部存储器包括第一内部存储器；所述第二组计算单元，还用于：
    在第一时钟周期内，从所述第一内部存储器的目标存储空间中读取所述第二数据以及与所述第二数据相邻的第三数据，并缓存所述第一组计算单元计算得到的所述第一数据；
    其中，所述第二数据用于所述第二组计算单元在所述第一时钟周期内计算第三代价聚合结果；所述第三数据为所述第二像素的第一代价聚合结果的数据，所述第三数据用于所述第二组计算单元在第二时钟周期内计算第三代价聚合结果；所述第二时钟周期为所述第一时钟周期的下一个时钟周期。
  30. 根据权利要求29所述的集成电路，其特征在于，所述第一组计算单元，还用于：
    在所述第二时钟周期内，将所述第一数据和第四数据写入所述第一内部存储器的所述目标存储空间中；所述第四数据为在所述第二时钟周期内所述第一组计算单元计算得到的所述第一像素的第一代价聚合结果的数据。
  31. 根据权利要求27所述的集成电路，其特征在于，所述第一组计算单元和所述第二组计算单元的单次计算均占用单个时钟周期；
    第二时钟周期内所述第一组计算单元计算得到的第一数据，被存入所述内部存储器中第一时钟周期内所述第二组计算单元计算使用的第二数据对应的存储空间；其中，所述第一数据为所述第一像素的第一代价聚合结果的数据，所述第二数据为所述第二像素的第一代价聚合结果的数据；所述第二时钟周期为所述第一时钟周期的下一个时钟周期。
  32. 根据权利要求31所述的集成电路，其特征在于，所述内部存储器包 括第一内部存储器和第二内部存储器；
    所述第一组计算单元，还用于在第一时钟周期内，将所述第一组计算单元计算得到的所述第一数据写入所述第一内部存储器中；
    所述第二组计算单元，还用于在所述第一时钟周期内，从所述第二内部存储器的目标存储空间中读取所述第二数据；所述第二数据用于所述第二组计算单元在所述第一时钟周期内计算第三代价聚合结果。
  33. 根据权利要求32所述的集成电路，其特征在于，所述第二组处理单元，还用于在第二时钟周期内，从所述第一内部存储器中读取第三数据；
    所述第一组计算单元，还用于在所述第二时钟周期内，将所述第二组计算单元计算得到的第四数据写入所述第二内部存储器的所述目标存储空间中；
    其中，所述第三数据为所述第二像素的第一代价聚合结果的数据，所述第三数据用于所述第二组计算单元在第二时钟周期内计算得到的第三代价结果；所述第四数据为在所述第二时钟周期内所述第一组计算单元计算得到的所述第一像素的第一代价聚合结果的数据；。
  34. 根据权利要求28-33任一项所述的集成电路，其特征在于，所述数据的大小包括下述中的任意一种：
    单个像素在全部视差下的第一代价聚合结果的大小，单个像素在部分视差下的第一代价聚合结果的大小，多个像素在全部视差下的第一代价聚合结果的大小。
  35. 根据权利要求23-34任一项所述的集成电路，其特征在于，所述第一方向为多个方向；所述第一组计算单元，具体用于：
    从存储器中读取所述多个第一像素中部分像素在多个第一方向中的至少一个所述第一方向上的单方向代价聚合结果，并基于所述部分像素的所述第一代价聚合结果计算得到所述多个第一像素中其他像素的第一代价聚合结果，所述部分像素的在所述多个第一方向中的至少一个所述第一方向上的单方向代价聚合结果是通过先前处理获得并存储在所述存储器中。
  36. 根据权利要求35所述的集成电路，其特征在于，所述多个第一方向包括向下方向，所述部分像素包括所述多个第一像素中位于所述第一区块上边缘的像素。
  37. 根据权利要求35所述的集成电路，其特征在于，所述多个第一方向 包括向右方向，所述部分像素包括所述多个第一像素中位于所述第一区块左边缘的像素。
  38. 根据权利要求35所述的集成电路，其特征在于，所述多个第一方向包括向上方向，所述部分像素包括所述多个第一像素中位于所述第一区块下边缘的像素。
  39. 根据权利要求35所述的集成电路，其特征在于，所述多个第一方向包括向左方向，所述部分像素包括所述多个第一像素中位于所述第一区块右边缘的像素。
  40. 根据权利要求35-39任一项所述的集成电路，其特征在于，所述多个区块中目标区块中的多个像素在所述多个第一方向上的第一聚合结果，是在所述先前处理的过程中获得，所述目标区块为所述多个区块中经所述第二组计算单元第一个处理的区块。
  41. 根据权利要求25-40任一项所述的集成电路，其特征在于，所述第一代价聚合结果、所述第二代价聚合结果以及所述第三代价聚合结果包括像素在视差范围内各视差下的聚合结果。
  42. 根据权利要求41所述的集成电路，其特征在于，所述第二组计算单元用于基于所述多个第二像素在第一方向上的第一代价聚合结果和所述多个第二像素在所述第二方向上的第二代价聚合结果，计算得到所述多个第二像素的第三代价聚合结果，具体包括：
    针对于所述多个第二像素中的各像素，将所述像素在各视差下的第一代价聚合结果与所述像素在各所述视差下的第二代价聚合结果对应相加，得到所述像素在各所述视差下的第三代价聚合结果。
  43. 根据权利要求42所述的集成电路，其特征在于，所述第二组计算单元，还用于将所述像素在所述视差范围内聚合结果最小的视差，作为所述像素的最终视差。
  44. 根据权利要求23-43任一项所述的集成电路，其特征在于，所述第一方向和所述第二方向对应相反。
  45. 一种图像处理装置，其特征在于，包括：权利要求23-44任一项所述的集成电路。
  46. 一种无人机，其特征在于，所述无人机包括机身、设置于所述机身上的动力系统、拍摄装置和集成电路；
    所述动力系统，用于为所述无人机提供动力；
    所述拍摄装置，用于在所述无人机飞行过程中进行拍摄，以获得对应不同视点的两幅图像，所述两幅图像中的目标图像被划分为多个区块，所述多个区块之间存在一定的处理顺序；
    所述集成电路包括：并行处理的第一组计算单元和第二组计算单元；
    所述第一组计算单元，用于计算得到位于所述多个区块中第一区块中的多个第一像素在第一方向上的第一代价聚合结果；
    所述第二组计算单元，用于计算得到位于所述多个区块中第二区块中的多个第二像素在第二方向上的第二代价聚合结果。
  47. 根据权利要求46所述的无人机，其特征在于，所述第一区块和所述第二区块相邻，且所述处理顺序中所述第二区块位于所述第一区块之前。
  48. 根据权利要求47所述的无人机，其特征在于，所述第二组计算单元，还用于基于所述多个第二像素在第一方向上的第一代价聚合结果和所述多个第二像素在所述第二方向上的第二代价聚合结果，计算得到所述多个第二像素的第三代价聚合结果。
  49. 根据权利要求48所述的无人机，其特征在于，位于所述第一区块中的第一像素的第一代价聚合结果和位于所述第二区块中的第二像素的第一代价聚合结果均存储在所述集成电路的内部存储器中。
  50. 根据权利要求49所述的无人机，其特征在于，位于所述第一区块中的第一像素的第一代价聚合结果和位于所述第二区块中的第二像素的第一代价聚合结果，共用所述内部存储器中的同一块存储空间。
  51. 根据权利要求50所述的无人机，其特征在于，所述第一组计算单元和所述第二组计算单元的单次计算均占用单个时钟周期；
    一时钟周期内所述第一组计算单元计算得到的第一数据，被存入所述内部存储器中该时钟周期内所述第二组计算单元计算使用的第二数据对应的存储空间；其中，所述第一数据为所述第一像素的第一代价聚合结果的数据，所述第二数据为所述第二像素的第一代价聚合结果的数据。
  52. 根据权利要求51所述的无人机，其特征在于，所述内部存储器包括第一内部存储器；所述第二组计算单元，还用于：
    在第一时钟周期内，从所述第一内部存储器的目标存储空间中读取所述第二数据以及与所述第二数据相邻的第三数据，并缓存所述第一组计算单元 计算得到的所述第一数据；
    其中，所述第二数据用于所述第二组计算单元在所述第一时钟周期内计算第三代价聚合结果；所述第三数据为所述第二像素的第一代价聚合结果的数据，所述第三数据用于所述第二组计算单元在第二时钟周期内计算第三代价聚合结果；所述第二时钟周期为所述第一时钟周期的下一个时钟周期。
  53. 根据权利要求52所述的无人机，其特征在于，所述第一组计算单元，还用于：
    在所述第二时钟周期内，将所述第一数据和第四数据写入所述第一内部存储器的所述目标存储空间中；所述第四数据为在所述第二时钟周期内所述第一组计算单元计算得到的所述第一像素的第一代价聚合结果的数据。
  54. 根据权利要求50所述的无人机，其特征在于，所述第一组计算单元和所述第二组计算单元的单次计算均占用单个时钟周期；
    第二时钟周期内所述第一组计算单元计算得到的第一数据，被存入所述内部存储器中第一时钟周期内所述第二组计算单元计算使用的第二数据对应的存储空间；其中，所述第一数据为所述第一像素的第一代价聚合结果的数据，所述第二数据为所述第二像素的第一代价聚合结果的数据；所述第二时钟周期为所述第一时钟周期的下一个时钟周期。
  55. 根据权利要求54所述的无人机，其特征在于，所述内部存储器包括第一内部存储器和第二内部存储器；
    所述第一组计算单元，还用于在第一时钟周期内，将所述第一组计算单元计算得到的所述第一数据写入所述第一内部存储器中；
    所述第二组计算单元，还用于在所述第一时钟周期内，从所述第二内部存储器的目标存储空间中读取所述第二数据；所述第二数据用于所述第二组计算单元在所述第一时钟周期内计算第三代价聚合结果。
  56. 根据权利要求55所述的无人机，其特征在于，所述第二组处理单元，还用于在第二时钟周期内，从所述第一内部存储器中读取第三数据；
    所述第一组计算单元，还用于在所述第二时钟周期内，将所述第二组计算单元计算得到的第四数据写入所述第二内部存储器的所述目标存储空间中；
    其中，所述第三数据为所述第二像素的第一代价聚合结果的数据，所述第三数据用于所述第二组计算单元在第二时钟周期内计算得到的第三代价结 果；所述第四数据为在所述第二时钟周期内所述第一组计算单元计算得到的所述第一像素的第一代价聚合结果的数据；。
  57. 根据权利要求51-56任一项所述的无人机，其特征在于，所述数据的大小包括下述中的任意一种：
    单个像素在全部视差下的第一代价聚合结果的大小，单个像素在部分视差下的第一代价聚合结果的大小，多个像素在全部视差下的第一代价聚合结果的大小。
  58. 根据权利要求46-57任一项所述的无人机，其特征在于，所述第一方向为多个方向；所述第一组计算单元，具体用于：
    从存储器中读取所述多个第一像素中部分像素在多个第一方向中的至少一个所述第一方向上的单方向代价聚合结果，并基于所述部分像素的所述第一代价聚合结果计算得到所述多个第一像素中其他像素的第一代价聚合结果，所述部分像素的在所述多个第一方向中的至少一个所述第一方向上的单方向代价聚合结果是通过先前处理获得并存储在所述存储器中。
  59. 根据权利要求58所述的无人机，其特征在于，所述多个第一方向包括向下方向，所述部分像素包括所述多个第一像素中位于所述第一区块上边缘的像素。
  60. 根据权利要求58所述的无人机，其特征在于，所述多个第一方向包括向右方向，所述部分像素包括所述多个第一像素中位于所述第一区块左边缘的像素。
  61. 根据权利要求58所述的无人机，其特征在于，所述多个第一方向包括向上方向，所述部分像素包括所述多个第一像素中位于所述第一区块下边缘的像素。
  62. 根据权利要求58所述的无人机，其特征在于，所述多个第一方向包括向左方向，所述部分像素包括所述多个第一像素中位于所述第一区块右边缘的像素。
  63. 根据权利要求58-62任一项所述的无人机，其特征在于，所述多个区块中目标区块中的多个像素在所述多个第一方向上的第一聚合结果，是在所述先前处理的过程中获得，所述目标区块为所述多个区块中经所述第二组计算单元第一个处理的区块。
  64. 根据权利要求58-63任一项所述的无人机，其特征在于，所述第一代 价聚合结果、所述第二代价聚合结果以及所述第三代价聚合结果包括像素在视差范围内各视差下的聚合结果。
  65. 根据权利要求64所述的无人机，其特征在于，所述第二组计算单元用于基于所述多个第二像素在第一方向上的第一代价聚合结果和所述多个第二像素在所述第二方向上的第二代价聚合结果，计算得到所述多个第二像素的第三代价聚合结果，具体包括：
    针对于所述多个第二像素中的各像素，将所述像素在各视差下的第一代价聚合结果与所述像素在各所述视差下的第二代价聚合结果对应相加，得到所述像素在各所述视差下的第三代价聚合结果。
  66. 根据权利要求65所述的无人机，其特征在于，所述第二组计算单元，还用于将所述像素在所述视差范围内聚合结果最小的视差，作为所述像素的最终视差。
  67. 根据权利要求46-66任一项所述的无人机，其特征在于，所述第一方向和所述第二方向对应相反。
PCT/CN2020/100105 2020-07-03 2020-07-03 图像处理方法、集成电路、装置及设备 WO2022000456A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2020/100105 WO2022000456A1 (zh) 2020-07-03 2020-07-03 图像处理方法、集成电路、装置及设备
CN202080005628.4A CN112889030A (zh) 2020-07-03 2020-07-03 图像处理方法、集成电路、装置及设备

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/100105 WO2022000456A1 (zh) 2020-07-03 2020-07-03 图像处理方法、集成电路、装置及设备

Publications (1)

Publication Number Publication Date
WO2022000456A1 true WO2022000456A1 (zh) 2022-01-06

Family

ID=76046101

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/100105 WO2022000456A1 (zh) 2020-07-03 2020-07-03 图像处理方法、集成电路、装置及设备

Country Status (2)

Country Link
CN (1) CN112889030A (zh)
WO (1) WO2022000456A1 (zh)

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2913998A1 (en) * 2014-02-28 2015-09-02 Ricoh Company, Ltd. Disparity value deriving device, equipment control system, movable apparatus, robot, disparity value deriving method, and computer-readable storage medium
US20150332472A1 (en) * 2014-05-16 2015-11-19 Nokia Corporation Method, apparatus and computer program product for disparity estimation in images
CN106023147A (zh) * 2016-05-06 2016-10-12 西安航天天绘数据技术有限公司 一种基于gpu的线阵遥感影像dsm快速提取方法
CN107220994A (zh) * 2017-06-01 2017-09-29 成都通甲优博科技有限责任公司 一种立体匹配的方法及系统
CN107220997A (zh) * 2017-05-22 2017-09-29 成都通甲优博科技有限责任公司 一种立体匹配方法及系统
CN107481271A (zh) * 2017-07-25 2017-12-15 成都通甲优博科技有限责任公司 一种立体匹配方法、系统及移动终端
CN107990878A (zh) * 2017-11-08 2018-05-04 中国科学院西安光学精密机械研究所 基于微光双目相机的测距系统和测距方法
CN110310320A (zh) * 2019-07-09 2019-10-08 南京美基森信息技术有限公司 一种双目视觉匹配代价聚合优化方法
CN110310220A (zh) * 2018-03-27 2019-10-08 广东瑞图万方科技股份有限公司 一种半全局实时立体匹配方法
CN110533710A (zh) * 2019-08-22 2019-12-03 桂林电子科技大学 一种基于gpu的双目匹配算法的方法及处理装置

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10080007B2 (en) * 2016-03-17 2018-09-18 Texas Instruments Incorporated Hybrid tiling strategy for semi-global matching stereo hardware acceleration
CN109743562B (zh) * 2019-01-10 2020-12-25 中国科学技术大学 基于Census算法的匹配代价计算电路结构及其工作方法

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2913998A1 (en) * 2014-02-28 2015-09-02 Ricoh Company, Ltd. Disparity value deriving device, equipment control system, movable apparatus, robot, disparity value deriving method, and computer-readable storage medium
US20150332472A1 (en) * 2014-05-16 2015-11-19 Nokia Corporation Method, apparatus and computer program product for disparity estimation in images
CN106023147A (zh) * 2016-05-06 2016-10-12 西安航天天绘数据技术有限公司 一种基于gpu的线阵遥感影像dsm快速提取方法
CN107220997A (zh) * 2017-05-22 2017-09-29 成都通甲优博科技有限责任公司 一种立体匹配方法及系统
CN107220994A (zh) * 2017-06-01 2017-09-29 成都通甲优博科技有限责任公司 一种立体匹配的方法及系统
CN107481271A (zh) * 2017-07-25 2017-12-15 成都通甲优博科技有限责任公司 一种立体匹配方法、系统及移动终端
CN107990878A (zh) * 2017-11-08 2018-05-04 中国科学院西安光学精密机械研究所 基于微光双目相机的测距系统和测距方法
CN110310220A (zh) * 2018-03-27 2019-10-08 广东瑞图万方科技股份有限公司 一种半全局实时立体匹配方法
CN110310320A (zh) * 2019-07-09 2019-10-08 南京美基森信息技术有限公司 一种双目视觉匹配代价聚合优化方法
CN110533710A (zh) * 2019-08-22 2019-12-03 桂林电子科技大学 一种基于gpu的双目匹配算法的方法及处理装置

Also Published As

Publication number Publication date
CN112889030A (zh) 2021-06-01

Similar Documents

Publication Publication Date Title
CN110073359B (zh) 用于卷积神经网络的有效数据布局
RU2623806C1 (ru) Способ и устройство обработки стереоизображений
US11580367B2 (en) Method and system for processing neural network
WO2017173755A1 (zh) 片上数据划分读写方法、系统及其装置
CN105761233A (zh) 一种基于fpga的实时全景图像拼接方法
US9036044B1 (en) Adjusting camera parameters associated with a plurality of images
CN107566688A (zh) 一种基于卷积神经网络的视频防抖方法及装置
CN111861883B (zh) 基于同步积分surf算法的多路视频拼接方法
CN109743562B (zh) 基于Census算法的匹配代价计算电路结构及其工作方法
CN112799599B (zh) 一种数据存储方法、计算核、芯片和电子设备
Ding et al. Real-time stereo vision system using adaptive weight cost aggregation approach
WO2016000129A1 (en) Adaptive partition mechanism with arbitrary tile shape for tile based rendering gpu architecture
CN111951368A (zh) 一种点云、体素和多视图融合的深度学习方法
WO2022000458A1 (zh) 图像深度信息确定方法、装置、设备和存储介质
WO2022000456A1 (zh) 图像处理方法、集成电路、装置及设备
Bhayani et al. Partially calibrated semi-generalized pose from hybrid point correspondences
US20230245277A1 (en) Image restoration method and device
WO2022000454A1 (zh) 图像处理方法、集成电路、装置、可移动平台及存储介质
WO2023103551A1 (zh) 图像数据处理方法、装置、设备及存储介质
Lin et al. Matching cost filtering for dense stereo correspondence
WO2023240764A1 (zh) 混合代价体的双目立体匹配方法、设备及存储介质
WO2023184754A1 (zh) 可配置实时视差点云计算装置及方法
WO2022247394A1 (zh) 图像拼接方法及装置、存储介质及电子设备
US20220121551A1 (en) Method and device for calculating runtime of neural network on processor
Wang et al. An implementation method of the box filter on FPGA

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20943562

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20943562

Country of ref document: EP

Kind code of ref document: A1