CN117115071A

CN117115071A - Parallax image determining method and device, integrated circuit chip and computer equipment

Info

Publication number: CN117115071A
Application number: CN202210528232.9A
Authority: CN
Inventors: 张承果; 周阳; 李鹲翱; 段智涓; 谭其林; 唐林; 唐诗然; 鲁良; 郭怀成; 李燕华
Original assignee: Chengdu Pudu Robot Co ltd; Shenzhen Pudu Technology Co Ltd
Current assignee: Chengdu Pudu Robot Co ltd; Shenzhen Pudu Technology Co Ltd
Priority date: 2022-05-16
Filing date: 2022-05-16
Publication date: 2023-11-24

Abstract

The present application relates to a parallax image determination method, apparatus, integrated circuit chip, computer device, storage medium, and computer program product. The method comprises the following steps: performing stereo matching on the left and right camera images to obtain a parallax image to be processed, and sliding through the parallax image to be processed by utilizing a first sliding window; in each local parallax image corresponding to the first sliding window, determining whether the difference absolute value of each parallax value in the local parallax image and the central parallax value corresponding to the central position of the local parallax image is smaller than or equal to a first threshold value; a statistical quantity of disparity values having an absolute value of the statistical difference value less than or equal to a first threshold; if the statistical quantity is smaller than or equal to the second threshold value, reserving a center disparity value; if the statistical quantity is greater than the second threshold value, setting the central parallax value to 0; after traversing the parallax images to be processed by using the first sliding window, determining a target parallax image. By adopting the method, the convenience and the efficiency of determining the parallax image can be improved.

Description

Parallax image determining method and device, integrated circuit chip and computer equipment

Technical Field

The present application relates to the field of computer vision, and in particular, to a method, an apparatus, an integrated circuit chip, a computer device, a storage medium, and a computer program product for determining a parallax image.

Background

Binocular stereo matching is also known as disparity estimation (disparity estimation), or binocular depth estimation; the binocular camera is used for shooting left and right view images (namely left and right camera images) of the same scene, and then a stereo matching algorithm is used for calculating the left and right camera images so as to obtain parallax images. Parallax is a pixel level difference of a point in a three-dimensional scene at a position corresponding to the point in left and right camera images; the parallax image refers to a parallax image formed from a parallax value corresponding to each pixel in a reference image (typically, a left image is taken as a reference image). After the base line distance and the focal length of the camera are given, a corresponding depth image can be calculated according to the parallax image, and the subsequent implementation can be realized by technologies in the fields of depth perception, automatic driving, security monitoring and the like according to the depth image. The binocular stereo matching mainly comprises four steps of initial cost calculation, cost aggregation, parallax calculation and parallax post-processing, and has the characteristics of high calculation complexity, high precision requirement and the like, and the characteristics make the deployment of the stereo matching algorithm on an embedded processing platform particularly difficult.

The parallax post-processing stage comprises denoising operation on the parallax image obtained through parallax calculation. In the conventional technical scheme, for each parallax value in the parallax image, determining that at least one parallax value on the upper, lower, left and right (i.e. around) of the parallax image meets the parallax communication condition, and then detecting whether the surrounding parallax values meet the parallax communication condition by taking the parallax values as starting points (called propagation) respectively; every time a new communication point is detected, the mark position 1 of the corresponding point is increased by one, the counter is increased until the surrounding points of each new communication point do not meet the parallax communication condition, and the counting is stopped; then judging whether the count value (namely the number of the pixel points in the connected area with the current processed parallax value) is larger than a preset threshold value or not; if the parallax value is larger than the preset value, the parallax value processed currently is considered to be effective, and the parallax value is reserved; otherwise, the currently processed disparity value is considered to be a noise point, and thus the currently processed disparity value is removed. However, the method for determining the parallax image needs to perform judgment and calculation for each parallax value, has a complex processing process, and needs to consume a large amount of logic resources and storage resources.

Therefore, how to improve the convenience and efficiency of determining the parallax image is a technical problem that needs to be solved by those skilled in the art at present.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a parallax image determination method, apparatus, integrated circuit chip, computer device, computer-readable storage medium, and computer program product that can improve the convenience and efficiency of determining a parallax image.

In a first aspect, the present application provides a method of determining a parallax image. The method comprises the following steps:

performing stereo matching on the left and right camera images to obtain a parallax image to be processed, and sliding and traversing the parallax image to be processed by utilizing a first sliding window;

in each local parallax image corresponding to the first sliding window, determining whether the difference absolute value of each parallax value in the local parallax image and the central parallax value corresponding to the central position of the local parallax image is smaller than or equal to a first threshold value;

counting the number of the parallax values of which the absolute value of the difference is smaller than or equal to the first threshold value;

if the statistical quantity is smaller than or equal to a second threshold value, reserving the center disparity value;

if the statistical quantity is larger than the second threshold value, setting the central parallax value as a target value;

and determining a target parallax image after traversing the parallax image to be processed by using the first sliding window.

In one embodiment, the stereo matching of the left and right camera images to obtain the parallax image to be processed includes:

acquiring the left and right camera images;

calculating initial cost according to the left and right camera images;

performing cost aggregation operation according to the initial cost to obtain a final cost;

and performing parallax calculation on the final cost to determine the parallax image to be processed.

In one embodiment, the calculating the initial cost according to the left and right camera images includes:

traversing the first image by utilizing a second sliding window, determining each first partial pixel image corresponding to the second sliding window, and determining a first transformation vector corresponding to each first partial pixel image; wherein the left and right camera images include the first image and a second image;

determining a second transformation vector of a second local pixel image corresponding to the first local pixel image in the second image;

determining a local initial cost corresponding to the first local pixel image according to the first transformation vector and the second transformation vector;

after traversing the first image by using the second sliding window, obtaining the initial cost corresponding to the first image; the initial cost includes the local initial cost.

In one embodiment, the determining the second transformation vector of the second partial-pixel image corresponding to the first partial-pixel image in the second image includes:

determining a preset parallax range, and calculating second transformation vectors to be confirmed, which correspond to the second local pixel images corresponding to the first local pixel images in the second image respectively in the preset parallax range; the second transformation vector to be confirmed comprises the second transformation vector;

the determining, according to the first transformation vector and the second transformation vector, a local initial cost corresponding to the first local pixel image includes:

respectively determining corresponding local initial cost to be confirmed according to the first transformation vector and each second transformation vector to be confirmed;

and determining the minimum value in the local initial cost to be confirmed as the local initial cost.

In one embodiment, after the calculating the initial cost according to the left and right camera images and before the performing the cost aggregation operation according to the initial cost, the method further includes:

and carrying out saturation operation on the initial cost to obtain the updated initial cost.

In one embodiment, the performing saturation operation on the initial cost to obtain the updated initial cost includes:

if the initial cost is greater than or equal to a third threshold value, setting the initial cost as a first preset value;

otherwise, deleting the lowest bit of the binary corresponding to the initial cost, and updating the initial cost by using the binary value with the lowest bit deleted.

In one embodiment, the performing a cost aggregation operation according to the initial cost to obtain a final cost includes:

and performing four-direction aggregation cost operation according to the initial cost to obtain the final cost.

In one embodiment, the method further comprises:

grouping each local initial cost to be confirmed and the local initial cost according to a preset rule;

and performing information splicing on the grouped local initial cost to be confirmed and the local initial cost, and storing the local initial cost to be confirmed and the local initial cost to a preset storage position according to a splicing result.

In one embodiment, the left and right camera images include:

acquiring two groups of left and right camera images by using a double-path binocular camera module;

And respectively executing a parallax image determining method on the two groups of left and right camera images in a time division multiplexing mode.

In a second aspect, the present application also provides a device for determining a parallax image. The device comprises:

the sliding window traversing module is used for carrying out three-dimensional matching on the left camera image and the right camera image to obtain a parallax image to be processed, and utilizing a first sliding window to slide and traverse the parallax image to be processed;

a communication determining module, configured to determine, in each local parallax image corresponding to the first sliding window, whether a difference absolute value of each parallax value in the local parallax image and a center parallax value corresponding to a center position of the local parallax image is smaller than a first threshold;

a statistics module, configured to count a number of statistics of the disparity values in which the absolute value of the difference value is smaller than the first threshold;

the execution module is used for reserving the center disparity value if the statistical quantity is smaller than or equal to a second threshold value; if the statistical quantity is larger than the second threshold value, setting the central parallax value as a target value;

and the image determining module is used for determining a target parallax image after traversing the parallax image to be processed by utilizing the first sliding window.

In a third aspect, the present application also provides an integrated circuit chip, where the integrated circuit chip is configured to implement the steps of the method described above.

In a fourth aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the steps of the method described above when the processor executes the computer program.

In a fifth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the method described above

In a sixth aspect, the application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of the method described above.

The method comprises the steps of utilizing a first sliding window to slide through a parallax image to be processed, and determining whether the absolute value of the difference value of each parallax value in the partial parallax image and the central parallax value corresponding to the central position of the partial parallax image is smaller than or equal to a first threshold value in each partial parallax image corresponding to the first sliding window; and counting the statistic quantity of the parallax values of which the absolute value is smaller than or equal to the first threshold value, and deleting noise points according to the statistic result to obtain the target parallax image. Compared with the prior art, the method is used for judging and calculating the local parallax image corresponding to the first sliding window, so that the complexity of the judging and calculating process can be reduced, and the convenience and the efficiency of determining the parallax image can be improved.

Drawings

FIG. 1 is a flow chart of a method for determining a difference image in one embodiment;

FIG. 2 is a schematic diagram of a process for calculating an initial cost in one embodiment;

FIG. 3 is a schematic diagram of a process for computing a transformation vector in one embodiment;

FIG. 4 is a schematic diagram of a process for calculating an initial cost in one embodiment;

FIG. 5 is a schematic diagram of a first image or a second image of left and right camera images in one embodiment;

fig. 6 is a schematic diagram of a parallax image to be processed corresponding to an initial cost of not performing saturation processing;

fig. 7 is a schematic diagram of a parallax image to be processed corresponding to an initial cost after saturation processing;

fig. 8 is a schematic diagram of a disparity map corresponding to an eight-direction aggregation cost operation in a conventional technical scheme;

fig. 9 is a schematic diagram of a disparity map corresponding to a four-way aggregation cost operation in one embodiment;

FIG. 10 is a timing diagram of time division multiplexing in one embodiment;

FIG. 11 is a block diagram showing the construction of a device for determining a difference image in one embodiment;

fig. 12 is an internal structural diagram of a computer device in one embodiment.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

The parallax image determining method provided by the embodiment of the application can be applied to terminal equipment provided with a binocular camera; the terminal equipment acquires left and right camera images by using a binocular camera, performs stereo matching on the left and right camera images to obtain a parallax image to be processed, and slides through the parallax image to be processed by using a first sliding window; in each local parallax image corresponding to the first sliding window, determining whether the difference absolute value of each parallax value in the local parallax image and the central parallax value corresponding to the central position of the local parallax image is smaller than or equal to a first threshold value; a statistical quantity of disparity values having an absolute value of the statistical difference value less than or equal to a first threshold; if the statistical quantity is smaller than or equal to the second threshold value, reserving a center disparity value; if the statistical quantity is larger than a second threshold value, setting the central parallax value as a target value; after traversing the parallax images to be processed by using the first sliding window, determining a target parallax image. The terminal device may be, but not limited to, various intelligent driving devices and portable wearable devices, wherein the intelligent driving devices may be unmanned automobiles, sweeping robots, etc., and the portable wearable devices may be intelligent watches, intelligent bracelets, headsets, etc. It can be appreciated that the method can also be applied to a system including a binocular camera and a server, the binocular camera acquires left and right camera images and transmits the left and right camera images to the server, and the server executes a parallax image determining method to determine a target parallax image. The server may be implemented as a stand-alone server or as a server cluster formed by a plurality of servers.

In one embodiment, a method for determining a parallax image is provided, which is described by taking application of the method to a terminal device as an example, and includes the following steps:

and 102, performing stereo matching on the left and right camera images to obtain a parallax image to be processed, and sliding the parallax image to be processed by utilizing a first sliding window.

In the present embodiment, the left and right camera images refer to images acquired by a binocular camera, and generally include a first image and a second image. In a specific embodiment, the first image corresponds to a left image, the second image corresponds to a right image, and the left image is generally used as the reference image. The parallax image to be processed refers to a parallax image needing sliding window denoising; the parallax image refers to a parallax image formed according to a parallax value corresponding to each pixel in the reference image; parallax is the pixel level difference of the position of a point in the three-dimensional scene at the corresponding point in the left and right camera images. In actual operation, the parallax images to be processed are obtained by stereo matching of the left and right camera images.

The first sliding window refers to a sliding window with a first preset format; and after the left camera image and the right camera image are subjected to stereo matching to obtain a parallax image to be processed, sliding the parallax image to be processed by utilizing a first sliding window according to a first sliding step length. In this embodiment, the window size of the first sliding window is generally m×n; wherein m and n are both odd numbers, and m and n may be equal; the first sliding step length corresponding to the first sliding window is 1.

Step 104, determining whether the absolute value of the difference value of each parallax value in the local parallax images and the central parallax value corresponding to the central position of the local parallax image is smaller than or equal to a first threshold value in each local parallax image corresponding to the first sliding window;

step 106, counting the number of the parallax values with the absolute value of the statistical difference value smaller than or equal to the first threshold value;

step 108, if the statistical quantity is smaller than or equal to the second threshold value, reserving a center disparity value; if the statistical quantity is larger than the second threshold value, the center disparity value is set as a target value.

Specifically, in the process of sliding through the parallax image to be processed by using the first sliding window, after each sliding of the first sliding window, the partial image corresponding to the first sliding window in the parallax image to be processed is a partial parallax image. And aiming at each corresponding partial parallax image when the first sliding window slides to traverse the parallax image to be processed, carrying out the following processing:

according to the first sliding window and the current local parallax image, determining the central position of the local parallax image, and acquiring a central parallax value corresponding to the central position; determining parallax values corresponding to other positions in the local parallax image; calculating difference absolute values of each parallax value in the local parallax image, which correspond to the central parallax value respectively; comparing each absolute value of the difference with a first threshold (threshold 0) respectively, and judging whether each absolute value of the difference is smaller than or equal to the first threshold (threshold 0); a statistical number of parallax values for which the absolute value of the statistical difference is less than or equal to a first threshold (threshold 0); judging whether the counted number is smaller than or equal to a second threshold (threshold 1); if the statistical quantity is smaller than or equal to a second threshold (threshold 1), reserving a center disparity value; if the statistical quantity is greater than a second threshold (threshold 1), setting the central parallax value as a target value; wherein the target value may be 0. Thus, the sliding window denoising of the current partial parallax image is completed.

Then, the first sliding window continues to slide according to the first sliding step length, and the corresponding partial parallax image after sliding is used as the current partial parallax image and is continuously processed.

It should be noted that, in this embodiment, the first threshold and the second threshold are set according to actual experience, and specific values of the first threshold and the second threshold are not limited in this embodiment.

Step 110, after traversing the parallax image to be processed by using the first sliding window, determining a target parallax image.

In the step, after traversing the parallax images to be processed by using the first sliding window, respectively performing sliding window denoising operation on each local parallax image corresponding to the first sliding window, and determining a target parallax image. That is, the target parallax image is an image obtained by traversing the parallax image to be processed with the first sliding window and performing the sliding window denoising operation.

After the target parallax image is determined, the terminal device may perform operations such as obstacle avoidance, three-dimensional modeling, detection and recognition according to the target parallax image.

According to the method for determining the parallax images, the parallax images to be processed are traversed in a sliding mode through the first sliding window, and whether the absolute value of the difference value of each parallax value in the local parallax images and the central parallax value corresponding to the central position of the local parallax images is smaller than or equal to a first threshold value or not is determined in each local parallax image corresponding to the first sliding window; and counting the statistic quantity of the parallax values of which the absolute value is smaller than or equal to the first threshold value, and deleting noise points according to the statistic result to obtain the target parallax image. Compared with the prior art, the method is used for judging and calculating the local parallax image corresponding to the first sliding window, so that the complexity of the judging and calculating process can be reduced, and the convenience and the efficiency of determining the parallax image can be improved.

As a preferred embodiment, the statistical number of the disparity values having the absolute value of the statistical difference value smaller than the first threshold value includes: setting a parallax value, in which the absolute value of the difference in the partial parallax image is less than or equal to a first threshold value, to 1; setting a parallax value, in which the absolute value of the difference value in the partial parallax image is greater than a first threshold value, to 0; and calculating the accumulated value of each updated parallax value in the local parallax image to obtain the statistical quantity.

Specifically, in this embodiment, after comparing each absolute value of the difference with the first threshold (threshold 0), it is determined whether each absolute value of the difference is less than or equal to the first threshold (threshold 0); setting a parallax value, in which the absolute value of the difference in the partial parallax image is less than or equal to a first threshold (threshold 0), to 1, and setting a parallax value, in which the absolute value of the difference in the partial parallax image is greater than the first threshold (threshold 0), to 0; that is, the pixel corresponding to the position is indicated to be communicated with the pixel corresponding to the central parallax value by the parallax value 1, and the pixel corresponding to the position is indicated to be not communicated with the pixel corresponding to the central parallax value by the parallax value 0. Then, the cumulative value of each updated parallax value in the local parallax image is calculated to obtain the statistical quantity, namely the number of 1 in the current local parallax image is calculated to obtain the statistical quantity of the parallax value with the absolute value of the difference value smaller than or equal to the first threshold value. It should be noted that, after determining the statistical quantity according to the method of the present embodiment, the parallax value needs to be further restored to the parallax value before adjustment, so as to perform processing calculation by using the parallax value later.

As another preferred embodiment, the statistical number of the disparity values with the absolute value of the statistical difference value smaller than the first threshold value is preset, and the statistical variable is initialized to 0; sequentially judging whether the calculated absolute value of each difference is smaller than or equal to a first threshold value, if so, adding 1 to the statistical variable, otherwise, keeping the statistical variable unchanged; and continuously judging whether the absolute value of the next difference value is smaller than or equal to the first threshold value or not until all the absolute values of the corresponding differences in the local parallax images are judged, wherein the value corresponding to the statistical variable is the statistical quantity.

Therefore, according to the method of the embodiment, the statistical quantity of the parallax values with the absolute value of the difference value smaller than or equal to the first threshold value can be conveniently and accurately determined.

On the basis of the above embodiment, the technical solution is further described and optimized in this embodiment, and specifically, in this embodiment, stereo matching is performed on the left and right camera images to obtain a parallax image to be processed, where the method includes:

acquiring left and right camera images;

calculating initial cost according to the left and right camera images;

And performing parallax calculation on the final cost to determine a parallax image to be processed.

Specifically, after the left and right camera images are acquired by using the binocular camera, the corresponding initial cost is calculated according to the left and right camera images. The initial cost corresponding to the left and right camera images can be calculated through a Census transformation method.

The cost aggregation operation is based on the operation that similar areas have similar parallaxes, and is used for reducing the influence of matching blurring and image noise in the initial cost; the cost aggregation operation comprises methods such as image segmentation, self-adaptive weight, self-adaptive window and the like; and after the cost aggregation operation is carried out on the initial cost, obtaining the final cost.

And after determining the final cost, obtaining a parallax image to be processed by carrying out parallax calculation on the final cost. More specifically, the corresponding parallax may be calculated by way of WTA (winner takes all) parallax calculation; the specific manner of calculating the parallax in this embodiment is not limited.

It can be seen that according to the method of the embodiment, the parallax image to be processed can be rapidly and conveniently determined.

In the actual operation, after performing the parallax calculation, the parallax optimization operation may further be performed, where the parallax optimization operation includes error matching rejection, small connected regions (Remove Peaks), uniqueness detection (uniques Check), sub-pixel difference value, and left-right consistency detection. The parallax optimization processing is further carried out on the parallax images to be processed, so that the parallax accuracy is improved, and the parallax images to be processed are more reliable and more accurate; therefore, the accuracy of the subsequent sliding window denoising operation based on the parallax image to be processed can be improved.

In addition, in the actual operation, after the left and right camera images are acquired, the image correction operation may be further performed on the left and right camera images, and then the initial cost is calculated for the left and right cameras after the image correction operation. The image correction operation comprises preprocessing steps such as lens shading correction, bilateral filtering denoising, downsampling, automatic exposure and the like, and the preprocessed left and right camera image data needs to be subjected to distortion removal and parallel correction operation to obtain corrected left and right camera images.

Therefore, the image correction operation is performed on the obtained left and right camera images, so that effective information in the left and right camera images can be enhanced, the image quality of the parallax image to be processed determined in the left and right camera images is improved, and the image quality of the target parallax image is further improved.

As shown in fig. 2, a schematic process of calculating an initial cost is provided in this embodiment. On the basis of the above embodiment, the present embodiment further describes and optimizes a technical solution, and specifically, in this embodiment, the calculating initial cost according to the left and right camera images includes:

step 202: traversing the first image by utilizing the second sliding window, determining each first partial pixel image corresponding to the second sliding window, and determining a first transformation vector corresponding to each first partial pixel image; wherein the left and right camera images include a first image and a second image.

Wherein the left and right camera images include a first image (left image) and a second image (right image), and the first image (left image) is used as a reference image. The second sliding window refers to a sliding window with a second preset format; and sliding and traversing the first image by utilizing the second sliding window according to a second sliding step length, wherein in the process of traversing the first image by utilizing the second sliding window, the partial image corresponding to the second sliding window in the first image is a first partial pixel image. For each local pixel image, a first transformation vector corresponding to the first local pixel image is determined.

A schematic process of calculating a transformation vector as shown in fig. 3; assuming that the second sliding window is a sliding window in a 3*3 format, comparing each pixel value of the second sliding window with a pixel value of a sliding window center, setting a position where the pixel value is less than or equal to the pixel value of the sliding window center to 0, setting a position where the pixel value is greater than the pixel value of the sliding window center to 1, and determining a first transformation vector (census vector).

Step 204: a second transformation vector of a second partial-pixel image in the second image corresponding to the first partial-pixel image is determined.

Determining a second partial pixel image in the second image according to the same sliding window position according to the position of the second sliding window in the first image; and determining a second transformation vector corresponding to the second partial pixel image in the same manner as described above.

Step 206: a local initial cost corresponding to the first local pixel image is determined from the first transform vector and the second transform vector.

After corresponding first transformation vectors and second transformation vectors are respectively determined for the first partial pixel image and the second partial pixel image corresponding to the second sliding window, hamming distances of the first transformation vectors and the second transformation vectors are calculated; specifically, performing exclusive-or operation on sequences corresponding to the first transformation vector and the second transformation vector bit by bit, and counting the number of bits 1 in an exclusive-or operation result to obtain a corresponding hamming distance; namely, determining the local initial cost corresponding to the position of one sliding window. Wherein the hamming distance is a value representing the similarity between the first transformation vector and the second transformation vector, and the smaller the hamming distance is, the larger the similarity is.

Step 208: after traversing the first image by using the second sliding window, obtaining an initial cost corresponding to the first image; the initial cost includes a local initial cost.

Specifically, traversing the first image by using the second sliding window, and respectively determining the local initial cost of the first local pixel image corresponding to each sliding window position aiming at each sliding window position of the second sliding window; after traversing the first image by using the second sliding window, obtaining local initial costs corresponding to each first local pixel image of the first image, and obtaining the initial costs corresponding to the first image through arrangement.

Specifically, a process diagram of calculating an initial cost is shown in fig. 4. Specifically, the hamming distances corresponding to the first local pixel images in the 3*3 format are obtained, and the hamming distances in the 3*3 format are summed to obtain the corresponding initial cost.

Therefore, the method for determining the initial cost in the embodiment reserves the position characteristics of the pixels in the second sliding window, is more robust to the brightness deviation, and can reduce mismatching caused by the illumination difference.

As a preferred embodiment, step 204: determining a second transformation vector for a second partial-pixel image in the second image that corresponds to the first partial-pixel image, comprising:

determining a preset parallax range, and calculating second transformation vectors to be confirmed, which correspond to the second local pixel images corresponding to the first local pixel images in the second image respectively in the preset parallax range; the second transformation vector to be confirmed comprises a second transformation vector;

in this embodiment, a preset parallax range is first determined, after a second local pixel image corresponding to the same sliding window position in the first image is determined in the second image, for each candidate parallax value in the preset parallax range, a second transformation vector corresponding to the second local pixel image is calculated respectively, when the assumed parallax value is the candidate parallax value, so as to obtain a second transformation vector to be confirmed.

Correspondingly, step 206: determining a local initial cost corresponding to the first local pixel image from the first transform vector and the second transform vector, comprising:

Correspondingly, after a plurality of second transformation vectors to be confirmed corresponding to the second partial pixel image are determined according to the preset difference range, hamming distances respectively corresponding to the first transformation vector and each second transformation vector to be confirmed are calculated, and then a plurality of partial initial costs to be confirmed corresponding to each first partial pixel image are obtained. And then comparing the sizes of the plurality of to-be-confirmed local initial costs corresponding to the first local pixel image, and determining the minimum value in the plurality of to-be-confirmed local initial costs as the local initial cost.

Therefore, the initial cost determined by using each minimum local initial cost is the minimum initial cost, so that the logic resources required to be occupied by the initial cost can be reduced.

On the basis of the above embodiment, the technical solution is further described and optimized in this embodiment, and specifically, in this embodiment, after calculating the initial cost according to the left and right camera images, and before performing the cost aggregation operation according to the initial cost to obtain the final cost, the method further includes:

Specifically, the saturation operation refers to an operation of adjusting the initial cost so that a better processing effect can be achieved when the operation is performed by using the adjusted initial cost. In actual operation, the initial cost can be directly calculated, and a value corresponding to half or other proportions of the initial cost is used as the updated initial cost; or directly subtracting a preset value from the initial cost, and taking the calculation result as the updated initial cost; the specific process of the saturation operation is not limited in this embodiment.

It should be noted that, the saturation operation is further performed on the initial cost, so that the adjusted initial cost can reduce occupation of logic resources under the condition that the accuracy of the determined target parallax image is not affected.

As a preferred embodiment, performing saturation operation on the initial cost to obtain an updated initial cost, including:

The third threshold is a judgment value for determining the mode of adjusting the initial cost; the actual value of the third threshold is not limited in this embodiment, and may be set according to actual requirements. In one practical operation, in order to reduce the adjusted initial cost below 4 bits, the binary corresponding maximum value 31 of 5 bits is therefore set to the third threshold. Wherein the first preset value is set to 15 according to the maximum value 1111 corresponding to the binary of 4 bits.

In a specific embodiment, assuming that the third threshold is 31, the first preset value is 15, the initial cost is saturated, and the operation principle is as follows:

according to the above operation principle, if the initial cost (cost_block) is greater than or equal to the third threshold 31, the initial cost is directly set to the first preset value 15; if the initial cost (cost_block) is smaller than the third threshold 31, deleting the lowest bit of the binary corresponding to the initial cost, and updating the initial cost by using the binary value after deleting the lowest bit. For example, if the initial cost is 35 (100011); the initial cost is set directly to a first preset value 15; if the initial cost is 30 (11110), deleting the lowest bit corresponding to the initial cost to obtain a binary 1111, and determining that the updated initial cost is 15 according to the binary 1111 obtained by deleting the lowest bit; if the initial cost is 12 (1100), deleting the least significant bit corresponding to the initial cost to obtain a binary system 110, and determining that the updated initial cost is 6 according to the binary system 110 obtained by deleting the least significant bit.

It can be appreciated that by saturating the initial cost, the logical resources that are required to be occupied by the initial cost can be reduced. And, because the minimum initial cost is used for calculation in the subsequent processing process, the parallax accuracy influence caused by the saturation processing of the initial cost is negligible. For example, fig. 5 is a schematic diagram of a first image or a second image in left and right camera images, and a corresponding parallax image to be processed is obtained according to whether to perform saturation processing on an initial cost to set a corresponding simulation algorithm effect; fig. 6 is a schematic diagram of a parallax image to be processed corresponding to an initial cost of not performing saturation processing, fig. 7 is a schematic diagram of a parallax image to be processed corresponding to an initial cost of performing saturation processing, and comparing fig. 6 and fig. 7 can show that if the initial cost is subjected to saturation processing, a phenomenon of dysphoria occurs at the edge of the obtained parallax image to be processed, but no obvious error exists in depth information, most parallax information of the image is reserved, and the error of the depth information of the edge is negligible; therefore, the saturation processing of the initial cost can be described, the precision of the depth information is reserved, and the consumption of logic resources is greatly reduced.

Based on the foregoing embodiments, the present embodiment further describes and optimizes a technical solution, and specifically, in this embodiment, a cost aggregation operation is performed according to an initial cost, to obtain a final cost, where the method includes:

Specifically, in this embodiment, the four-way cost aggregation operation is performed on the initial cost after the saturation processing, and the cost aggregation principle is as follows:

wherein L is _r (p, d) represents the cost aggregation value of the matching point p after aggregation in the r direction;

cost (p, d) represents the matching Cost value of the matching point p;

L _r (p-r, d) represents a cost aggregation value of a last matching point of the matching point p under the same parallax;

L _r (p-r, d-1) represents matchingThe disparity of the last matching point of point p minus the cost aggregate value;

L _r (p-r, d+1) represents a matching cost aggregate value added to the parallax of the last matching point of the matching point p;

min _i (L _r (p-r, i)) represents the minimum value of the cost aggregate value at all parallaxes of the last matching point of the matching point p;

p1 and P2 are adjustable parameters for compensating the parallax and finely adjusting the algorithm; p1 represents a penalty factor for a disparity difference of 1, and P2 represents a penalty factor for the direction of minimum initial cost.

The multi-directional aggregate cost value is:

in the embodiment, the cost aggregation operation is performed on the W, NW, N, NE four directions, so that the data quantity to be cached is greatly reduced, the calculation complexity is reduced, and the cost aggregation efficiency can be improved; moreover, through testing, cost aggregation is performed according to the method of the embodiment, so that the accuracy of aggregation cost can be effectively reserved, and the effect is compared with that shown in fig. 8 and 9; fig. 8 is a schematic diagram of a disparity map corresponding to an eight-direction aggregation cost operation in the conventional technical solution, and fig. 9 is a schematic diagram of a disparity map corresponding to a four-direction aggregation cost operation in the present embodiment; as can be seen by comparing fig. 8 and fig. 9, the disparity map information does cause information missing, but most of depth information is not lost, and is not affected for most of applications such as obstacle avoidance, detection and recognition.

On the basis of the above embodiment, the technical solution is further described and optimized in this embodiment, and specifically, in this embodiment, the method further includes:

grouping the local initial cost and the local initial cost to be confirmed according to a preset rule;

The local initial cost and the local initial cost to be confirmed are grouped according to the aggregation direction of cost aggregation, so that data corresponding to each aggregation direction respectively is obtained, wherein the data comprises the local initial cost and the local initial cost to be confirmed.

After data are grouped, the grouped local initial cost to be confirmed and the local initial cost are subjected to information splicing, and spliced information is obtained; and then storing the local initial cost and the local initial cost to be confirmed to a preset storage position according to the splicing result.

It can be understood that when parallel cost aggregation operation is performed, the local initial cost and the local initial cost to be confirmed, which correspond to each aggregation direction respectively, need to be cached, and if a parallax independent caching mode is adopted, a large amount of on-chip storage resources need to be occupied.

And a splicing caching method can be adopted in the cost aggregation and parallax calculation processes. In a specific operation, under the condition that the resolution of the image is 800x500 and the parallax range is taken 74, the memory occupation resource conditions respectively corresponding to the parallax independent cache and the splicing cache are compared as follows:

table 1 memory resource occupancy contrast for cache methods

Caching method	Independent caching	Splicing buffer memory
			Occupy bram resource (36 Kb)	185	63

As can be seen from the above table, according to the splicing caching method of the embodiment, occupation of on-chip storage resources can be greatly reduced.

On the basis of the above embodiment, the technical solution is further described and optimized in this embodiment, and specifically, in this embodiment, the left and right camera images include:

acquiring two groups of left and right camera images by using a two-way binocular camera module;

the method for determining the parallax images is performed on the two groups of left and right camera images respectively by a time-division multiplexing manner.

The binocular camera module refers to a camera capable of capturing two left and right images of a shooting object, and the common binocular camera module has multiple types according to actual requirements, and the specific type of the binocular camera module is not limited in this embodiment. The two-path binocular camera module refers to shooting the same shooting object by using two groups of binocular camera modules.

Assume that the two-way binocular camera module comprises a binocular camera module 0 and a binocular camera module 1, and the binocular camera module 0 and the binocular camera module 1 respectively acquire corresponding left and right camera images, and respectively execute a parallax image determining method on the two groups of left and right camera images in a time division multiplexing mode. That is, the group includes the binocular camera module 0 and the binocular camera module 1 to generate image frame header, image effective signal and image data in a time-sharing manner, and the processing channel proc_channel is switched according to the frame header count, so as to output two paths of target parallax images corresponding to the binocular camera module 0 and the binocular camera module 1 in a time-sharing manner.

A time-division multiplexing timing diagram as shown in fig. 10; wherein clk is a clock signal and rst_n is a reset signal; img_frm_vld1 and img_frm_vld2 represent image frame header signals corresponding to the binocular camera module 0 and the binocular camera module 1, respectively; img_row_vld1 and img_row_vld2 represent image line valid signals corresponding to the binocular camera module 0 and the binocular camera module 1 respectively; imgl_dat1 and imgr_dat1 represent left and right images of the binocular camera module 0; imgl_dat2 and imgr_dat2 represent left and right images of the binocular camera module 1; proc_channel represents a channel being processed, and by switching the channel to be the binocular camera module 0 or the binocular camera module 1, the processing of left and right camera images corresponding to the binocular camera module 0 or the binocular camera module 1 is represented; dispfrm vld represents the depth map data frame header; disp_row_vld represents the depth row valid signal; disp_dat represents the derived target parallax image.

According to the method of the embodiment, the processing mechanism for performing time-sharing multiplexing on the two groups of left and right camera images acquired by the two-path binocular camera module can ensure the processing speed of the two-path binocular camera module, reduce the exposure frequency of the single module, and remarkably reduce the working power consumption of the binocular camera module.

In order to enable those skilled in the art to better understand the technical scheme of the present application, the technical scheme in the embodiment of the present application is described in detail below in conjunction with practical application scenarios. The method for determining the parallax image provided by the embodiment of the application comprises the following specific steps:

a method for determining parallax images is respectively carried out on the two groups of left and right camera images in a time division multiplexing mode; the left and right camera images acquired by each binocular camera module comprise a first image and a second image;

traversing the first image by utilizing the second sliding window, determining each first partial pixel image corresponding to the second sliding window, and determining a first transformation vector corresponding to each first partial pixel image;

determining the minimum value in the local initial cost to be confirmed as the local initial cost;

information splicing is carried out on the local initial cost to be confirmed and the local initial cost after grouping, and the local initial cost to be confirmed are stored to a preset storage position according to a splicing result;

after traversing the first image by using the second sliding window, obtaining an initial cost corresponding to the first image; the initial cost includes a local initial cost;

carrying out saturation operation on the initial cost to obtain updated initial cost; performing saturation operation on the initial cost to obtain updated initial cost, including:

otherwise, deleting the lowest bit of the binary corresponding to the initial cost, and updating the initial cost by using the binary value with the lowest bit deleted;

performing four-direction aggregation cost operation according to the updated initial cost to obtain a final cost;

performing parallax calculation on the final cost to determine a parallax image to be processed;

sliding and traversing the parallax image to be processed by utilizing the first sliding window;

Setting a parallax value, in which the absolute value of the difference in the partial parallax image is less than or equal to a first threshold value, to 1;

setting a parallax value, in which the absolute value of the difference value in the partial parallax image is greater than a first threshold value, to 0;

calculating the accumulated value of each updated parallax value in the local parallax image to obtain the statistical quantity;

if the statistical quantity is smaller than or equal to the second threshold value, reserving a center disparity value;

if the statistical quantity is greater than the second threshold value, setting the central parallax value to 0;

after traversing the parallax images to be processed by using the first sliding window, determining a target parallax image.

According to the method for determining the parallax images, the parallax images to be processed are traversed in a sliding mode through the first sliding window, and whether the absolute value of the difference value of each parallax value in the local parallax images and the central parallax value corresponding to the central position of the local parallax images is smaller than or equal to a first threshold value is determined in each local parallax image corresponding to the first sliding window; and counting the statistic quantity of the parallax values of which the absolute value is smaller than or equal to the first threshold value, and deleting noise points according to the statistic result to obtain the target parallax image. Compared with the prior art, the method is used for judging and calculating the local parallax image corresponding to the first sliding window, so that the complexity of the judging and calculating process can be reduced, and the convenience and the efficiency of determining the parallax image can be improved.

It should be understood that, although the steps in the flowcharts related to the above embodiments are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiment of the present application also provides a parallax image determining apparatus for implementing the above-mentioned related parallax image determining method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in the embodiment of the determining device for one or more parallax images provided below may be referred to the limitation of the determining method for parallax images hereinabove, and will not be repeated here.

In one embodiment, as shown in fig. 11, there is provided a parallax image determining apparatus including: a sliding window traversal module 1102, a connectivity determination module 1104, a statistics module 1106, an execution module 1108, and an image determination module 1110, wherein:

the sliding window traversing module 1102 is configured to perform stereo matching on the left and right camera images to obtain a parallax image to be processed, and slide and traverse the parallax image to be processed by using a first sliding window;

a communication determining module 1104, configured to determine, in each local parallax image corresponding to the first sliding window, whether a difference absolute value of each parallax value in the local parallax image and a center parallax value corresponding to a center position of the local parallax image is smaller than a first threshold;

a statistics module 1106, configured to count a number of parallax values with absolute differences smaller than a first threshold;

an execution module 1108, configured to reserve a center disparity value if the statistical quantity is less than or equal to a second threshold; if the statistical quantity is larger than a second threshold value, setting the central parallax value as a target value;

the image determining module 1110 is configured to determine a target parallax image after traversing the parallax image to be processed by using the first sliding window.

The parallax image determining device provided by the embodiment of the invention has the same beneficial effects as the parallax image determining method.

In one embodiment, the sliding window traversal module comprises:

the acquisition submodule is used for acquiring left and right camera images;

the initial cost calculation sub-module is used for calculating initial cost according to the left and right camera images;

the cost aggregation sub-module is used for carrying out cost aggregation operation according to the initial cost to obtain the final cost;

and the parallax calculation sub-module is used for carrying out parallax calculation on the final cost and determining a parallax image to be processed.

In one embodiment, the initial cost calculation submodule includes:

a first determining unit configured to traverse the first image using the second sliding window, determine each first partial pixel image corresponding to the second sliding window, and determine a first transformation vector corresponding to each first partial pixel image; the left and right camera images comprise a first image and a second image;

a second determining unit configured to determine a second transformation vector of a second partial-pixel image corresponding to the first partial-pixel image in the second image;

a third determining unit, configured to determine a local initial cost corresponding to the first local pixel image according to the first transformation vector and the second transformation vector;

a fourth determining unit, configured to obtain an initial cost corresponding to the first image after traversing the first image using the second sliding window; the initial cost includes a local initial cost.

In one embodiment, the second determining unit includes:

the first determining subunit is used for determining a preset parallax range and calculating second transformation vectors to be confirmed, wherein the second transformation vectors to be confirmed respectively correspond to second partial pixel images corresponding to the first partial pixel images in the second images in the preset parallax range; the second transformation vector to be confirmed comprises a second transformation vector;

a third determination unit including:

the second determining subunit is used for respectively determining corresponding local initial cost to be confirmed according to the first transformation vector and each second transformation vector to be confirmed;

and the third determining subunit is used for determining the minimum value in the local initial cost to be confirmed as the local initial cost.

In one embodiment, the parallax image determining apparatus further includes:

and the saturation operation module is used for carrying out saturation operation on the initial cost to obtain updated initial cost.

In one embodiment, the saturation operation module includes:

the execution submodule is used for setting the initial cost as a first preset value if the initial cost is greater than or equal to a third threshold value; otherwise, deleting the lowest bit of the binary corresponding to the initial cost, and updating the initial cost by using the binary value with the lowest bit deleted.

In one embodiment, the cost aggregation submodule includes:

and the cost aggregation unit is used for conducting four-direction cost aggregation operation according to the initial cost to obtain the final cost.

In one embodiment, the parallax image determining apparatus further includes:

the grouping module is used for grouping the local initial cost to be confirmed and the local initial cost according to a preset rule;

the storage module is used for carrying out information splicing on the local initial cost to be confirmed and the local initial cost after grouping, and storing the local initial cost to be confirmed and the local initial cost to a preset storage position according to a splicing result.

The respective modules in the above-described parallax image determination apparatus may be implemented in whole or in part by software, hardware, and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one embodiment, an integrated circuit chip is provided that, when run, performs the steps of the method described above.

The integrated circuit chip may be an FPGA (Field-Programmable Gate Array, field programmable gate array) or a CPLD (Complex Programmable Logic Device ), and the specific type of the integrated circuit chip is not limited in this embodiment.

The integrated circuit chip provided by the embodiment of the application has the same beneficial effects as the method for determining the parallax image; and the integrated circuit chip is utilized to execute the method, so that the processing speed is high.

In one embodiment, a computer device is provided, which may be a terminal device, and an internal structure diagram thereof may be as shown in fig. 12. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a method of determining a parallax image. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by those skilled in the art that the structure shown in FIG. 12 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

In an embodiment a computer device is provided comprising a memory and a processor, the memory having stored therein a computer program, the processor performing the steps of the method described above when the computer program is executed.

The computer equipment provided by the embodiment of the application has the same beneficial effects as the method for determining the parallax image.

In one embodiment, a computer readable storage medium is provided, having stored thereon a computer program which, when executed by a processor, implements the steps of the method described above.

The computer readable storage medium provided by the embodiment of the application has the same beneficial effects as the method for determining the parallax image.

In an embodiment, a computer program product is provided comprising a computer program which, when executed by a processor, implements the steps of the method described above.

The computer program product provided by the embodiment of the application has the same beneficial effects as the method for determining the parallax image.

The user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or sufficiently authorized by each party.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.

Claims

1. A method of determining a parallax image, the method comprising:

2. The method according to claim 1, wherein the stereo matching of the left and right camera images to obtain the parallax image to be processed includes:

acquiring the left and right camera images;

calculating initial cost according to the left and right camera images;

3. The method of claim 2, wherein said calculating an initial cost from said left and right camera images comprises:

4. A method according to claim 3, wherein said determining a second transformation vector for a second partial-pixel image of said second image corresponding to said first partial-pixel image comprises:

5. A method according to claim 3, wherein after said calculating an initial cost from said left and right camera images and before said performing a cost aggregation operation from said initial cost to obtain a final cost, the method further comprises:

6. The method of claim 5, wherein saturating the initial cost to obtain the updated initial cost comprises:

7. The method according to claim 2, wherein the performing a cost aggregation operation according to the initial cost to obtain a final cost includes:

8. The method according to claim 4, wherein the method further comprises:

9. The method of any one of claims 1 to 8, wherein the left and right camera images comprise:

10. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 9 when the computer program is executed.