WO2023171203A1

WO2023171203A1 - Imaging device

Info

Publication number: WO2023171203A1
Application number: PCT/JP2023/003972
Authority: WO
Inventors: 邦博今村
Original assignee: パナソニックＩｐマネジメント株式会社
Priority date: 2022-03-08
Filing date: 2023-02-07
Publication date: 2023-09-14

Abstract

An imaging device (1) comprises a projection unit (10) that projects light having a substantially uniform polarization direction, an imaging unit (20) that captures a subject onto which the light is projected, and a signal processing unit (31) that processes a captured image obtained by the imaging unit (20) to measure the distance to the subject. The imaging unit (20) comprises an imaging lens (22), a polarizing filter (23) that extracts light having the same polarization direction as the light emitted from the projection unit (10), and an image sensor (21) that receives light from the subject through the imaging lens (22) and the polarizing filter (23). The signal processing unit (31) measures the distance for each area on the captured image without limiting the number of distances to measurement subjects to one.

Description

Imaging device

The present invention relates to an imaging device that can measure the distance to a subject.

Conventionally, imaging devices are known that measure the distance to a subject by irradiating the subject with light. This type of imaging device uses, for example, a method that uses a stereo camera to detect parallax to measure the distance to the subject, or a method that measures the time difference between projecting light and receiving the reflected light (TOF: Time Of Flight). A method of detecting and measuring the distance to the subject, or a method of illuminating the subject with light with a unique pattern (intensity distribution) and measuring the distance to the subject by detecting the amount of pixel shift of the pattern with respect to the reference image as parallax. A method to do so is used.

Patent Document 1 below describes an imaging device that irradiates a subject with light having a unique pattern (intensity distribution).

Patent No. 6657880

In each of these types of imaging devices, the subject to be imaged may be composed of a transparent member such as glass and a member located behind the transparent member. In this case, since light is reflected by both the transparent member in the front and the member behind it, it is difficult to properly measure the distance to the transparent member.

In view of such problems, the present invention is capable of appropriately measuring the distance to the transparent member when the subject is composed of a transparent member and a member located behind the transparent member, and furthermore, it is possible to appropriately measure the distance to the transparent member. It is an object of the present invention to provide an imaging device that can also measure the distance to a certain member.

An imaging device according to a main aspect of the present invention includes a projection unit that projects light with a substantially uniform polarization direction, an imaging unit that images a subject onto which the light is projected, and processes a captured image acquired by the imaging unit. and a signal processing unit that measures the distance to the subject. The imaging unit includes an imaging lens, a polarizing filter that extracts light having the same polarization direction as the light emitted from the projection unit, and an imaging element that receives light from the subject via the imaging lens and the polarizing filter. and. The signal processing unit measures the distance for each region on the captured image without limiting the distance to be measured to one.

According to the imaging device according to this aspect, the polarizing filter that extracts light having the same polarization direction as the polarization direction of the light projected from the projection unit is disposed in the imaging unit, so that the polarization filter that extracts light with the same polarization direction as the polarization direction of the light projected from the projection unit is disposed in the imaging unit. light is more easily received by the image sensor. Therefore, when the subject is composed of a transparent member and a member located behind the transparent member, the distance to the transparent member can be appropriately detected. In addition, the signal processing unit measures the distance without limiting the distance to be measured to one, so in addition to the distance to the transparent member, it also measures the distance based on the light reflected by the member behind it. Can be obtained. Therefore, when the subject is composed of a transparent member and a member located at the back, the distance to the member located at the back can also be measured.

As described above, according to the present invention, when a subject is composed of a transparent member and a member located behind the transparent member, it is possible to appropriately measure the distance to the transparent member, and furthermore, it is possible to appropriately measure the distance to the transparent member. It is possible to provide an imaging device that can also measure the distance to a member located in the area.

The effects and significance of the present invention will become clearer from the description of the embodiments shown below. However, the embodiment shown below is merely one example of implementing the present invention, and the present invention is not limited to what is described in the embodiment below.

FIG. 1 is a diagram showing the configuration of an imaging device according to an embodiment. FIGS. 2A and 2B are diagrams each schematically showing a method of setting pixel blocks for a captured image according to the embodiment. FIGS. 3A to 3D are diagrams each schematically showing a process of searching a reference image for a matching pixel block that matches a target pixel block on a captured image, according to an embodiment. . FIG. 4 is a diagram schematically illustrating an imaging state of pattern light in a case where the subject is composed of a transparent member and a member located behind the transparent member, according to the embodiment. FIG. 5 is a graph illustrating the relationship between the search position and the correlation value when pattern light from two members is incident on one pixel block, according to the embodiment. FIG. 6 is a flowchart illustrating processing for obtaining distance to a subject according to the embodiment. FIG. 7 is a flowchart illustrating a distance acquisition process to a subject according to modification example 1. FIG. 8 is a flowchart illustrating a process for obtaining a distance to a subject according to modification example 2. FIG. 9 is a graph illustrating the relationship between the search position and the correlation value when pattern light from two members is incident on one pixel block according to Modification Example 2. FIG. 10 is a diagram for explaining interpolation processing according to modification example 3. FIG. 11 is a diagram showing the configuration of an imaging device according to modification example 4. FIG. 12 is a flowchart illustrating a process for obtaining a distance to a subject according to modification example 4.

However, the drawings are solely for illustrative purposes and do not limit the scope of the invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. For convenience, mutually orthogonal X, Y, and Z axes are shown in each figure. The X-axis direction is the direction in which the projection section and the imaging section are lined up, and the positive Z-axis direction is the imaging direction of the imaging section.

FIG. 1 is a diagram showing the configuration of an imaging device 1.

The imaging device 1 includes a projection section 10, an imaging section 20, and an image processing section 30.

The projection unit 10 projects pattern light in which light is distributed in a predetermined pattern onto the field of view of the imaging unit 20. The direction in which the pattern light is projected by the projection unit 10 is the positive direction of the Z-axis. The projection unit 10 includes a light source 11 , a collimator lens 12 , a pattern generator 13 , a projection lens 14 , and a polarizing filter 15 .

The light source 11 emits light of a predetermined wavelength. The light source 11 is, for example, an LED (Light Emitting Diode). The light source 11 emits light in an infrared wavelength band, for example. The light source 11 may be another type of light source such as a semiconductor laser. The collimator lens 12 converts the light emitted from the light source 11 into substantially parallel light.

The pattern generator 13 generates pattern light having a unique pattern (intensity distribution) using the light emitted from the light source 11. In this embodiment, the pattern generator 13 is a transmissive optical diffraction element (DOE). An optical diffraction element (DOE), for example, has a diffraction pattern with a predetermined number of steps on its entrance surface. Due to the diffraction effect of this diffraction pattern, the laser beam that has entered the diffractive optical element (pattern generator 13) is split into a plurality of beams and converted into a predetermined pattern of light. The generated pattern is a pattern that can maintain uniqueness for each pixel block 212, which will be described later.

In this embodiment, the pattern generated by the optical diffraction element (DOE) is a pattern in which a plurality of dot areas (hereinafter referred to as "dots"), which are light passage areas, are randomly distributed. However, the pattern generated by the optical diffraction element (DOE) is not limited to a pattern of dots, and may be any other pattern. Furthermore, the pattern generator 13 may be a reflective optical diffraction element or a photomask. Alternatively, the pattern generator 13 may be a device that generates a fixed pattern of pattern light based on a control signal, such as a DMD (Digital Mirror Device) or a liquid crystal display.

The projection lens 14 projects the pattern light generated by the pattern generator 13. The projection lens 14 does not need to be one lens, and may be configured by combining a plurality of lenses. Further, instead of the projection lens 14, a concave reflecting mirror may be used. The optical axis of the projection lens 14 is parallel to the Z axis.

The polarizing filter 15 selectively transmits light in a predetermined polarization direction and blocks light in a polarization direction perpendicular to this polarization direction. Here, the polarizing filter 15 selectively transmits light with a polarization direction parallel to the X-axis and blocks light with a polarization direction parallel to the Y-axis direction. Therefore, the light transmitted through the polarizing filter 15 has a substantially uniform polarization direction.

The position of the polarizing filter 15 is not limited to the rear side of the projection lens 14 (positive side of the Z-axis) when viewed from the light source 11, but may be between the light source 11 and the projection lens 14. Furthermore, the polarizing filter 15 does not necessarily have to be an independent member, but may be a surface of another member through which light from the light source 11 passes, such as the surface of the projection lens 14, the collimator lens 12, or the pattern generator 13. The polarizing film may be integrally formed with the polarizing film.

Furthermore, if the light source 11 is a laser light source such as a semiconductor laser, the polarizing filter 15 may be omitted. In this case, the light source 11 (laser light source) may be arranged so that the direction of linearly polarized light is parallel to the X axis.

The imaging unit 20 images the subject onto which the pattern light is projected. The imaging unit 20 includes an imaging element 21, an imaging lens 22, and a polarizing filter 23.

The image sensor 21 is a CMOS image sensor. The image sensor 21 may be a CCD. A filter that selectively transmits light in the wavelength band emitted by the light source 11 is formed on the imaging surface of the image sensor 21 . Alternatively, a similar filter may be arranged within the imaging section 20 separately from the imaging element 21.

The imaging lens 22 focuses light from the viewing range onto the imaging surface of the imaging element 21. The optical axis of the imaging lens 22 is parallel to the Z-axis. That is, the optical axis of the imaging lens 22 and the optical axis of the projection lens 14 are parallel to each other. The optical axis of the imaging lens 22 may be inclined with respect to the optical axis of the projection lens 14 so that the optical axis of the imaging lens 22 approaches the optical axis of the projection lens 14 in the positive Z-axis direction.

The polarizing filter 23 selectively transmits light in the same polarization direction as the light (pattern light) emitted from the projection unit 10, and blocks light in a polarization direction perpendicular to this polarization direction. Here, the polarizing filter 23 selectively transmits light in a polarization direction parallel to the X-axis and blocks light in a polarization direction parallel to the Y-axis direction.

The position of the polarizing filter 23 is not limited to the latter side of the imaging lens 22 (positive side of the Z-axis) when viewed from the imaging element 21, but may be between the imaging element 21 and the imaging lens 22. Further, the polarizing filter 23 does not necessarily have to be an independent member, and may have a configuration in which a polarizing film is integrally formed on the surface of the imaging lens 22 through which the light from the light source 11 passes.

The image processing section 30 includes a signal processing section 31, a light source driving section 32, an imaging processing section 33, and a communication interface 34.

The signal processing section 31 includes an arithmetic processing circuit such as a microcomputer and a memory, and controls each section according to a predetermined program. The signal processing unit 31 also processes pixel signals input from the imaging processing unit 33 to calculate the distance to the subject.

The signal processing unit 31 holds a reference image in which dots are distributed in a pattern similar to the pattern light projected from the projection unit 10. The reference image corresponds to a captured image when the subject is located at a predetermined distance. The signal processing unit 31 compares and processes this reference image with the captured image captured by the imaging processing unit 33, searches for stereo corresponding points, and searches for stereo corresponding points up to the subject in the direction corresponding to each pixel block on the captured image. Calculate distance.

That is, the signal processing unit 31 sets a pixel block from which the distance is to be obtained (hereinafter referred to as a "target pixel block") on the captured image, and sets a pixel block corresponding to this target pixel block, that is, a target pixel block. A pixel block that best matches (hereinafter referred to as a "compatible pixel block") is searched for in a search range defined on the reference image. The signal processing unit 31 then processes pixels between a pixel block located at the same position as the target pixel block on the reference image (hereinafter referred to as "reference pixel block") and the matching pixel block extracted from the reference image by the above search. The amount of deviation is obtained, and the distance to the subject at the position of the target pixel block is calculated from the obtained amount of pixel deviation by calculation based on the triangulation method.

The signal processing unit 31 performs such distance calculation for all pixels (pixel signals) of the captured image from the image sensor 21. The signal processing unit 31 transmits one screen worth of distance (distance image) acquired through this processing to an external device via the communication interface 34.

The signal processing unit 31 may perform the distance calculation process using a semiconductor integrated circuit consisting of an FPGA (Field Programmable Gate Array). Alternatively, this processing may be performed by other semiconductor integrated circuits such as a DSP (Digital Signal Processor), a GPU (Graphics Processing Unit), and an ASIC (Application Specific Integrated Circuit).

The light source driving section 32 drives the light source 11 under control from the signal processing section 31. The image processing unit 33 controls the image sensor 21 and performs processing such as brightness correction and camera calibration on the pixel signal of the captured image output from the image sensor 21.

FIGS. 2(a) and 2(b) are diagrams schematically showing a method of setting the pixel block 212 for the image sensor 21. FIG. 2(a) shows a method of setting the pixel blocks 212 for the entire imaging surface, and FIG. 2(b) shows an enlarged view of a part of the imaging surface.

As shown in FIGS. 2A and 2B, the imaging surface of the image sensor 21 is divided into a plurality of pixel blocks 212 each including a predetermined number of pixel regions 211. The pixel area 211 is an area corresponding to one pixel on the image sensor 21. In the examples shown in FIGS. 2A and 2B, one pixel block 212 is composed of nine pixel regions 211 arranged in three rows and three columns. However, the number of pixel regions 211 included in one pixel block 212 is not limited to this.

On the imaging surface of the image sensor 21, dots of pattern light that are projected from the projection unit 10 and reflected by the subject are projected. Therefore, pixel block 212 includes a portion of the projected dot group. Because the dots are distributed in a random pattern. The distribution of dots included in each pixel block 212 is also unique. Thereby, the distance to the subject in each pixel block 212 is appropriately calculated by searching for stereo corresponding points.

FIGS. 3(a) to 3(d) schematically illustrate the process of searching a reference image for a compatible pixel block that matches the target pixel block TB1 (pixel block 212 from which distance is to be obtained) on the captured image. FIG.

First, as shown in FIG. 3(a), the signal processing unit 31 sets a search range R0 on the reference image. A starting position ST1 of the search range R0 is set to a position corresponding to the target pixel block TB1 on the reference image. The search range R0 extends from the start position ST1 in the direction of separation between the projection unit 10 and the imaging unit 20 by a predetermined number of pixels (the number of pixels corresponding to the distance detection range). The vertical width of the search range R0 is the same as the vertical width of the target pixel block TB1.

The signal processing unit 31 sets the start position ST1 as the search position. The signal processing unit 31 sets a reference pixel block RB1 having the same size as the target pixel block TB1 at this search position, and calculates a correlation value between the target pixel block TB1 and the reference pixel block RB1.

Here, the correlation value is, for example, a value obtained by calculating differences in pixel values (luminance) for mutually corresponding pixel areas of the target pixel block TB1 and reference pixel block RB1, and integrating all the absolute values of the calculated differences. (SAD). Alternatively, the correlation value may be obtained as a value (SSD) obtained by integrating all the squared values of the differences. However, the method of calculating the correlation value is not limited to these, and other calculation methods may be used as long as a correlation value that is an index of the correlation between the target pixel block TB1 and the reference pixel block RB1 can be obtained. .

When the processing for one reference pixel block RB1 is thus completed, the signal processing unit 31 sets the next search position, as shown in FIG. 3(b). Specifically, the signal processing unit 31 sets a position obtained by shifting the previous search position by one pixel toward the end of the search range R0 as the current search position. Then, the signal processing unit 31 calculates the correlation value between the reference pixel block RB1 and the target pixel block TB1 at the current search position by processing similar to the above.

The signal processing unit 31 repeatedly executes similar processing while shifting the search position by one pixel in the terminal direction. 3(c) shows a state in which the reference pixel block RB1 is set at the previous search position from the last search position in the search range R0, and FIG. 3(d) shows the state in which the reference pixel block RB1 is set at the last search position in the search range R0. This shows a state in which the reference pixel block RB1 is set.

In this way, when the processing for the last search position is completed, the signal processing unit 31 specifies the search position where the correlation value is the peak among the sequentially set search positions, and determines the pixel shift between the specified search position and the start position ST1. Based on the amount, the distance to the subject is calculated by triangulation. The signal processing unit 31 repeats the same process for all target pixel blocks TB1 on the captured image. After calculating the distances to the subject for all target pixel blocks TB1 in this way, the signal processing unit 31 transmits a distance image in which the calculated distances are mapped to each target pixel block TB1 to an external device via the communication interface 34. do.

FIG. 4 is a diagram schematically showing the imaging state of the patterned light L0 when the subject 40 is composed of a transparent member 41 such as glass and a member 42 located behind the transparent member 41.

In the subject 40 shown in FIG. 4, the patterned light L0 is reflected on both the surface of the member 41 and the member 42 located deep therein. That is, part of the pattern light L0 is specularly reflected on the surface of the member 41, and the rest is transmitted through the member 41. The pattern light L0 that has passed through the member 41 is diffusely reflected on the surface of the member 42 on the back side, and is directed toward the member 41 on the front side. Thereafter, part of this pattern light passes through the member 41.

Therefore, the imaging lens 22 of the imaging unit 20 captures the pattern light specularly reflected on the surface of the front member 41 and the pattern light diffusely reflected on the surface of the rear member 42. Therefore, these two types of pattern light are guided to the image sensor 21. At this time, in the image sensor 21, the above two types of pattern light may be incident on the same pixel block 212.

For example, in the example of FIG. 4, a light portion L1 and a light portion L2 of the pattern light L0 projected from the projection unit 10 are guided to the same pixel block 212. Therefore, this pixel block 212 includes dots DT included in the light portion L1 and dots DT included in the light portion L2.

FIG. 5 is a graph illustrating the relationship between the search position and the correlation value when pattern light from two

members

41 and 42 is incident on one pixel block 212. Here, SAD or SSD is assumed as the correlation value on the vertical axis.

In this example, two peak positions are obtained in a range smaller than the threshold Th0 by the processing shown in FIGS. 3(a) to 3(d). For example, the search position P1 in FIG. 4 is a search position where the dot pattern of the light portion L1 specularly reflected on the surface of the front member 41 matches the reference image, and the search position P2 is the search position on the back side in FIG. The dot pattern of the light portion L2 that is specularly reflected on the surface of the member 42 is the search position that matches the reference image. In this way, in the object as shown in FIG. 4, the pattern light from the surfaces of the

members

41 and 42 can be incident on the same pixel block 212, so two search positions can be obtained for each pixel block 212.

In FIG. 5, the threshold Th0 is used to prevent this search position from being detected as a search position corresponding to the distance to the subject when the correlation value reaches a peak due to the influence of noise or the like. The threshold Th0 may be a fixed value, or may be obtained by multiplying the maximum value, average value, median value, or mode of the correlation values obtained at all search positions in the search range R0 by a predetermined ratio, Alternatively, it may be set to a value obtained by adding or subtracting a predetermined value.

Note that although FIG. 5 shows an example in which two search positions P1 and P2 are obtained for the

members

41 and 42, the dots DT from the

members

41 and 42 overlap each other, so that It is also possible that the uniqueness of the dot pattern is reduced. In this case, it may be assumed that the peak of the correlation value included in the range smaller than the threshold Th0 may not be obtained. In such a case, the distance to the pixel block 212 may be interpolated using the surrounding pixel blocks 212. As a result, a distance image in which distances are mapped to all pixel blocks 212 can be obtained.

FIG. 6 is a flowchart showing the process of obtaining the distance to the subject.

The signal processing unit 31 acquires a captured image from the image sensor 21 (S101). Next, the signal processing unit 31 executes the corresponding point search process shown in FIGS. 3A to 3D on the target pixel block TB1 on the captured image (S102). Then, the signal processing unit 31 determines whether a plurality of corresponding points (compatible search positions) are obtained in the corresponding point search process (S103). That is, it is determined whether there are a plurality of correlation value peaks in a range smaller than the threshold value Th0 shown in FIG. 5, and thereby a plurality of search positions matching the target pixel block TB1 have been obtained.

If a plurality of corresponding points are obtained for the target pixel block TB1 (S103: YES), the signal processing unit 31 selects the corresponding point with the shortest pixel shift from the reference position (starting position ST1 in FIG. 3(a)). The point is selected as a corresponding point for distance acquisition (S104). Then, the signal processing unit 31 calculates the distance in the target pixel block TB1 using the triangulation method from the pixel shift amount of the selected corresponding point with respect to the start position ST1 (S105).

On the other hand, if only one corresponding point is obtained for the target pixel block TB1 (S103: NO, S109: YES), the signal processing unit 31 calculates a triangle from the pixel shift amount of the obtained corresponding point with respect to the starting position ST1. The distance in the target pixel block TB1 is calculated by the surveying method (S105). Further, if a corresponding point is not obtained for the target pixel block TB1 (S103: NO, S109: NO), the signal processing unit 31 returns the process to step S106 without calculating the distance to the target pixel block TB1. Proceed. In this case, the signal processing unit 31 sets a flag indicating that the distance cannot be obtained for the target pixel block TB1.

Here, since the corresponding point selected in step S104 is the corresponding point with the shortest pixel shift from the start position ST1, in the example of FIG. There is a high possibility that the dot pattern is a search position that matches the reference image. Further, as shown in FIG. 1, since the imaging unit 20 is configured to mainly extract pattern light specularly reflected by the subject using the polarizing filter 23, the determination in step S109 is YES and there is only one corresponding point. If the corresponding point is obtained, there is a high possibility that the corresponding point is a search position where the dot pattern of the light portion L1 specularly reflected on the surface of the member 41 closest to the front matches the reference image.

Therefore, in the flowchart of FIG. 6, the distance is calculated for the search position (S104 or S109: YES) where the dot pattern of the light portion L1 specularly reflected on the surface of the nearest member 41 matches the reference image (S104 or S109: YES). S105) If such a search position is not obtained (S109: NO), the process proceeds to step S106 without calculating the distance.

After performing the processing on the target pixel block TB1, the signal processing unit 31 determines whether distance calculation processing has been performed on all the pixel blocks 212 on the captured image (S106). If the processing has not been completed for all the pixel blocks 212 (S106: NO), the signal processing unit 31 sets the next pixel block 212 as the target pixel block TB1, and executes the processing from step S102 onwards.

In this way, when the distance calculation process is completed for all the pixel blocks 212 on the captured image (S106: YES), the signal processing unit 31 calculates the distance for the pixel blocks 212 for which the distance was not obtained, that is, the determination in step S109 is completed. A distance interpolation process is executed for the flagged pixel block 212 with a NO answer (S107). More specifically, the signal processing unit 31 interpolates the distance to the pixel block 212 for which the distance could not be obtained using distances around the pixel block 212. As a result, a distance image in which distances are associated with all the pixel blocks 212 is constructed.

The signal processing unit 31 transmits the distance image configured in this way to an external device via the communication interface 34 (S108). Thereby, the signal processing unit 31 ends the process of FIG.

<Effects of embodiment>
According to the above embodiment, the following effects are achieved.

As shown in FIG. 1, the polarizing filter 23 that extracts light having the same polarization direction as that of the light projected from the projection section 10 is disposed in the imaging section 20, so that the projection section is mirror-reflected on the surface of the subject. The light from 10 is more easily received by the image sensor 21. Therefore, as shown in FIG. 4, when the subject 40 is composed of a transparent member 41 and a member 42 located behind the transparent member 41, the distance to the transparent member 41 can be appropriately detected.

As shown in FIG. 6, when a plurality of distances can be obtained for the region (target pixel block TB1) for which distances are to be obtained (S103: YES), the signal processing unit 31 selects the shortest distance as the measurement result. (S104, S105). Thereby, as explained with reference to FIG. 5, the distance to the transparent member 41 can be appropriately detected.

As shown in FIG. 1, the projection unit 10 uses the pattern generator 13 to generate pattern light with a predetermined intensity distribution and projects it onto the subject, and the signal processing unit 31 generates pattern light with a predetermined intensity distribution when the subject is located at a predetermined distance. The distance to the subject is measured by comparing the reference image, which is the captured image, with the captured image captured by the imaging unit 20. Thereby, even if the reflectance and light absorption rate of the subject surface are uniform, the distance to each position on the subject surface can be measured. Moreover, the distance to the transparent member 41 can be appropriately detected by the process shown in FIG.

As shown in FIG. 1, the projection unit 10 includes a polarization filter 15 for generating substantially uniformly polarized light. Thereby, even if the polarization direction of the light source 11 is random, light can be projected onto the subject in a substantially uniform polarization direction.

<Change example 1>
In the above embodiment, as shown in FIG. 6, when a plurality of distances are obtained for the area (target pixel block TB1) for which distances are to be obtained (S103: YES), the shortest distance is selected as the measurement result. (S104, S105). On the other hand, in modification example 1, when a plurality of distances can be obtained for the region (target pixel block TB1) for which distances are to be obtained, the longest distance is selected as the measurement result.

FIG. 7 is a flowchart illustrating a process for obtaining a distance to a subject according to modification example 1.

In the flowchart of FIG. 7, step S104 of the flowchart of FIG. 6 is replaced with step S111, and step S109 of FIG. 6 is omitted. The processing in other steps in FIG. 7 is similar to the processing in the corresponding steps in FIG.

Referring to FIG. 7, when a plurality of corresponding points are extracted for the target pixel block TB1 through the corresponding point search (S102) (S103: YES), the signal processing unit 31 detects the pixel deviation from the starting position ST1. The corresponding point with the longest is selected as the corresponding point for distance acquisition (S111). Then, the signal processing unit 31 calculates the distance in the target pixel block TB1 using the triangulation method from the pixel shift amount of the selected corresponding point with respect to the start position ST1 (S105).

On the other hand, if a plurality of corresponding points are not obtained for the target pixel block TB1 (S103: NO), the signal processing unit 31 advances the process to step S106 without calculating the distance to the target pixel block TB1. In this case, the signal processing unit 31 sets a flag indicating that the distance cannot be obtained for the target pixel block TB1. Regarding this target pixel block TB1, the distance value is interpolated in step 107 as in the above embodiment.

According to the configuration of modification example 1, when a plurality of distances can be obtained for the region (target pixel block TB1) for which distances are to be obtained (S103: YES), the longest distance is selected as the measurement result (S111, S105). Thereby, as explained with reference to FIG. 5, the distance to the member 42 located at the back of the transparent member 41 can be detected.

Note that in the flowchart of FIG. 7, if the determination in step S103 is NO, only one corresponding point may be obtained for the target pixel block TB1. However, as shown in FIG. 1, since the imaging unit 20 is configured to mainly extract pattern light specularly reflected by the subject using the polarizing filter 23, when only one corresponding point is obtained in this way, The corresponding point is likely to be a search position where the dot pattern of the light portion L1 specularly reflected on the surface of the member 41 closest to the user matches the reference image. Therefore, the distance acquired from the corresponding point is likely to be the distance to the transparent member 41 in the front, rather than the distance to the member 42 on the back side. Therefore, if this distance is included in the measurement result, a distance image that includes only the distance to the member 42 on the back side cannot be obtained with high accuracy.

From this point of view, in the flowchart of FIG. 7, step S109 of FIG. 6 is omitted, and if a plurality of corresponding points are not extracted for the target pixel block TB1, the distance is uniformly obtained for the target pixel block TB1. treated as if it had not happened. This makes it easier for the distance image to include only the distance to the member 42 on the back side, and it is possible to improve the accuracy of the distance image.

Note that in the process of FIG. 6 as well, step S109 may be omitted and the process may proceed to step S106 if the determination in step S103 is NO. This process also makes it possible to accurately acquire a distance image that includes the distance to the transparent member 41 in the foreground.

<Change example 2>
In modification example 2, when a distance included in a preset range is obtained for the region (target pixel block TB1) for which the distance is to be obtained, this distance is obtained as the measurement result.

FIG. 8 is a flowchart illustrating a process for obtaining a distance to a subject according to modification example 2.

In the flowchart of FIG. 8, steps S103 and S111 in the flowchart of FIG. 7 are replaced with steps S112 and S113. The processing in other steps in FIG. 8 is similar to the processing in the corresponding steps in FIGS. 6 and 7.

Referring to FIG. 8, if a corresponding point is extracted within the set range (S103: YES) by the corresponding point search (S102), the signal processing unit 31 converts the corresponding point into a corresponding point for distance acquisition. (S113). Then, the signal processing unit 31 calculates the distance in the target pixel block TB1 using the triangulation method from the pixel shift amount of the selected corresponding point with respect to the start position ST1 (S105).

On the other hand, if no corresponding point is obtained within the set range (S112: NO), the signal processing unit 31 advances the process to step S106 without calculating the distance to the target pixel block TB1. In this case, the signal processing unit 31 sets a flag indicating that the distance cannot be obtained for the target pixel block TB1. Regarding this target pixel block TB1, the distance value is interpolated in step 107 as in the above embodiment.

The setting range in step S112 is set based on a command received from an external device via the communication interface 34, for example. For example, when the external device is a robot arm that grips an article and places it on an object on a belt conveyor, the imaging device 1 can be installed in the gripping section of the robot arm. In this case, the placed object is the subject. This placed object has, for example, a transparent member 41 and a member 42 at the back thereof as shown in FIG.

The robot arm roughly calculates the distance between the gripping part and the object (corresponding to the distance to the transparent member 41 that is closest to the front of the object) based on the current position of the gripping part and the position of the object. Then, a predetermined distance range including this distance is transmitted to the signal processing section 31. The signal processing unit 31 sets the search range corresponding to the received distance range as the set range in step S112.

Alternatively, the robot arm processes a signal to determine the approximate value of the distance between the gripping part and the object to be placed (corresponding to the distance to the transparent member 41 that is closest to the front of the object) or the position of the gripping part and the object to be placed. It may also be transmitted to the section 31. In this case, the signal processing unit 31 sets a predetermined search range centered on the search position corresponding to the distance to the nearest transparent member 41 to the range set in step S112 based on the received information. Just set it.

According to the configuration of modification example 2, for example, as shown in FIG. The corresponding points (search position P1) included in the range ΔW are selected as distance acquisition targets. Therefore, the distance to the transparent member 41 located closest to the user can be appropriately obtained.

In addition, in the example of FIG. 9, the setting range ΔW was set so that the distance to the transparent member 41 located closest to the front was obtained, but the setting range ΔW was set so that the distance to the member 42 on the far side was obtained. ΔW may be set. In this case, the corresponding point (search position P2) corresponding to the distance to the far side member is selected as the distance acquisition target. Thereby, the distance to the member 42 located at the back of the transparent member 41 can be appropriately detected.

<Change example 3>
In the embodiment described above, in the interpolation process in step S107 in FIG. 6, interpolation is performed using the pixel blocks 212 around the pixel block 212 for which the distance was not acquired. In contrast, in modification example 3, the range of pixel blocks 212 used for interpolation is expanded.

FIG. 10 is a diagram for explaining interpolation processing according to modification example 3.

For example, if fine irregularities suddenly occur in a part of the surface of the transparent member 41 shown in FIG. 4, specular reflection is unlikely to occur on the surface of the member 41 in this range, and diffuse reflection becomes dominant. . Therefore, in this range, it is difficult to extract corresponding points corresponding to the surface of the member 41 in the corresponding point search, and as a result, it is difficult to obtain the distance to the member 41.

For example, in FIG. 10, in the surrounding range R11 including the pixel block 212 to be interpolated, due to the above phenomenon, diffuse reflection becomes dominant, and the distance to the surface of the transparent member 41 may become difficult to calculate. That is, even in the pixel blocks 212 surrounding the pixel block 212 to be interpolated, the distance acquisition may become unstable, such as the distance not being acquired due to the influence of unexpectedly occurring minute irregularities. Therefore, when surrounding pixel blocks 212 are used for interpolation, it may not be possible to properly interpolate the distance to the pixel block 212 to be interpolated.

Therefore, in modification example 3, the range of the pixel block 212 used for interpolation is expanded from the range of the pixel blocks 212 surrounding the pixel block 212 to be interpolated. For example, in the example shown in FIG. 10, a range R12 that is wider than the range R11 in which diffuse reflection is normally assumed to be dominant due to fine irregularities, etc. is set around the pixel block 212 to be interpolated, and this range R12 is The distance of the pixel block 212 to be interpolated is interpolated from the distance of the included pixel block 212. As the interpolation method, a conventionally known interpolation method such as linear interpolation may be applied.

According to modification example 3, as described above, the range to be interpolated is expanded, so that the distance to the pixel block 212 to be interpolated can be interpolated more appropriately. Therefore, the quality of the distance image in which the distance to the surface of the transparent member 41 is mapped can be improved.

<Change example 4>
In modification example 4, an irradiation position changing unit that changes the irradiation position of the pattern light on the subject is arranged in the imaging device 1. Furthermore, in modification example 4, an imaging position changing unit that changes the imaging position of the imaging unit 20 with respect to the subject is arranged in the imaging device 1.

FIG. 11 is a diagram showing the configuration of the imaging device 1 according to modification example 4.

The projection unit 10 is supported by a support shaft 51 parallel to the Y-axis. The support shaft 51 is fixed to the center of the gear 52. The gear 52 meshes with a gear 54 fixed to a drive shaft of a motor 53. Motor 53 is, for example, a stepping motor. When the motor 53 is driven, the gear 52 rotates and the support shaft 51 rotates. As a result, the projection unit 10 rotates about the support shaft 51 in parallel to the XZ plane. Thereby, the irradiation position of the pattern light on the subject 40 is changed in the X-axis direction. The support shaft 51, the gear 52, the motor 53, and the gear 54 constitute an irradiation position changing unit 50 that changes the irradiation position of the pattern light onto the subject.

The imaging unit 20 is supported by a support shaft 61 parallel to the Y-axis. The support shaft 61 is fixed to the center of the gear 62. The gear 62 meshes with a gear 64 fixed to the drive shaft of a motor 63. Motor 63 is, for example, a stepping motor. When the motor 63 is driven, the gear 62 rotates and the support shaft 61 rotates. As a result, the imaging unit 20 rotates about the support shaft 61 in parallel to the XZ plane. Thereby, the imaging position of the imaging unit 20 with respect to the subject 40 is changed in the X-axis direction. The support shaft 61, the gear 62, the motor 63, and the gear 64 constitute an imaging position changing unit 60 that changes the imaging position of the imaging unit 20 with respect to the subject.

According to this configuration, when the irradiation position changing unit 50 changes the irradiation position of the pattern light on the subject, the dot pattern of the pattern light that enters each pixel block 212 on the image sensor 21 changes. In this case, in each pixel block 212, both the dot pattern of the pattern light specularly reflected on the surface of the transparent member 41 and the dot pattern of the pattern light diffusely reflected on the surface of the member 42 behind it change. . That is, in each pixel block 212, the combination of dot patterns of the pattern light incident from the transparent member 41 and the member 42 changes before and after the irradiation position of the pattern light on the subject is changed by the irradiation position changing unit 50.

Therefore, for the pixel block 212 whose dot pattern specificity had decreased due to the superposition of these two dot patterns before the irradiation position was changed, the dot pattern specificity can be restored by changing the irradiation position. As a result, as shown in FIG. 5, a plurality of corresponding points (search positions P1, P2) may be obtained. Therefore, the distance to the pixel block 212 can be obtained without interpolation processing using the corresponding points obtained after changing the irradiation position. In this case, the signal processing unit 31 may further take into consideration the change in the irradiation position by the irradiation position changing unit 50 and perform distance calculation processing based on the corresponding points.

FIG. 12 is a flowchart illustrating a process for obtaining a distance to a subject according to modification example 4.

In this process, the signal processing section 31 further controls the irradiation position changing section 50. The signal processing unit 31 first executes the processes of steps S101 to S106 and S109 in the same manner as in FIG. 6, with the projection unit 10 set at the same position as in the above embodiment without changing the irradiation position. Next, the signal processing unit 31 drives the motor 53 of the irradiation position changing unit 50 by a predetermined number of steps to change the irradiation position of the pattern light (S121). Then, the signal processing unit 31 remeasures the distance for the pixel block 212 for which the distance could not be obtained in the processes of steps S101 to S106 and S109 (S122). Specifically, the signal processing unit 31 performs the same stereo corresponding point search as in step S102 again on these pixel blocks 212, and calculates the distances in steps S103 to S105 and S109.

If there is a pixel block 212 for which the distance could not be obtained even through this re-measurement process, the signal processing unit 31 performs interpolation processing on this pixel block 212 (S107). Once the distances have been set for all the pixel blocks 212 in this way, the signal processing unit 31 transmits one screen worth of distance images to the external device via the communication interface 34.

According to modification example 4, as described above, since the projection position of the pattern light on the subject can be changed by the irradiation position changing unit 50, the distance can be obtained by, for example, the processing of steps S102 to S105 and S109 in FIG. The distance to the pixel block 212 that did not exist can be obtained without interpolation processing by the processing in steps S121 and S122. Therefore, the quality of the distance image can be improved.

Note that in the process of FIG. 12, the irradiation position is changed only once, but the irradiation position may be changed multiple times and the distance to the pixel block 212 for which the distance could not be obtained may be remeasured. . As a result, the number of pixel blocks 212 for which distances could not be finally obtained is reduced, and the number of pixel blocks 212 targeted for the interpolation process in step S107 is reduced. Therefore, the quality of the distance image can be further improved.

Further, the process of re-measuring the distance in response to a change in the irradiation position may also be applied to the processes in FIGS. 7 and 8. Thereby, the quality of the distance image obtained by these processes can be improved.

Further, in the above, the irradiation position is changed by driving the motor 53 several steps, but the irradiation position may be changed significantly to the extent that the pattern light projection range is changed to another range. . This allows the distance to the subject to be measured over a wider range.

However, in this case, the pattern light may not be projected onto the image sensor 21 due to a change in the irradiation position. In such a case, the signal processing section 31 may drive the imaging position changing section 60 so that the pattern light is projected onto the imaging device 21. For example, the signal processing unit 31 stores in advance a table in which the drive direction and drive amount (number of steps) of the motor 53 are associated with the drive direction and drive amount (step number) of the motor 63, and based on this table, , it is sufficient to drive the imaging position changing unit 60 by the driving direction and driving amount corresponding to the driving of the irradiation position changing unit 50.

In addition, in the configuration of FIG. 11, the projection position and the imaging position are changed by rotating the projection unit 10 and the imaging unit 20 about the

support shafts

51 and 61 parallel to the Y axis. The changing method is not limited to this. For example, the projection unit 10 and the imaging unit 20 may be moved in the X-axis direction or the Y-axis direction, or the subject 40 may be moved in the X-axis direction or the Y-axis direction. In this case, for example, an XY stage 70 is used as a mounting table on which the subject 40 is mounted. In addition, the configuration may be such that the subject 40 is tilted from a state parallel to the XY plane, or the relative distance between the subject 40 and the projection unit 10 and the imaging unit 20 may be changed.

<Other change examples>
In the above embodiment, as shown in FIG. 6, after first extracting corresponding points corresponding to the transparent member 41 (S103, S104, S109), distances are calculated for the extracted corresponding points. When points are extracted, the distances may be calculated once for all corresponding points, and then the shortest distance among them may be selected as the distance corresponding to the transparent member 41. This point also applies to the processing in FIGS. 7 and 8.

Furthermore, the processes in FIGS. 6 and 7 may be performed in parallel to obtain the distances from the subject 40 to the

members

41 and 42 at the same time. Similarly, in the process of FIG. 8, a plurality of setting ranges may be set and the distances to the

members

41 and 42 may be acquired at the same time.

Furthermore, the number of imaging units 20 does not necessarily have to be one, and a plurality of imaging units 20 that capture images from mutually different directions may be arranged. The emission wavelength of the light source 11 does not necessarily have to be in the infrared wavelength band, but may be in the visible wavelength band, for example.

Further, in the above embodiment, the search range of the stereo corresponding point search is in the row direction, but the search range may be in the column direction, or the search range may be in a combination of rows and columns. good.

Further, the image processing unit 33 may correct the distortion caused by the distortion of the imaging lens 22 on the captured image, and may also correct the distortion caused by the distortion of the imaging lens 22 on the captured image. Distortion may be corrected by distortion or the like.

Further, in the above embodiment, a correlation value is obtained for each pixel block 212, but correlation values between pixel blocks 212 may be further obtained through processing such as parabola fitting, and a search for corresponding points may be performed. .

Further, the configuration of the imaging device 1 is not limited to the configuration shown in the above embodiment, and for example, a photosensor array in which a plurality of photosensors are arranged in a matrix may be used as the imaging element 21. .

In addition, in the above embodiment, the distance to the subject is calculated by searching for stereo corresponding points between the reference image held in advance and the captured image, but the distance to the subject is calculated by searching for stereo corresponding points between the reference image held in advance and the captured image. One of the images may be used as a reference image, and the distance to the subject may be calculated by searching for stereo corresponding points between the other captured image and the reference image. In this case, these two imaging units have the same configuration as the imaging unit 20 in FIG. 1 . The projection unit 10 may be arranged so as to project pattern light onto a range where the imaging fields of these two imaging units overlap. In this case as well, the polarizing filters arranged in each imaging section may be arranged so as to extract light in the same polarization direction as the light emitted from the projection section 10. Also in this case, the number of imaging units is not limited to two, and three or more imaging units that capture images from different directions may be arranged.

Further, the distance measurement method does not necessarily have to be a measurement method using stereo corresponding point search, and may be a measurement method using TOF, for example. In this case, for example, by omitting the pattern generator 13 from the configuration of FIG. 1, a TOF configuration for performing distance measurement using a so-called flash method can be realized. In this configuration, the projection lens 14 may have a diffusion effect such as a concave lens.

In this configuration, the signal processing unit 31 causes the light source 11 to emit pulsed light, and measures the distance to the subject based on the time difference between the timing of this emission and the timing of reception of reflected light from the subject in each pixel of the image sensor 21. . That is, each pixel area becomes a unit area for distance measurement.

In this case, in the subject shown in FIG. 4, both the reflected light reflected from the surface of the transparent member 41 and the reflected light reflected from the member 42 located further therein are incident. The signal processing unit 31 obtains the above-mentioned time differences based on these two reflected lights, and calculates a distance based on each time difference. Then, the signal processing unit 31 selects the shorter one, the longer one, or the one included in a predetermined setting range from the two types of distances obtained for each pixel, and generates a distance image.

With this configuration as well, the distance to the

transparent member

41 or 42 can be appropriately acquired, as in the above embodiment and Modifications 1 and 2.

Note that when using the TOF method, it does not necessarily have to be a flash method, and a method in which a circular beam is scanned two-dimensionally or a method in which a line beam is scanned in the short axis direction may be used. In these cases, the pattern generator 13 and projection lens 14 are omitted from the configuration of FIG. 1, and a configuration for scanning the beam is added.

For example, in the case of a method in which a circular beam is scanned two-dimensionally, the light parallelized by the collimator lens 12 is scanned two-dimensionally by an optical deflector such as a MEMS mirror. In addition, in the case of a method in which the line beam is scanned in the short axis direction, a cylindrical lens is placed after the collimator lens 12 to generate the line beam, and the line beam is scanned in the short axis direction near the focal position of the cylindrical lens. optical deflectors are arranged.

Also in the case of the TOF method, the light source 11 may be a laser light source. In this case, the polarizing filter 15 may be omitted from the projection section 10 and the laser light source may be arranged such that the direction of linearly polarized light from the laser light source is along the direction of the polarizing filter 23 of the imaging section 20.

In addition, the embodiments of the present invention can be appropriately modified in various ways within the scope of the technical idea shown in the claims.

1 Imaging device 10 Projection unit 15 Polarizing filter 20 Imaging unit 21 Imaging element 22 Imaging lens 23 Polarizing filter 31 Signal processing unit 40 Subject 50 Irradiation position changing unit 60 Imaging position changing unit ΔW setting range

Claims

a projection unit that projects light with a substantially uniform polarization direction;
an imaging unit that captures an image of a subject onto which the light is projected;
a signal processing unit that processes the captured image acquired by the imaging unit and measures the distance to the subject;
The imaging unit includes:
an imaging lens;
a polarizing filter that extracts light in the same polarization direction as the light emitted from the projection section;
an imaging element that receives light from the subject via the imaging lens and the polarizing filter;
The signal processing unit measures the distance for each region on the captured image without limiting the distance to be measured to one.
An imaging device characterized by:
The imaging device according to claim 1,
The signal processing unit selects the shortest distance as a measurement result when a plurality of distances are obtained for the area.
An imaging device characterized by:
The imaging device according to claim 1,
The signal processing unit selects the longest distance as a measurement result when a plurality of distances are obtained for the area.
An imaging device characterized by:
The imaging device according to claim 1,
The signal processing unit acquires this distance as a measurement result when a distance included in a preset range for the area is obtained.
An imaging device characterized by:
The imaging device according to any one of claims 1 to 4,
further comprising an irradiation position changing unit that changes the irradiation position of the light on the subject;
An imaging device characterized by:
The imaging device according to any one of claims 1 to 5,
further comprising an imaging position changing unit that changes an imaging position of the imaging unit with respect to the subject;
An imaging device characterized by:
The imaging device according to any one of claims 1 to 6,
The projection unit projects pattern light with a predetermined intensity distribution,
The signal processing unit measures the distance to the subject by comparing a reference image that is a captured image when the subject is at a predefined distance with the captured image captured by the imaging unit.
An imaging device characterized by:
The imaging device according to any one of claims 1 to 7,
The projection unit includes a polarization filter for generating the substantially uniformly polarized light.
An imaging device characterized by: