JP2021144028A - Method for removing multipath and measuring device - Google Patents

Abstract

To provide a method for removing a multipath which can suppress image degradation due to a virtual image derived from the multipath.SOLUTION: The method for removing a multipath according to the present disclosure includes: a background distance estimation unit for estimating a distance of a background in an imaging space; an imaging step of acquiring a luminance image holding information on the amount of reception of reflected light on a pixel-by-pixel basis or acquiring a distance image holding information on a distance to an object on a pixel-by-pixel basis; and a pixel value determination step of updating the value of pixels which capture an object more distant than an estimated background distance estimated in the background distance estimation step in the luminance image or in the distance image. In the background distance estimation unit, a parameter on imaging conditions for a camera is used for calculation.SELECTED DRAWING: Figure 3

Description

The present disclosure relates to a distance measuring method and a distance measuring device for measuring a distance to an object using light.

Patent Document 1 discloses an object detection system in which the background is accurately set in the distance information to suppress the occurrence of false detection.

Patent Document 2 discloses a distance information acquisition device capable of reducing the processing load for detecting multipath.

JP-A-2019-144210 International Publication No. 2019/188348

In the methods of Patent Document 1 and Patent Document 2, when noise is superimposed on the distance image when the background distance is generated by imaging, or when the camera moves, or an object existing in the ranging range at the time of background imaging is present. There is a problem that the accuracy of the distance image deteriorates when the distance image is moved.

The present disclosure provides a multipath removing method and a distance measuring device that suppresses image quality deterioration due to a virtual image derived from multipath.

The multi-pass removal method in one aspect of the present disclosure includes a background distance estimation step for estimating the background distance in the shooting space, and a brightness image holding information on the amount of reflected light received for each pixel or an object for each pixel. A pixel value determination that updates the values of pixels that image an object farther than the estimated background distance estimated in the background distance estimation step in the brightness image or the distance image in the shooting step for acquiring the distance image holding the distance information. The background distance estimation step is calculated using parameters related to the shooting conditions of the camera.

The distance measuring device according to one aspect of the present disclosure is a distance measuring device based on a TOF method, and is a light emitting exposure that controls a light source, an image pickup element having a plurality of pixels, a light emission timing of the light source, and an exposure timing of the image pickup device. The distance between the control unit and the background in the shooting space between the control unit and the signal processing unit that generates a brightness image that holds information about the amount of reflected light received for each pixel or a distance image that holds information about the distance to the object for each pixel. Is provided as an estimated background distance, and a multi-pass removing unit for updating the value of a pixel that images an object farther than the estimated background distance is provided.

The multipath removing method in the present disclosure can suppress image quality deterioration due to a virtual image derived from multipath.

FIG. 1 is a diagram showing a configuration of a distance image generator according to the first embodiment. FIG. 2 is a timing chart for explaining the light emission control of the light source of the distance image generator and the exposure control of the image sensor according to the first embodiment. FIG. 3 is a diagram of a space including a relative position between the image sensor and the target range of the multipath removal process for explaining the background distance estimation method in the process of the multipath removal unit in the first embodiment. FIG. 4 is a flowchart for explaining the processing of the multipath removing unit in the first embodiment. FIG. 5 is a flowchart for explaining a process of matching the estimated background distance included in the process of the multipath removing unit in the first embodiment with the closest measured distance value. FIG. 6 is a graph for explaining the estimated background distance for each pixel of the distance image according to the first embodiment. FIG. 7 is a graph for explaining a value in which the estimated background distance for each pixel of the distance image in the first embodiment is matched with the closest measured distance value. FIG. 8 is a flowchart for explaining an estimated background distance correction method included in the processing of the multipath removing unit in the first embodiment. FIG. 9 is a diagram schematically showing an example of searching for a subrange boundary in a distance image according to the first embodiment. FIG. 10 is a flowchart for explaining a pixel value determination method according to the first embodiment. FIG. 11 is a diagram illustrating that the pixel value determination method in the first embodiment removes an image derived from multipath. FIG. 12 is a diagram showing an example of a distance image output by the distance image generator according to the first embodiment. FIG. 13 is a flowchart for explaining an estimated background distance correction method included in the processing of the multipath removing unit in the second embodiment. FIG. 14 is a flowchart for explaining a division point search method included in the processing of the multipath removing unit in the second embodiment. FIG. 15 is a diagram illustrating a method of updating the estimated background distance based on the density of a point cloud having a distance value within the distance measurement range of the distance image in the second embodiment.

Hereinafter, embodiments will be described in detail with reference to the drawings. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of already well-known matters or duplicate explanations for substantially the same configuration may be omitted. This is to avoid unnecessarily redundant explanations below and to facilitate the understanding of those skilled in the art.

It should be noted that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

(Embodiment 1)
Hereinafter, the first embodiment will be described with reference to FIGS. 1 to 12.

FIG. 1 is a block diagram showing a configuration of a distance image generation device 1 according to an embodiment.

The distance image generation device in the figure is a distance measuring device based on the TOF method, and is a light emission exposure control unit 101, a light source 102, a projection optical system 103, a light receiving optical system 104, an image sensor 105, a setting unit 106, and a signal processing unit 107. , And a multipath removing unit 108.

The light emission exposure control unit 101 generates a light emission control signal having a light emission pulse instructing the light source 102 to emit light and an exposure control signal having an exposure pulse instructing the image pickup device 105 to emit light.

The light source 102 is a semiconductor laser light source, and emits infrared light, for example, according to the emission pulse of the emission control signal. When the distance image generator 1 is used in an urban area or the like, the intensity of the projected light emitted from the light source 102 is set so as to satisfy a predetermined safety standard. The light source 102 may be another light source such as an LED light source or a halogen lamp. The wavelength of the projected light may be other than the wavelength of the infrared light, and the projected light may be generated by mixing light in a plurality of wavelength bands.

The projection optical system 103 converts the projected light emitted from the light source 102 into substantially parallel light and projects it into the ranging range. The projection optical system 103 is composed of one lens. The projection optical system 103 may include a plurality of lenses, concave mirrors, and the like.

When the projected light is projected onto an object existing in the ranging range, the projected light is reflected by the object, and the reflected projected light returns to the distance image generator as reflected light. The light receiving optical system 104 collects the reflected light on the light receiving surface of the image sensor 105.

A filter that transmits the wavelength band of the projected light and blocks the light in other wavelength bands may be arranged between the light receiving optical system 104 and the image pickup element 105. As a result, it is possible to prevent unnecessary light other than the reflected light from entering the image sensor 105.

The image sensor 105 receives the reflected light by a plurality of pixels arranged on the light receiving surface, and outputs a signal corresponding to the received light. An avalanche photodiode is arranged on each pixel. Other photodetector elements may be arranged in each pixel. Each pixel is configured to be able to switch between an exposed state that receives reflected light and a non-exposed state that does not receive reflected light. The image sensor 105 outputs a detection signal based on the reflected light received by each pixel in the exposed state.

The setting unit 106 receives the camera parameter setting and transmits it to the multipath removing unit 108. The setting unit 106 is, for example, an input unit such as a keyboard or a mouse, an input screen displayed on a display or the like, and the like. The user can input the camera parameters measured in advance by operating the setting unit 106. The input camera parameter is transmitted to the multipath removing unit 108 and stored as a set value of the camera parameter. The setting unit 106 may include a camera parameter measuring sensor, automatically measure the camera parameter, and transmit the camera parameter to the multipath removing unit 108. The camera parameters are sufficient for calculating the estimated background distance in the multi-pass removing unit 108, for example, an image (virtual image) derived from the multi-pass of the object is reflected on a flat floor surface, and the virtual image is removed. In this case, it is the horizontal angle of view and the vertical angle of view of the light receiving optical system 104, the installation height of the camera, and the pitch angle of the camera. In this way, the camera parameters indicate the plane or curved surface on which the virtual image derived from multipath is assumed.

The setting unit 106 may also be used to set the ranging range.

The signal processing unit 107 processes the detection signal output from each pixel of the image sensor 105 to generate image data. For example, the signal processing unit 107 generates image data in which the detection signal output from each pixel is associated with the position of each pixel.

The multi-pass removing unit 108 calculates the estimated distance of the background in the information corresponding to each pixel of the image data generated by the signal processing unit 107, determines whether or not the virtual image information is stored, and the virtual image information is stored. If it is determined that it is stored, the value corresponding to that pixel is removed, that is, the value is updated to a value outside the ranging range. The pixel determined to store the virtual image information may be updated so as to have the same value as the estimated background distance.

FIG. 2 is a timing chart for explaining the light emission control of the light source of the distance image generator and the exposure control of the image pickup device. FIG. 2 shows a case where the distance measurement range is divided into n distance ranges for distance measurement. n is an integer of 1 or more. Further, in order to acquire the image data corresponding to each distance range in chronological order, a plurality of consecutive frames are set on the time axis. The time width of each frame is the same. In addition, a section (subrange) corresponding to the number of distance ranges is set in one frame. In the example shown in FIG. 2, n subranges are set for each frame. Here, the time width of each section is the same, but it does not necessarily have to be the same.

The light emission signal of FIG. 2 indicates a drive signal input to the light source 102 in response to an instruction from the light emission exposure control unit 101. When the drive signal is at a high level, the light source 102 is driven and the projected light is emitted from the light source 102. When the drive signal is at low level, the light source 102 is stopped and no projected light is emitted from the light source 102.

From the start of each subrange, the light emission exposure control unit 101 raises the drive signal to a high level by the time width ΔT of the light emission period. When the setting unit 106 accepts the setting of the ranging range, the light emission timing is set based on this setting.

When an object exists in the target distance range, the reflected light from the object is focused on the light receiving surface of the image sensor 105. Here, the reflected light is incident on the pixels of the image sensor 105 on which the image of the object is projected with a delay time corresponding to the distance to the object. That is, the reflected light is received at a light receiving timing that is deviated from the light emitting timing by a time corresponding to the distance to the object. The light emission exposure control unit 101 controls the exposure state of all the pixels as follows so that only the reflected light from the target distance range is received by each pixel.

The exposure signal of FIG. 2 indicates the exposure timing of the pixels controlled in response to an instruction from the emission exposure control unit 101. During the period when this waveform rises to a high level, each pixel of the image sensor 105 is exposed. For example, the time difference from the rising timing of the light emission signal to the rising timing of the exposure signal in the subrange #i (i is an integer from 1 to n) is i × ΔTa. ΔTa = ΔT may be satisfied. For example, when ΔTa = ΔT = 10 nanoseconds, subrange = 1 corresponds to a distance range from 0 m to 1.5 m. Subrange #i corresponds to the distance range from 1.5 × (i-1) to 1.5 × i. When the setting unit 106 accepts the setting of the ranging range, the exposure timing is set based on this setting.

When the light emission period and the exposure period are set as described above, only the reflected light from the set distance range is received by each pixel of the image sensor in each exposure period. Then, the signal processing unit 107 generates image data in each subrange based on the detection signal output from each pixel, synthesizes images of one or more subranges in one frame, and outputs the image as a distance image. .. Images of a plurality of subranges are combined by storing, for example, a distance value corresponding to the subrange that maximizes the light receiving intensity at a position corresponding to the same pixel. The distance image may be generated by another method.

FIG. 3 is a diagram for explaining the space of the world coordinate system including the relative relationship between the image sensor 105 and the target range 201, which is the background included in the ranging range and is the target range of the multipath removal process. be. The target range 201 may include a partial range that is blocked by an object between the image sensor 105 and the image sensor 105 and is not distance-measured. The coordinates (x _c , y _c , z _c ) indicate the camera coordinate system. In the first embodiment, the target range is a part of a horizontal floor surface as an example. The target range may be a flat surface such as a wall or ceiling instead of the floor surface. The distance image generator 1 may move in parallel to the target range 201, here the floor surface, and the camera parameters are not affected by the translation. Therefore, the implementation may be performed on the premise that the distance image generator 1 moves in parallel, for example, by installing it in an automobile traveling on a flat road surface. In the background distance estimation step, the multipath removing unit 108 estimates the coordinates (x, y, z) of the corresponding points 202 in the target range 201 corresponding to each pixel and calculates the estimated background distance d. The calculation of the target range 201 and the calculation of the corresponding point 202 are calculated by a method using a trigonometric function.

FIG. 4 is a flowchart illustrating processing in the multipath removing unit 108. The multi-path removing unit 108 modifies the estimated background distance, the camera parameter setting step S1 for storing the camera parameters received from the setting unit 106, the background distance estimation step S2 for calculating the estimated background distance using the camera parameters, and the background distance estimation step S2. The estimated background distance correction step S3 and the pixel value determination step S4 for determining the pixel value by comparing the estimated background distance and the actually measured distance are executed.

FIG. 5 is a flowchart illustrating the background distance estimation step S2. The processing is performed for each pixel. Although FIG. 5 shows a flowchart of sequential processing for each pixel as an example, a plurality of pixels may be processed in parallel. The corresponding point 202 of the pixel of interest is estimated (S201), and the distance between the corresponding point 202 and the image sensor 105, for example, the distance in the definition of the Euclidean distance is calculated (S202). The distance between the corresponding point 202 and the image sensor 105 may be calculated by another distance definition such as the Manhattan distance. Next, the closest measured distance among the measured distances of all the pixels is updated as the estimated background distance d of the pixel of interest (S203).

In the first embodiment, when step S203 in the method shown in FIG. 5 is not executed, the value that can be taken by the estimated background distance d calculated for each pixel is larger than the value that can be taken by the measured distance, so that the target range 201 There are pixels in which the image of is removed and pixels in which the image is not removed. According to the method shown in FIG. 5, by matching the possible value of the estimated background distance d with the actually measured distance, it is possible to reduce the number of pixels for removing the image in the target range 201. Even when it is preferable to remove the image of the target range 201, it is possible to reduce the number of pixels in which the image of the target range 201 is not removed. Note that S203 does not have to be executed.

FIG. 6 is a graph showing a three-dimensional plot of the position in the image and the estimated background distance d when step S203 is not executed when the target range 201 is the area included in the floor surface. The coordinates (u, v) in the figure indicate the position in the image, that is, the coordinate system of the two-dimensional image of the image sensor 105. In the figure, the set of estimated background distances d constitutes one plane or one curved surface.

FIG. 7 shows a three-dimensional plot of the position in the image and the estimated background distance when step S203 is executed when the target range 201 is the area included in the floor surface. In the figure, the set of estimated background distances d constitutes a plurality of stepped planes.

FIG. 8 is a flowchart illustrating the estimated background distance correction step S3. In the distance image, the pixel corresponding to the boundary of the subrange is searched (S301). In the search for the subrange boundary, for example, when the target range is a part of a horizontal floor surface, the subrange boundary appears horizontally in the image, so edge enhancement is performed using a differential filter and a straight line is extracted using the Hough transform. Later, it is possible to select a horizontal straight line and estimate it as a subrange image. The edge enhancement may be performed by another method such as a Laplacian filter, and the straight line extraction may be performed by a method other than the Hough transform. Further, the subrange boundary may be searched by a method other than linear extraction. The difference between the estimated background distance and the measured distance is calculated for all the pixels corresponding to the searched subrange boundary, the calculated difference is integrated for all the pixels corresponding to the subrange boundary, and the integrated value is used as the evaluation value (S302). .. As long as the evaluation value is equal to or higher than the predetermined threshold value (no in S303), the camera parameter is updated (S304), and the process of executing the background distance estimation step S2 is looped.

Since the distance range corresponding to the subrange has a thickness, the actual distance value of the image in the same subrange image is approximated and saved as the measured distance value. Therefore, an approximation error is likely to occur, but according to the subrange boundary, the influence of the approximation by the thickness is small, so that it can be selected as an evaluation value when correcting the estimated background distance.

FIG. 9 is a diagram schematically showing an example of a distance image as a result of searching the subrange boundary in the subrange boundary search step S31. The distance image 301 is an example in which the subrange boundary is shown as a straight line 203a, a straight line 203b, ..., A straight line 203f. The same hatched parts in the figure indicate the same distance.

FIG. 10 is a flowchart illustrating the pixel value determination step S4. The processing is performed for each pixel (S405, S406). In the pixel of interest, when the estimated background distance is farther than the measured distance (yes in S401), the measured distance is determined as the pixel value (S402), and when the estimated background distance is the same as or close to the measured distance (no in S401). , A value outside the distance measurement range is determined as a pixel value (S403), and the pixel value determined in step S402 or S403 is stored (S404).

FIG. 11 is a diagram illustrating that the pixel value determination step S4 removes the image derived from the multipath. The image derived from the multipath appears when the projected light emitted from the light source 102 is reflected by the object, further reflected by the floor surface, and reaches the image sensor 105 as the reflected light. Therefore, the image derived from the multipath is imaged through a longer optical path than when the reflected light reaches the image pickup device 105 from the object without passing through the floor surface, and the actual measurement distance becomes longer. For example, the measured distance of the end portion 200a of the object is closer than the corresponding point 202a within the target range 201 of the floor surface. On the other hand, the actually measured distance of the end portion 200b of the virtual image derived from the multipath is closer than the corresponding point 202b within the target range 201 of the floor surface.

Therefore, the pixel value determination step S4 removes the image derived from the multipath.

FIG. 12 is an example of a distance image before and after executing the process of the multipath removing unit 108. In the image 302 before multipath removal, a virtual image 204 derived from the same person's multipath is reflected from the foot of the person imaged on the left side of the screen like a mirror image with the floor surface as a mirror. On the other hand, the virtual image is removed from the image 303 after the multipath removal process.

In the present embodiment, the method of removing the virtual image from the distance image has been described, but the multipath is removed from the luminance image in which the image to be removed of the virtual image stores information on the light receiving amount of the image of one or more subranges. You may. When the luminance image is targeted, the measured distance may be a representative value of the distance corresponding to the subrange in which the luminance image is acquired. Further, in S402 in the pixel value determination step S4, information on the actually measured light receiving amount is determined as the pixel value instead of the measured distance, and in S403, the information is not a value outside the ranging range and indicates that light reception was not performed. Can be determined as a pixel value.

As described above, the multi-pass removal method according to the first embodiment includes a background distance estimation step for estimating the background distance in the shooting space, and a brightness image or a brightness image holding information on the amount of reflected light received for each pixel. The value of the image of the imaging step of acquiring the distance image holding the information of the distance to the object for each pixel, and the image of the object farther than the estimated background distance estimated in the background distance estimation step in the brightness image or the distance image. The background distance estimation step is calculated by using parameters related to the shooting conditions of the camera.

For example, in the shooting step, the distance measurement range is divided into subranges having one or more division distances, and the distance values corresponding to each of the subranges are stored for each pixel to acquire the distance image and obtain the background. The distance estimation step searches for pixels corresponding to the subrange boundary between the step of calculating the estimated background distance from the parameters and the subranges adjacent to each other within the one or more subranges, and the pixels corresponding to the subrange boundary. The step of updating the parameters so that the error between the actually measured distance and the estimated background distance becomes small may be repeated one or more times.

For example, in the background distance estimation step, the estimated background distance may be updated to match the closest value among the measured distances of all the pixels.

For example, in the background distance estimation step, it is stored as a set of curves passing through pixels at a boundary where the estimated background distance changes, and in the pixel value determination step, the measured distance and the estimated background are referred to with reference to the set of curves. The perspective of the distance may be determined.

For example, the parameter may indicate a plane or curved surface on which a virtual image derived from multipath is assumed.

For example, the parameter may include one or more selected from the horizontal angle of view of the camera, the vertical angle of view of the camera, the installation height of the camera, and the pitch angle of the camera.

For example, the multipath removal method may include an acquisition step of acquiring the parameters by a measurement sensor.

Further, the distance measuring device according to the first embodiment is a distance measuring device based on the TOF method, and controls a light source, an image pickup element having a plurality of pixels, a light emission timing of the light source, and an exposure timing of the image pickup device. A light source exposure control unit, a signal processing unit that generates a brightness image that holds information about the amount of reflected light received for each pixel, or a distance image that holds information about the distance to the object for each pixel, and a signal processing unit in the shooting space. It is provided with a multi-pass removing unit that estimates the background distance as an estimated background distance and updates the value of a pixel that images an object farther than the estimated background distance.

(Embodiment 2)
In the first embodiment, the estimated background distance is corrected with reference to the subrange boundary in the estimated background distance correction step S3, but in the second embodiment, a point in the image of the object is used under the condition that the target range and the object are in contact with each other. The estimated background distance is corrected by using the difference between the group density and the point group density in the virtual image derived from the multipath. In the flow of the multipath removing unit 108 in the second embodiment, the estimated background distance correction step S3 is replaced with the flow shown in FIG. In the second embodiment, when the target range 201 is a part of a flat floor surface and a virtual image derived from the multipath of a person standing in contact with the target range 201 is reflected in the target range 201. The multipath removal method will be described.

Hereinafter, the second embodiment will be described with reference to FIGS. 13 to 15.

FIG. 13 is a diagram for explaining the flow of the background distance correction step in the second embodiment. The signal (raw data) from the signal processing unit is saved (S305), the pixel value determination step S4 is executed, and the output of the pixel value determination step S4 is used as an intermediate image. Next, a difference image storing the difference between the raw data and the intermediate image for each pixel is generated and saved (S306). When a partial image of a certain area of the difference image is extracted (S307) and the number of pixels for storing non-zero pixel values is equal to or greater than the threshold value (yes in S309), it is assumed that this partial image contains a virtual image. do. In the intermediate image, it is assumed that an image of an object (for example, a person) is shown in the partial image located on the upper side of the partial image adjacent to the partial image assumed to contain the virtual image. Therefore, the partial image that is assumed to include the virtual image and the partial image of the same area adjacent to the upper side are collectively cut out and used as the division point search target image (S301). The division point search step S5 is executed in the division point search target image, and a line (division point) for dividing the image of the object (for example, a person) and the virtual image of the object is specified and saved. Next, an evaluation function is set in which a large number of pixel values below the division point are removed in the image to be searched for the division point, and the value becomes larger as the pixel values above the division point are less likely to be removed (S312). Further, the background distance estimation step S2 is executed while changing the camera parameters until the evaluation function exceeds the threshold value (yes in S313), and the estimated background distance correction step S3 is completed.

FIG. 14 is a diagram illustrating the flow of the division point search step S5. In the figure, in the division point search step, the point cloud density is used to search for a point that divides the image of the object and the virtual image derived from the multipath of the object. First, in the figure, the average pixel value of the top row of the division point search target image cut out in step S310 of FIG. 13 is calculated (S501). In step S502 and below, the processing for each line of the image to be searched for the division point is shown.

Unless the row in which the pixel value average is calculated is below the image to be searched for the division point (no in S502), the pixel value average in the lower row of the calculated row is calculated (S504), and the pixel value average in the upper row is calculated. The difference from the average pixel value in the lower row is calculated and saved (S505).

When the line on which the average pixel value is calculated is lower than the image to be searched for the division point (yes in S502), the saved line number having the maximum difference is saved (S503). The saved line number identifies and saves a line (division point) that divides an image of an object (for example, a person) and a virtual image of the object. Focusing on the row of the image to be searched for the division point, the pixel value average is calculated, and the difference from the average of the rows above the row of interest is calculated. The number of the row having the largest difference in the image to be searched for is saved as the division point.

FIG. 15 is a diagram illustrating the background distance correction method and its effect. The lower part of the figure shows an example in which the relative position (relative angle) between the distance image generator 1 and the floor surface fluctuates with time. The upper end of the figure schematically shows an image of an object and a virtual image derived from the multipath of the object.

In this way, in the estimated background distance correction step S3, the relative position of the virtual floor surface with respect to the camera can be corrected by using the image of the object (for example, a person) reflected in the distance image as a clue.

As described above, in the multipath removing method according to the second embodiment, in the background distance estimation step, when the object is in contact with the plane of the background, the step of calculating the background distance from the parameters is executed. After that, among the division point search steps for searching for a point that accurately divides the image of the object and the image derived from the multipath of the object, and the image derived from the multipath with reference to the division point. The step of updating the estimated background distance so that the area remaining without being removed becomes small is repeatedly executed one or more times.

For example, the distance image is generated by using a distance measuring device including a light source and an image pickup element that receives the reflected light reflected by the light projected by the light source on an object existing in the distance measuring range. In the division point search step, the point group density of the distance image may be used to search for a point that divides the image of the object and the image derived from the multipath of the object.

For example, the distance image is a distance image generated by an imaging device that repeats information regarding the presence or absence of an object existing in the distance measurement range one or more times and generates image data in which the information is associated with the position of each pixel. ,
In the division point search step, a point for dividing an image of an object and an image derived from the multipath of the object may be searched based on the information.

The multipath removing method and the imaging device according to one or more aspects have been described above based on the embodiment, but the present disclosure is not limited to this embodiment. As long as the gist of the present disclosure is not deviated, various modifications that can be conceived by those skilled in the art are applied to the present embodiment, and a form constructed by combining components in different embodiments is also within the scope of one or more embodiments. May be included within.

According to the present disclosure, it is possible to provide a multipath removing method that suppresses these effects even when the camera moves or an object existing in the ranging range at the time of background imaging moves. , Useful as an imaging device.

101 Emission exposure control unit 102 Light source 103 Projection optical system 104 Light receiving optical system 105 Image sensor 106 Setting unit 107 Signal processing unit 108 Multipath removal unit 201 Target range 202 Corresponding points 203a, 203b Straight line 204 Virtual image 301 Distance image 302 Before multipath removal image

Claims (11)

A background distance estimation step that estimates the background distance in the shooting space,
A shooting step of acquiring a luminance image that holds information on the amount of reflected light received for each pixel or a distance image that holds information on the distance to an object for each pixel.
In the luminance image or the distance image, a pixel value determination step for updating the value of a pixel that images an object farther than the estimated background distance estimated in the background distance estimation step, and a pixel value determination step.
Have,
The background distance estimation step is calculated using parameters related to the shooting conditions of the camera.
Multipath removal method. In the shooting step
The distance image is acquired by dividing the distance measurement range into subranges having one or more division distances and storing the distance values corresponding to each of the subranges for each pixel.
The background distance estimation step is
The step of calculating the estimated background distance from the parameters and
Search for pixels corresponding to subrange boundaries between subranges adjacent to each other within the one or more subranges, and refer to the pixels corresponding to the subrange boundaries so that the error between the measured distance and the estimated background distance becomes small. Steps to update the parameters and
The multipath removing method according to claim 1, wherein the above is repeatedly executed once or more. In the background distance estimation step
The multipath removing method according to claim 1, wherein the estimated background distance is updated so as to match the closest value among the measured distances of all the pixels. In the background distance estimation step
It is stored as a set of curves passing through the pixels of the boundary where the estimated background distance changes, and is stored.
The pixel value determination step is
The multipath removing method according to claim 3, wherein the perspective determination of the measured distance and the estimated background distance is performed with reference to the set of curves. In the background distance estimation step
When the object touches the plane of the background,
After executing the step of calculating the background distance from the parameters,
A division point search step for searching for a point that accurately divides the image of the object and the image derived from the multipath of the object, and
A step of updating the estimated background distance so that the area remaining without being removed from the image derived from the multipath is reduced by referring to the division point.
The multipath removing method according to claim 1, wherein the above is repeatedly executed once or more. The distance image includes a light source, an image sensor that receives reflected light reflected by the light projected by the light source on an object existing within the distance measuring range, and an image sensor.
It was generated using a distance measuring device equipped with
The multipath removing method according to claim 5, wherein in the division point search step, a point for dividing an image of an object and an image derived from the multipath of the object is searched by using the point cloud density of the distance image. The distance image is
Generate information about the presence or absence of objects in the ranging range one or more times,
It is a distance image generated by an imaging device that generates image data in which the information is associated with the position of each pixel.
The multipath removing method according to claim 5, wherein in the division point search step, a point for dividing an image of an object and an image derived from the multipath of the object is searched based on the information. The multipath removing method according to any one of claims 1 to 7, wherein the parameter indicates a plane or a curved surface on which a virtual image derived from the multipath is assumed. The parameters of claims 1 to 8 include the horizontal angle of view of the camera, the vertical angle of view of the camera, the installation height of the camera, and one or more selected from the pitch angles of the camera. The multi-pass removal method according to any one item. The multipath removing method according to any one of claims 1 to 9, further comprising an acquisition step of acquiring the parameter by a measurement sensor. It is a distance measuring device based on the TOF method.
Light source and
An image sensor with multiple pixels and
A light emission exposure control unit that controls the light emission timing of the light source and the exposure timing of the image pickup apparatus.
A signal processing unit that generates a luminance image that holds information about the amount of reflected light received for each pixel or a distance image that holds information about the distance to the object for each pixel.
A distance measuring device including a multipath removing unit that estimates the distance of the background in the shooting space as an estimated background distance and updates the value of pixels that image an object farther than the estimated background distance.

Images