JP2014130086A - Range image sensor, processor and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a range image sensor capable of precisely obtaining a distance value of a pixel corresponding to a contour in a discontinuous portion of the distance value in a depth map.SOLUTION: The range image sensor comprises: a first image generation unit 10 that generates a depth map; a second image generation unit 20 that generates a high-definition image which has preciseness higher than the depth map; and a processor 30. The processor 30 includes: a boundary extraction unit 33; a pixel extraction unit 34; and a correction unit 35. The boundary extraction unit 33 extracts positions of two boundary pixels interposed by a boundary of an object of interest in a target space from a high-definition image. The pixel extraction unit 34 extracts target pixels including an object boundary and reference pixels which are positioned at the rear side of the target pixel neighboring the target pixels in the depth map based on the position of boundary pixels. The correction unit 35 corrects the distance value of the target pixels using the position of the boundary in a range of the target pixels and a correction value which is preset with respect to the density difference of the position between the target pixel and a reference pixel.

Description

  The present invention relates to a distance image sensor that generates a distance image whose pixel value is a distance value, a processing device used for the distance image sensor, and a program that causes a computer to function as the processing device.

  Conventionally, a distance image sensor that generates a distance image whose pixel value is a distance value is known. As a technique for generating a distance image, a technique that uses the principle of triangulation and a technique that measures the time of flight of light are mainly used. As a technique using the principle of the triangulation method, an active type that performs light projection and reception like a light cutting method and a phase shift method, and a passive type that uses parallax like a stereo image method are known. In addition, as a technique for measuring the flight time, the intensity modulated light whose intensity changes under a predetermined condition is projected, and the flight time measured using the phase relationship of the intensity modulated light at the time of light projection and at the time of light reception is measured as a distance. The technology to convert to is known.

  It is known that this type of distance image sensor generally has a lower number of pixels and lower resolution than a grayscale image or a color image. Therefore, if the boundary of the object is located within the pixel of the distance image, the pixel value (distance value) of the pixel is not an appropriate value, and the position of the object existing in the real space cannot be measured accurately. Occurs.

  In order to solve this type of problem, Patent Document 1 uses a color image having a resolution higher than that of the distance image in addition to the distance image, so that the distance of the contour portion in which an error occurs in the distance image having a low resolution. Techniques for estimating values are described. In Patent Document 1, in order to estimate the distance value (distance information) of the contour portion, the distance value around the pixel of interest is estimated, and the pixel position of the contour information extracted from the color image and the position of the pixel of interest in the distance image A value obtained by interpolating the distance value around the pixel of interest using the relationship is used as the distance value of the pixel of interest. In short, a value obtained by linearly interpolating the distance values of the four corners of the target pixel is used as the distance value of the target pixel.

  Patent Document 1 also describes a technique for calculating the reliability of the distance value for each pixel of the distance image according to the intensity of the reflected light from the object, and correcting the distance value based on the reliability. Yes. Since this technology is affected by ambient light as the amplitude of reflected light is smaller, the reliability is set based on the amplitude of the reflected wave because the reliability of the distance value is lowered. Further, a pixel area of a plurality of pixels centering on the pixel of interest is determined, a weighted average with the reliability as a weight is obtained for this pixel area, and this result is estimated as the distance value of the pixel of interest.

JP 2010-71976 A

  As described above, Patent Document 1 describes a technique for interpolating the distance value of the pixel of interest by using the distance value of the pixel around the pixel of interest, and the distance value on the contour is the surrounding value. It is assumed that it is assumed to have a linear relationship with the distance value.

  However, when the contour is generated due to the difference in distance between the object and the background, the distance value is discontinuous across the contour, so the distance between the surrounding pixels as in the technique described in Patent Document 1 With the technique of interpolating using values, the distance value of the pixel of interest cannot be obtained with high accuracy.

An object of the present invention is to provide a distance image sensor capable of accurately obtaining a distance value of a pixel corresponding to a contour of a part where the distance value is discontinuous in the distance image, and further, the distance image sensor. It is an object of the present invention to provide a processing apparatus used in the above, and a program that causes a computer to function as the processing apparatus.
To do.

  A distance image sensor according to the present invention generates a first image generation device that generates a distance image whose pixel value is a distance value for a target space, and a high-definition image that is higher in definition than the distance image for the target space. A second image generation device; a boundary extraction unit that extracts positions of two boundary pixels that sandwich the boundary of an object of interest in the high-definition image; and a pixel that includes the boundary of the object in the distance image as a target pixel A pixel extraction unit for extracting the target pixel and the reference pixel based on the position of the boundary pixel using a pixel adjacent to the target pixel located on the background side as a reference pixel, and a target pixel obtained from the high-definition image Correction that is set in advance for the relationship between the position of the boundary in the range and the density difference at the position of the target pixel and the reference pixel obtained from the high-definition image With, characterized in that it comprises a correcting section for correcting the pixel value of the target pixel included in the pixel extracting portion is extracted.

  Another distance image sensor according to the present invention includes a first image generation device that generates a distance image whose pixel value is a distance value and a grayscale image that is a density value for the target space, and a height higher than the distance image for the target space. A second image generation device that generates a high-definition image that is fine, a boundary extraction unit that extracts positions of two boundary pixels that sandwich the boundary of the object of interest in the high-definition image, and the object in the distance image A pixel extraction unit that extracts the target pixel and the reference pixel based on the position of the boundary pixel, using the pixel including the boundary as a target pixel, the pixel adjacent to the target pixel and positioned on the background side as a reference pixel, Regarding the relationship between the position of the boundary in the range of the target pixel obtained from the high-definition image and the density difference at the position of the target pixel and the reference pixel obtained from the grayscale image Using the correction value that is Luo beforehand set, characterized by comprising a correction section for correcting the pixel value of the target pixel included in the pixel extracting portion is extracted.

  In the distance image sensor, an area extracting unit that extracts a first area corresponding to the object extracted from the distance image and a second area corresponding to the object extracted from the high-definition image is further provided. The pixel extraction unit preferably associates the position of the boundary pixel with the position of the target pixel by performing pattern matching between the first region and the second region extracted by the region extraction unit. .

  In the distance image sensor, the pixel extraction unit associates the position of the boundary pixel with the position of the target pixel by using a parallax with respect to the object of the first image generation device and the second image generation device. It is preferable.

  In this distance image sensor, the first image generating device and the second image generating device share an optical axis with respect to the target space by branching the incident light beam from the target space into a plurality of paths. Preferably further elements are provided.

  In this distance image sensor, it is preferable that the image pickup elements for picking up images of the first image generation device and the second image generation device are formed on a single substrate so as to form a single chip.

  The distance image sensor further includes an evaluation unit that compares the distance image generated by the first image generation device with the distance image corrected by the correction unit and evaluates the validity of the correction value. The area is an area excluding the background in the distance image, substantially forming a connected area with the area to which the correction value is applied, and a distance value obtained from a pixel to which the correction value is not applied, It is preferable to determine that the correction value is appropriate when the difference from the distance value of the pixel to which the correction value is applied is equal to or less than a predetermined value.

  In the distance image sensor, the boundary extraction unit extracts pixels corresponding to the object in the target space from the high-definition image in an area including pixels in which the pixel value satisfies a predetermined extraction condition in the distance image. Is preferred.

  The distance image sensor further includes a background storage unit that stores the distance image obtained in a state where the object does not exist in the target space as a background distance image, and the boundary extraction unit includes the first image generation device. The extraction condition is that the pixel value is equal to or greater than a reference value in the difference image between the distance image generated by the image and the background distance image stored in the background storage unit, and the object in the target space from the high-definition image It is preferable to extract pixels corresponding to.

  A processing device according to the present invention generates a first image generation device that generates a distance image having a pixel value as a distance value for a target space, and a first image generation device that generates a high-definition image that is higher in definition than the distance image for the target space. 2 is a processing device for obtaining a distance to an object of interest in the target space and extracting the positions of two boundary pixels sandwiching the boundary of the object of interest in the high-definition image. A boundary extraction unit that performs detection, a pixel that includes the boundary of the object in the distance image as a target pixel, a pixel that is adjacent to the target pixel and is positioned on the background side as a reference pixel, and the target pixel and the reference pixel are the boundary pixel A pixel extraction unit that extracts based on the position of the target pixel, a boundary position in a range of the target pixel obtained from the high-definition image, and the target pixel obtained from the high-definition image A correction unit that corrects the pixel value of the target pixel extracted by the pixel extraction unit using a correction value set in advance with respect to the relationship with the density difference at the position with respect to the reference pixel. And

  Another processing device according to the present invention includes a first image generation device that generates a distance image whose pixel value is a distance value and a grayscale image that is a density value for the target space, and a higher definition than the distance image for the target space. A processing device that is used in conjunction with a second image generation device that generates a high-definition image, and obtains the distance to the object of interest in the target space, and sandwiches the boundary of the object of interest in the high-definition image A boundary extraction unit that extracts positions of two boundary pixels; and a pixel that includes the boundary of the object in the distance image as a target pixel, and a pixel that is adjacent to the target pixel and is positioned on the background side as a reference pixel. A pixel extraction unit that extracts a pixel and the reference pixel based on a position of the boundary pixel; a boundary position in a range of a target pixel obtained from the high-definition image; and the gray image Correction for correcting the pixel value of the target pixel extracted by the pixel extraction unit using a correction value set in advance for the relationship between the density difference at the position of the target pixel and the reference pixel obtained from And a section.

  The program according to the present invention causes a computer to function as the processing device described above.

  According to the configuration of the present invention, when a boundary is generated due to a distance difference between the object and the background, the boundary is extracted using a high-definition image having a higher definition than the distance image, and the first image generation apparatus By using the correction value set for the relationship between the density difference between the target pixel including the boundary in the grayscale image generated together with the distance image and the reference pixel that is the background side pixel adjacent to the target pixel, Since the distance value of the pixel corresponding to the boundary where the distance value becomes discontinuous is corrected, there is an advantage that the distance value of the part where the distance value is discontinuous can be obtained with high accuracy.

It is a block diagram which shows embodiment. It is a figure which shows the operation example of the 1st image generation apparatus used for the same as the above. It is a figure which shows the operation example of the 1st image generation apparatus used for the same as the above. It is a figure which shows the operation example of a 2nd image generation apparatus in the same as the above. It is a figure which shows the example of the correction value used for the same as the above. It is a schematic block diagram which shows the other examples of the optical system in the same as the above. It is a block diagram which shows the other structural example same as the above. It is a figure which shows the operation example in the case of correcting a correction value in the same as the above. It is a block diagram which shows another structural example in the same as the above.

  The distance image sensor described below includes a first image generation device that generates a distance image whose pixel value is a distance value and a grayscale image that is a density value, and a first image generation device that generates a high-definition image that is higher in definition than the distance image. 2 image generation devices.

  The first image generating apparatus is configured to obtain a distance value and a density value for each pixel, and generates a distance image and a grayscale image having the same pixel pitch. The high-definition image generated by the second image generation device is a grayscale image or a color image, and the distance image generated by the first image generation device is about 100 × 100 pixels, whereas it is 640 × 480 pixels. The resolution is selected from a range of about 1280 × 1024 pixels.

  Therefore, if the visual fields of the first image generation device and the second image generation device are set to be approximately equal, the high-definition image is several times to ten times the distance image in the horizontal direction and the vertical direction of the image. It has several times the resolution. The first image generation device and the second image generation device each include an independent image sensor.

  The imaging element has pixels arranged on lattice points of a two-dimensional lattice that maps the target space, and receives light from the target space through a light receiving optical system including a lens. Therefore, the direction of looking into the target space through the light receiving optical system is associated with the pixels of the image sensor. That is, in the images generated by the first image generation device and the second image generation device, the pixel value at the position (x, y) is the direction in which the target space is viewed from the pixel at the position (x, y) through the light receiving optical system. Contains information on (θ, φ).

  Here, x represents the position of the pixel in the horizontal direction of the image, and y represents the position of the pixel in the vertical direction of the image. In addition, θ represents an angle formed by the direction viewed from the pixel at the position (x, y) through the light receiving optical system with respect to the optical axis of the light receiving optical system, and φ represents an angle formed with respect to the x axis in the xy plane. Now, in the distance image, if the pixel value (distance value) of the pixel at the position (x, y) is d, x = d · sin θ · cos φ and y = d · sin θ · sin φ.

  The first image generation device generates a distance image and a grayscale image using light received from the target space by the image sensor. Technologies widely used for extracting distance information using an image sensor include a technology that uses the principle of triangulation and a technology that measures the time of flight of light.

  When a distance image is generated by measuring the time of flight of light, a light emission source that projects light into the target space is essential and an active type configuration is obtained. On the other hand, as a technique for generating a distance image using the principle of triangulation, an active configuration using a light source and a passive configuration using a stereo vision technique are known. The first image generation apparatus described in the present embodiment may have any configuration, but a configuration in which a distance image is generated by measuring the time of flight of light is assumed below.

  When employing a technique for measuring distance using the time of flight of light, the image sensor that constitutes the first image generation device includes natural light or illumination in addition to the signal light projected from the light source to the target space. Ambient light such as light is incident. Therefore, the image sensor used in the first image generation device needs a larger dynamic range than the image sensor used in the second image generation device in order to clearly distinguish the signal light and the ambient light. That is, the area of the pixel of the image sensor used for the first image generation device needs to be larger than the area of the pixel of the image sensor used for the second image generation device.

  As described above, the pixel area of the first image generating device is larger than the pixel area of the second image generating device. Therefore, when the sizes of the image sensors used for the first image generation apparatus and the second image generation apparatus are approximately the same, the resolution of the image sensor used for the first image generation apparatus is the same as that of the second image generation apparatus. The resolution is lower than the resolution of the image sensor used. In other words, the distance image and the grayscale image generated by the first image generation device have a lower resolution than the high-definition image generated by the second image generation device. For example, when the distance image and the grayscale image generated by the first image generation device are about 10,000 pixels, the second image generation is performed even if an image sensor having the same size is used for the second image generation device. A high-definition image generated by the apparatus is several hundred thousand pixels or more.

  Hereinafter, the configuration of the present embodiment will be specifically described with reference to the drawings. As shown in FIG. 1, the first image generation apparatus 10 includes a light emitting source 11 that projects light into a target space, and an image sensor 12 that receives light from the target space. The light source 11 is used with a light projecting optical system, and the image sensor 12 is used with a light receiving optical system. The light projecting optical system and the light receiving optical system may be combined.

The light emission source 11 is driven so that the light emission amount changes with time by the modulation signal output from the light emission control unit 13. As a modulation signal, a signal having periodicity such as a sine wave, a triangular wave, a sawtooth wave, or a rectangular wave is used, and a rectangular wave signal whose ON period and OFF period change at an integral multiple of a unit period is used. Is also possible. In the following, in order to simplify the explanation, the modulation signal is assumed to be a sine wave or rectangular wave having a constant period (about 10 ⁻⁷ seconds). The light emitting source 11 includes a light emitting diode or a laser diode and is driven by a modulation signal to project intensity modulated light whose intensity changes sinusoidally as signal light in a target space. The light projected from the light source 11 may be in the visible light region, but usually the infrared region is employed.

  Now, the modulation signal is a sine wave, one cycle of the sine wave is divided into four sections at a phase of 90 degrees, and the received light amount received by each pixel of the image sensor 12 for each section is A0, A1, and A2. , A3. That is, for the amount of light received by any pixel of the image sensor 12, the amount of received light in the section where the phase of the modulation signal is 0 to 90 degrees is A0, and the amount of light received in the section of 90 to 180 degrees is A1, 180 to 270 degrees. The amount of light received in the section of A2 is A2, and the amount of light received in the section of 270 to 360 degrees is A3. It is not essential to divide the phase of the modulation signal into sections of 90 degrees. For example, sections for obtaining the received light amounts A0, A1, A2, A3 are appropriately determined, for example, sections of 90 degrees are provided for each fixed time. It is possible to set.

From the calculation of the trigonometric function, ψ = tan ⁻¹ {(A 2 −A 0) / (A 3) -A1)}. If the phase difference ψ is known, the flight time from signal light projection to light reception is determined from the frequency of the modulation signal and the phase difference ψ, and the distance the light has traveled is determined from the flight time and the speed of light. Note that the delay time from when the light source 11 is driven by the modulation signal until the light source 11 projects the intensity-modulated light onto the target space is ignored.

  As described above, the image sensor 12 needs to obtain the received light amounts A0, A1, A2, and A3 for each section set in synchronization with the modulation signal. Therefore, the image sensor 12 receives a demodulated signal synchronized with the modulation signal from the light emission controller 13 and controls the light receiving period. When a CCD image sensor or a CMOS image sensor having a general configuration used as a TV camera is used for the image pickup device 12, it is possible to receive a demodulated signal and adjust the light receiving period by the same operation as the electronic shutter.

  Further, the image pickup device 12 may include an electrode arranged exclusively for adjusting the light receiving period. For example, the imaging device 12 may be configured to enable an operation of holding a charge corresponding to the amount of light received in at least one other section during a period in which light is received from the target space in one section. . Now, the imaging device 12 is driven so as to have a period in which charges corresponding to the received light quantity A0 and the received light quantity A2 are alternately held and a period in which charges corresponding to the received light quantity A1 and the received light quantity A3 are alternately held. Assuming that

  This configuration makes it possible to read out charges for two sections with only one operation of reading out charges from the image sensor 12. In addition, since the image pickup device 12 having this configuration alternately holds charges corresponding to the received light amounts in the two sections, it is possible to accumulate the charges in each section over a plurality of periods of the modulation signal, and a relatively large light receiving output. Can be taken out. This configuration is an example, and the mode for driving the image sensor 12 such as an operation for reading out charges for one section and an operation for accumulating and reading out charges for four sections is appropriately selected.

  The first image generation apparatus 10 includes a calculation unit 14 that performs the above-described calculation using the light reception output read from the image sensor 12. The calculation unit 14 performs the above-described calculation using charges corresponding to the received light amounts A0, A1, A2, and A3, and the desired values such as the phase difference ψ, the flight time from light projection to light reception, and the distance to the object in the target space. Find the value of. As described above, in the first image generation device 10, the calculation unit 14 calculates the distance to the object in the target space for each pixel of the image sensor 12, and generates a distance image with the pixel value as the distance value. The specific configuration of the first image generating apparatus 10 is well known and will not be described in detail.

  The first image generation device 10 also has a function of obtaining an average value of the received amounts A0, A1, A2, and A3 of intensity-modulated light for each pixel. That is, the arithmetic unit 14 calculates a density value for each pixel by removing the influence of the intensity of the projected light by averaging the received light output read from the image sensor 12, and the gray value is set as the pixel value. A grayscale image is generated.

  Note that the light emission control unit 13 may perform an operation of driving the light source 11 by a modulation signal and an operation of not driving the light source 11. In this case, the calculation unit 14 uses the density value corresponding to the component of only the ambient light required during the period when the light source 11 is not driven, and the ambient light included in the density value during the period when the light source 11 is driven. An operation for removing the component is performed. By this operation, the influence of ambient light is reduced, and the pixel value output from the calculation unit 14 corresponds to the reflection component of the signal light.

  As described above, the first image generation device 10 generates a distance image and a grayscale image, and the pixels of the first image generation device 10 have a distance value and a density value, respectively. That is, the distance value and the density value are obtained for each region corresponding to the pixel of the first image generation device 10 in the target space.

  The second image generation apparatus 20 includes an imaging element 21 using a CCD image sensor or a CMOS image sensor, and images a target space through a light receiving optical system (not shown). The second image generation device 20 also converts an output of the image sensor 21 into a digital signal (not shown) to generate a high-definition image (grayscale image) to be given to the processing device 30 described later. Prepared). The image sensor 12 of the first image generation apparatus 10 and the image sensor 21 of the second image generation apparatus 20 may be separate devices, but are formed on a single substrate and configured as a one-chip device. It is desirable. This is because when the image pickup element 12 and the image pickup element 21 are one-chip devices, the positional relationship between the image pickup element 12 and the image pickup element 21 is determined at the time of manufacture, and the positional accuracy of both is guaranteed.

  The distance image and the grayscale image generated by the calculation unit 14 of the first image generation device 10 and the high-definition image output from the second image generation device 20 are input to the processing device 30. The processing apparatus 30 includes a device that operates according to a program and a device that constitutes an interface unit as main hardware elements. An example of a device that operates according to a program includes a configuration in which a microprocessor and a memory, which are individual devices, are combined, and a microcomputer that includes a processor and a memory on one chip. In addition, a DSP (Digital Signal Processor) or an FPGA (Field-Programmable Gate Array) may be used for this type of device.

  By the way, since the distance image and the grayscale image generated by the first image generation device 10 divide the target space in units of pixels, if the boundary of an object existing in the target space is included in one pixel, The light from the object and the light from the background enter simultaneously. In a grayscale image, when light from an object and light from the background are simultaneously incident on one pixel, the pixel value of the pixel has a density corresponding to the average of the amounts of light received by both pixels.

  When the first image generation device 10 has an active configuration, when light from a stationary object and light from a background are simultaneously incident on one pixel, the pixel value of the pixel is intermediate between the object and the background. It becomes a distance value. Further, when the first image generation apparatus 10 has a passive configuration using the stereo vision technology, the pixel value of the pixel becomes indefinite or is determined depending on the processing of the calculation unit 14.

  Note that if the object moves while an image of one frame is obtained, the pixel value of the pixel that has passed through the boundary of the object in the distance image may become an abnormal value, but here it is substantially stationary. Assume that a distance image of an object is generated.

  In the present embodiment, since the first image generation apparatus 10 employs a technique for obtaining the distance value to the object by measuring the time of flight, the pixel value of the pixel including the boundary of the stationary object and the background is determined by the object. The distance between the background and the background. Further, since the distance value is calculated using the relationship between the amounts of received light, the first image generation apparatus 10 has a light amount of intensity-modulated light (signal light) received by the image sensor 12 with respect to the amount of ambient light. It is considered that the pixel value approaches the actual distance value of the object when the number is relatively large.

  Therefore, assuming an event that may occur at the time of distance image generation, change the reflectance of the background in three steps to change the amount of light from the background, and change the width of the object whose distance is measured in four steps. Experiments were performed to generate a distance image and a grayscale image. The four objects whose distances were measured were arranged at equidistant positions from the light receiving optical system, and were selected so as to occupy a range of 2 to 5 pixels in the distance image. Specifically, the width of each object was four types of 7.5 mm, 10.5 mm, 14.0 mm, and 17.5 mm, and the distance to the object was 2 m. The positions of three types of objects having a width of 7.5 mm, 10.5 mm, and 14.0 mm were adjusted so that the centers of adjacent pixels coincided with the centers of the objects in the horizontal direction of the distance image. Further, the position of an object having a width of 17.5 mm was adjusted so that the center of one pixel coincides with the center of the object in the horizontal direction of the distance image. The background reflectance was selected from 60%, 20%, and 5%.

  In FIG. 2 and FIG. 3, there are four regions where the values change relatively large. These areas indicate measurement results for the above-mentioned width object in order from the right. FIG. 2 shows the distance value output from the first image generation apparatus 10, and FIG. 3 shows the density value output from the first image generation apparatus 10. Further, FIG. 4 shows pixel values of a high-definition image output from the second image generation device 20. In each figure, the characteristic (1) indicates that the background reflectance is 60%, the characteristic (2) indicates that the background reflectance is 20%, and the characteristic (3) indicates that the background reflectance is 5%. ing. As is clear from the above description, the distance image shown in FIG. 2 and the grayscale image shown in FIG. 3 have the same resolution, and the high-definition image shown in FIG. 4 has a higher resolution than the images shown in FIGS. is there. In the illustrated example, the high-definition image includes approximately four times as many pixels as the distance image and the grayscale image in the horizontal direction of the image.

  As can be seen from FIG. 3, when the reflectance of the background is high, the density difference between the object and the background is less likely to appear in the grayscale image. However, when the reflectance of the background is relatively low, the density difference between the object and the background is small. Appear clearly. On the other hand, as can be seen from FIG. 2, the distance value near the boundary of the object is influenced by the reflectance of the background, and when the reflectance increases, the distance value tends to become larger than the actual value. For this reason, if the number of pixels occupied by the object in the distance image is small and the reflectance of the background is high, the object is erroneously recognized as being located farther than the actual distance.

  From the above, if light from the object and light from the background are mixed in one pixel in the distance image, an error occurs in the distance value, and the distance increases as the light quantity from the background increases relative to the light quantity from the object. It can be said that the error of the pixel value of the image becomes large. Here, light from the object and light from the background are mixed into one pixel mainly in the vicinity of the boundary of the object. Therefore, it is considered that the pixel value of the distance image can be brought close to the true value by extracting the pixel corresponding to the boundary of the object in the distance image and appropriately correcting the pixel value of the pixel.

  The processing device 30 has a function of obtaining a pixel value close to a true value by extracting a pixel corresponding to the boundary of the object of interest in the target space from the distance image and appropriately correcting the pixel value of the pixel. Therefore, the processing device 30 uses information obtained from the high-definition image generated by the second image generation device 20 in addition to the distance image generated by the first image generation device 10.

  As shown in FIG. 1, the processing device 30 is connected to an interface unit (hereinafter referred to as “I / F unit”) 31 to which the first image generation device 10 is connected and the second image generation device 20. And an I / F unit 32.

  The processing device 30 uses the information obtained from the high-definition image output from the second image generation device 20 to extract pixels including the object boundary in the distance image, so that the two boundaries sandwiching the object boundary A boundary extraction unit 33 that extracts pixels from the grayscale image is provided. The boundary extraction unit 33 extracts, from the high-definition image, two pixels sandwiching the boundary as boundary pixels in the direction intersecting the boundary of the object. When the pixel value of the high-definition image is a density value, the boundary extraction unit 33 extracts boundary pixels by binarization or differentiation of the grayscale image. When the high-definition image is a color image, boundary pixels can be extracted by color.

  The boundary of the object is assumed to be a contour line representing the outer shape of the object. The direction that intersects the boundary of the object is preferably an orthogonal direction. Since the two boundary pixels sandwich the boundary, the pixel value of one boundary pixel corresponds to the amount of light from the object, and the pixel value of the other boundary pixel corresponds to the amount of light from the background. Information regarding the position of the boundary pixel extracted by the boundary extraction unit 33 is transferred to the pixel extraction unit 34. Based on the position of the boundary pixel, the pixel extraction unit 34 extracts a pixel including the boundary of the object in the distance image as a target pixel, and extracts a pixel adjacent to the target pixel and positioned on the background side as a reference pixel. A plurality of boundary pixels are extracted along the boundary line of the object. Therefore, a plurality of target pixels and reference pixels are extracted in the same manner as the boundary pixels.

  The processing device 30 includes a correction unit 35 that corrects the pixel value of the target pixel. The correction unit 35 uses the correction value set according to the relationship between the pixel values (density values) in the grayscale image generated by the first image generation device 10 as the reference pixel value (distance value) and the target pixel. The pixel value of the target pixel is corrected by applying to the relationship with the pixel value (distance value). The correction calculation by the correction unit 35 is performed using the correction value set in the relationship shown in FIG.

  In the illustrated example, the pixel value (distance value) of the distance image output from the first image generation device 10 is D1 (= Db-d1), and the true value of the distance value is D0 (= Db-d0). Assuming that the correction value is α, the relationship d0 = α · d1 is obtained. Here, Db is the distance to the background (such as a wall). Further, d0 is the distance from the wall to the object when the object exists at the distance D0, and d1 is the distance from the wall to the object when it is assumed that the object exists in the pixel value D1 of the distance image. The reference pixel described above corresponds to a pixel having a pixel value Db, and the target pixel corresponds to a pixel having a pixel value D1.

  The horizontal axis in FIG. 5 represents the density difference between the reference pixel and the target pixel, and the vertical axis represents the correction value α. In other words, FIG. 5 shows the density of the pixel at the position corresponding to the target pixel including the boundary of the object and the reference pixel which does not include the boundary of the object and is the background in the grayscale image generated by the first image generation apparatus 10. The relationship between the correction value α and the difference is shown. Since the pixel pitch between the target pixel and the reference pixel is constant, the horizontal axis in FIG. 5 may be paraphrased as a density gradient at the position between the target pixel and the reference pixel.

  The characteristics (A) to (D) in FIG. 5 correspond to the difference in the horizontal width of the region occupied by the object in the grayscale image. Characteristic (A) corresponds to the minimum width (7.5 mm) and characteristic (D) corresponds to the maximum width (17.5 mm).

  As described above, each object is arranged so as to be symmetric with respect to the pixel in the horizontal direction of the distance image. Therefore, in the example shown in FIG. 3, the parts corresponding to the objects are symmetrical with respect to the objects of all widths. Here, if the object is moved in the horizontal direction of the distance image, the position of the boundary of the object changes in the range of the target pixel including the boundary of the object in the process of movement, and the background and the object in the target pixel are changed. The proportion of occupies changes. Therefore, the characteristic shown in FIG. 3 is asymmetrical, and the density value at the position of the target pixel changes with a change in the ratio of the background to the object in the target pixel. From this, the width of the object corresponding to the characteristics (A) to (D) reflects the ratio of the light from the background and the light from the object with respect to the light incident on the target pixel including the boundary of the object in the distance image. I can say that.

  As can be seen from FIG. 5, the correction value α increases as the proportion of the object in the target pixel decreases (as the amount of light from the background increases). Further, according to FIG. 5, it can be seen that the correction value α increases as the density difference between the target pixel and the reference pixel is smaller (the density gradient is smaller). With this knowledge, the correction value α is expressed as α = f (P, a) using the boundary position P within the range of the target pixel and the density difference a between the target pixel and the reference pixel. I can say. A large density difference a means that the reflectance of the background is relatively low. Therefore, in the distance image shown in FIG. 2, when the reflectance of the background is low, the pixel value approaches the true value. Matches.

  In order to determine the correction value α based on the relationship shown in FIG. 5, in addition to the density difference “a” between the target pixel and the reference pixel, it is necessary to obtain the boundary position P within the range of the target pixel. As described above, the pixel extraction unit 34 extracts the target pixel and the reference pixel in the distance image, and the positional relationship between the pixel of the distance image and the pixel of the high-definition image is known. Using this, the pixel extraction unit 34 obtains the position P of the boundary of the object in the target pixel based on the positional relationship between the plurality of pixels of the high-definition image whose field of view overlaps with the target pixel and the boundary pixel. In addition, the pixel extraction unit 34 obtains the density difference a between the target pixel and the reference pixel from the grayscale image output from the first image generation device 10.

  The pixel extraction unit 34 gives the obtained position P and density difference a to the correction unit 35, and the correction unit 35 corrects based on the relationship between the correction value α, the position P, and the density difference a shown in FIG. Determine the value α. Further, the correction unit 35 corrects the distance value using the obtained correction value α, and obtains the true value of the distance value in the target pixel.

  In this embodiment, the pixel value correction calculation is performed using the relationship d0 = α · d1, but can be performed using other relationships, for example, d0 = α1 · d1 + α2 ( α1 and α2 may be in the form of correction values. Alternatively, it is possible to adopt a format of D1 = α3 · D1 (α3 is a correction value).

  As described above, the pixel extraction unit 34 uses the distance image and the grayscale image output from the first image generation device 10 and the high-definition image output from the second image generation device 20 to use the target pixel and the reference image. The density difference “a” of the position with respect to the pixel and the position P of the boundary of the object in the target pixel are obtained. Furthermore, the correction unit 35 determines the correction value α from the relationship as shown in FIG. 5 and corrects the pixel value (distance value) of the target pixel by applying the correction value α, so that the true value of the object of interest is obtained. It is possible to output a distance value close to.

  By the way, the boundary pixel is extracted from the high-definition image, and the target pixel and the reference pixel are extracted from the distance image based on the position of the boundary pixel. Therefore, it is necessary to associate pixel positions of the high-definition image and the distance image. This association is performed by the pixel extraction unit 34. The pixel extraction unit 34 associates the pixel positions of the high-definition image and the distance image by using techniques described below alone or in combination.

  The first technique is based on the assumption that information on the boundary of the object of interest in the target space is common to the high-definition image and the distance image. The processing device 30 includes a region extraction unit 36 for extracting the position of the object from the high-definition image and the distance image.

  The region extraction unit 36 extracts a first region corresponding to the object extracted from the distance image and a second region corresponding to the object extracted from the high-definition image. For extraction of the first region and the second region, either binarization processing or differentiation processing is used. The binarization process means that the pixel value is binarized with an appropriate threshold, and the differentiation process uses a pixel adjacent to the pixel of interest (for example, 8 neighboring pixels) to increase the gradient of the pixel value. This means the calculation processing to be obtained. The differentiation process may be an edge enhancement process using a Sobel filter or the like.

  The pixel extraction unit 34 performs pattern matching on the shape of the boundary between the first region and the second region, using the boundary (contour line, partition line, etc.) of the object extracted by the region extraction unit 36. That is, the degree of coincidence between the first area and the second area is evaluated, and the first area and the second area are associated with the coordinate values so that the degree of coincidence is maximized. For the evaluation of the degree of coincidence between the first region and the second region, the distance between the pixel on the boundary of the first region and the pixel on the boundary of the second region is used. The pixel extraction unit 34 calculates the square root of the sum of squares of the difference in the horizontal direction and the difference in the vertical direction for the pixels on the respective boundaries between the first region and the second region. It is calculated as the distance. Furthermore, the pixel extraction unit 34 associates the coordinate values of the first region and the second region so that the obtained distance is minimized.

  In order to associate the pixel positions of the high-definition image and the distance image, the first technique uses pattern matching, but the second technique uses the first image generation apparatus 10 and the second image generation apparatus 20. Pixels are associated using the parallax between This technique is applied when there is another object in the vicinity of the object of interest, which is guaranteed to be the same distance as the object. Such an object corresponds to a case where the object of interest is a human finger and the other object is a palm, as will be described later.

  For example, assume that the same operation as a gesture operation in which operation content is associated with the movement of a finger touching the touch panel is associated with the time change of the finger position in the target space instead of the time change of the finger contact position with respect to the touch panel. To do. In this case, it is necessary to detect the position of the finger from the distance image. However, since the finger occupies a relatively small number of pixels in the distance image in the width direction, if the distance from the first image generation device 10 to the finger is relatively large, the pixel value of the region corresponding to the finger in the distance image Can cause large errors. That is, it is considered that the pixel value needs to be corrected in the area corresponding to the finger in the distance image. On the other hand, the finger does not exist alone in the target space, but moves integrally with an area having a relatively large number of pixels in the distance image, such as a palm.

  When the second technique described above is employed, this is used to obtain the parallax between the grayscale image and the distance image for the area recognized as a palm from the distance image, and this parallax is recognized as a finger in the distance image. Applies to the area That is, the pixel extraction unit 34 considers that the parallax of the part corresponding to the palm is equal to the parallax of the part corresponding to the finger in the grayscale image and the distance image, and applies the parallax obtained for the palm to the finger. To do.

  Here, when the pixel extracting unit 34 recognizes the palm, for example, a block including 3 × 3 pixels or more except for the background is set in the distance image, and the variation of the pixel value in the block is within 1%. The block is considered to correspond to the palm. When such a block is obtained, the pixels that substantially form the connected area with the block are considered to have the same parallax, and the pixel extraction unit 34 determines the pixel values of the pixels included in the connected area in the distance image. to correct. Substantial connected regions are binarized to remove pixel groups in the range where the difference in pixel value from the pixels in the block in the distance image is equal to or less than a predetermined value, and pixels that are the background in the distance image. This is one of the pixel groups in the connected region that is left as a background other than the background.

  If the parallax is known as described above, it is possible to accurately perform the pixel alignment of the distance image and the high-definition image, and the position of the object in the distance image and the high-definition image without performing pattern matching. Can be associated with each other. That is, there is a possibility that the amount of calculation can be reduced as compared with the case of performing pattern matching.

  The second technique requires calculation of parallax, but if the field of view of the distance image and the high-definition image match, calculation of parallax is not necessary. Therefore, as shown in FIG. 6, it is desirable to add an optical element 40 that branches the incident light flux from the target space into a plurality of paths. A typical example of this type of optical element 40 is a beam splitter using a half mirror, but a dichroic mirror may be used to branch the path depending on the wavelength.

  The optical element 40 is arranged so that the first image generation device 10 and the second image generation device 20 share the optical axis with respect to the target space. That is, since the first image generation device 10 and the second image generation device 20 expect the same field of view in the target space via the optical element 40, it is not necessary to associate pixels between the distance image and the grayscale image. Become.

  Incidentally, in the above-described operation example, the correction unit 35 determines the correction value α using the position P of the boundary of the object in the range of the target pixel and the density difference a between the position of the target pixel and the reference pixel. Therefore, the validity of the determined correction value α has not been verified. Of course, since the correction unit 35 determines the correction value α based on the relationship shown in FIG. 5, the error in the pixel value of the distance image is improved. The value α needs to be verified.

  Therefore, as shown in FIG. 7, the distance image output from the calculation unit 14 of the first image generation device 10 is compared with the distance image corrected by the correction unit 35 of the processing device 30, and the correction value α It is desirable to include an evaluation unit 37 that evaluates the validity of. The evaluation unit 37 determines whether or not there is a pixel that is a region excluding the background in the distance image and that substantially forms a connected region with the region to which the correction value α is applied and to which the correction value α is not applied. If there is a correct pixel, the distance value of the pixel is compared with the distance value to which the correction value α is applied.

  The evaluation unit 37 determines that the correction value α is valid if the comparison result is within the expected error range, and corrects the correction value α so that the correction value α is within the error range if the comparison result exceeds the error range. Here, the substantial connected region is a pixel group in a range in which a difference in pixel value from an adjacent pixel starting from a pixel to which the correction value α is applied in the distance image is equal to or less than a predetermined value, and a pixel that is the background in the distance image After binarization is performed to remove any of the pixels in the connected region that is left as other than the background.

  Note that the pixels to which the correction value α is not applied are, for example, pixels included in a region where variations in pixel values of pixels included in a block of 3 × 3 pixels or more excluding the background are within 1%. As described above, when performing a gesture operation, if the object of interest is a finger, the correction value α is not applied to the palm as an object continuous to the finger, and the correction value α is applied to the finger. It will be. Therefore, the evaluation unit 37 can evaluate the correction value α applied to the finger area, and accurately obtain the distance to the finger using the appropriate correction value α.

  As described above, in the present embodiment, when the number of pixels that the object of interest occupies in the distance image is small in either the horizontal direction or the vertical direction, the correction unit 35 determines the target pixel of the object of interest. Correct the pixel value. Here, when the object of interest is a finger, pixel values are compared between when the target pixel extracted from the distance image is corrected and when it is not corrected, and the result shown in FIG. 8 is obtained. In FIG. 8, the horizontal axis represents distance, and the vertical axis represents the number of pixels in the vertical direction. Here, the vertical pixel regions 40 to 45 and 60 to 65 represent the background (walls and the like). Further, the vertical pixel region of 45-50 represents the arm, the pixel region of 50-55 represents the palm, and the pixel region of 55-60 represents the finger. The pixel area of the arm and the palm is an area where correction is not performed, and correction is performed only for the pixel area corresponding to the finger. That is, if there is no correction, the pixel area corresponding to the fingertip among the vertical 55 to 60 pixel areas is about 15 cm away from the palm pixel area (see characteristic (A)), but after correction, the pixel area is almost the same. A result corresponding to the palm distance is obtained (see characteristic (B)).

  From the above, it is found that a distance error does not substantially occur for the palm and a distance error occurs for the finger, but the distance error is eliminated by applying the correction value α. It was. That is, the effectiveness of this embodiment was verified.

  In the operation example described above, the object of interest is separated from the background using the distance value. However, if the object of interest is separated from the background in advance, it is considered that the amount of calculation thereafter is reduced. In order to separate the object of interest from the background, a technique using an extraction condition using the characteristics of the object of interest and a technique using an extraction condition using the characteristics of the background are used alone or in combination. The processing for separating the object of interest from the background in the distance image is performed by the boundary extraction unit 33 provided in the foremost stage in the processing device 30.

  It is assumed that the object of interest is a hand and the hand is projected forward of the torso. In such a state, the boundary extraction unit 33 may use the distance range from the first image generation device 10 and the distance from the body to the hand as extraction conditions when using the characteristics of the object of interest. For example, when using the extraction condition that the distance range of the object of interest is within 1.5 m and the distance from the torso to the hand is 30 cm or more, and applying the rule that the connected region showing the area specified in the distance image is determined to be the torso The hand region can be extracted from the distance image. If the target object is different, the target object is extracted by changing the extraction condition.

  On the other hand, when an object of interest is extracted using the characteristics of the background, the processing device 30 stores a distance image obtained when no object exists in the target space as a background distance image, as shown in FIG. A background storage unit 38 is provided. The boundary extraction unit 33 has a function of generating a difference image between the distance image output from the first image generation device 10 and the background distance image stored in the background storage unit 38, and the pixel value in the difference image is An object of interest can be separated from the background using an extraction condition that is equal to or greater than a reference value.

  As described above, the boundary extraction unit 33 separates an area including pixels corresponding to the object of interest in the distance image from the background. As a result, the number of target pixels in the subsequent processing is reduced, and the amount of calculation can be reduced.

  It should be noted that when the high-definition image is a grayscale image, the technique that uses the threshold value for the density value can be replaced with a technique that uses color or luminance for the color image. When a color image is adopted as a high-definition image, boundary pixels with respect to an object can be separated by color, so that the target pixel and the reference image in the distance image may be easily determined.

  Further, as described above, in order to obtain the correction value α, the density difference a between the target pixel and the reference pixel is necessary, and the density difference a is output from the first image generation apparatus 10 as a grayscale image. It is possible to ask without. That is, the pixel value of the pixel corresponding to the position of the target pixel and the reference pixel is obtained from the high-definition image output by the second image generation device 20, and the average value of the pixel values respectively corresponding to the target pixel and the reference pixel Can be used as the density difference.

DESCRIPTION OF SYMBOLS 10 1st image generation apparatus 12 Image pick-up element 20 2nd image generation apparatus 21 Image pick-up element 30 Processing apparatus 33 Boundary extraction part 34 Pixel extraction part 35 Correction | amendment part 36 Area extraction part 37 Evaluation part 38 Background memory | storage part 40 Optical element

Claims (12)

A first image generation device that generates a distance image in which a pixel value is a distance value for a target space;
A second image generation device that generates a high-definition image that is higher-definition than the distance image for the target space;
A boundary extraction unit that extracts the positions of two boundary pixels that sandwich the boundary of the object of interest in the high-definition image;
Extracting the target pixel and the reference pixel based on the position of the boundary pixel using the pixel including the boundary of the object in the distance image as a target pixel, a pixel adjacent to the target pixel and positioned on the background side as a reference pixel A pixel extraction unit to
Correction set in advance for the relationship between the position of the boundary in the range of the target pixel obtained from the high-definition image and the density difference between the target pixel and the reference pixel obtained from the high-definition image A distance image sensor comprising: a correction unit that corrects a pixel value of the target pixel extracted by the pixel extraction unit using a value. A first image generating device that generates a distance image whose pixel value is a distance value and a grayscale image that is a density value for a target space;
A second image generation device that generates a high-definition image that is higher-definition than the distance image for the target space;
A boundary extraction unit that extracts the positions of two boundary pixels that sandwich the boundary of the object of interest in the high-definition image;
Extracting the target pixel and the reference pixel based on the position of the boundary pixel using the pixel including the boundary of the object in the distance image as a target pixel, a pixel adjacent to the target pixel and positioned on the background side as a reference pixel A pixel extraction unit to
Correction value set in advance for the relationship between the position of the boundary within the range of the target pixel obtained from the high-definition image and the density difference at the position of the target pixel and the reference pixel obtained from the grayscale image A distance image sensor, comprising: a correction unit that corrects a pixel value of the target pixel extracted by the pixel extraction unit. An area extracting unit that extracts a first area corresponding to the object extracted from the distance image and a second area corresponding to the object extracted from the high-definition image;
The pixel extracting unit associates the position of the boundary pixel with the position of the target pixel by performing pattern matching between the first region and the second region extracted by the region extracting unit. The distance image sensor according to claim 1 or 2. The pixel extraction unit associates the position of the boundary pixel with the position of the target pixel by using a parallax with respect to the object of the first image generation device and the second image generation device. Item 3. The distance image sensor according to item 1 or 2. An optical element for sharing the optical axis with respect to the target space by the first image generation device and the second image generation device by branching the incident light flux from the target space into a plurality of paths; The distance image sensor according to claim 1 or 2, characterized in that 6. The imaging device for imaging each of the first image generation device and the second image generation device is formed on a single substrate so as to be a single chip. The distance image sensor according to any one of the preceding claims. An evaluation unit that compares the distance image generated by the first image generation device with the distance image corrected by the correction unit and evaluates the validity of the correction value;
The evaluation unit is an area excluding the background in the distance image, and substantially forms a connected area with an area to which the correction value is applied, and a distance value obtained from a pixel to which the correction value is not applied. 7. The correction value is determined to be valid when a difference from a distance value of a pixel to which the correction value is applied is equal to or less than a predetermined value. Distance image sensor. The boundary extraction unit extracts pixels corresponding to the object in the target space from the high-definition image in a region including pixels whose pixel values satisfy a predetermined extraction condition in the distance image. The distance image sensor according to any one of 1 to 7. A background storage unit that stores the distance image obtained in a state where the object does not exist in the target space as a background distance image;
The extraction condition is that the boundary extraction unit has a pixel value greater than or equal to a reference value in a difference image between the distance image generated by the first image generation device and the background distance image stored in the background storage unit. The distance image sensor according to claim 8, wherein a pixel corresponding to the object in the target space is extracted from the high-definition image. In combination with a first image generation device that generates a distance image having a pixel value that is a distance value for the target space, and a second image generation device that generates a high-definition image that is higher in definition than the distance image for the target space. A boundary extraction unit for extracting a position of two boundary pixels sandwiching a boundary of an object of interest in the high-definition image, and a processing device for obtaining a distance to the object of interest in the target space;
Extracting the target pixel and the reference pixel based on the position of the boundary pixel using the pixel including the boundary of the object in the distance image as a target pixel, a pixel adjacent to the target pixel and positioned on the background side as a reference pixel A pixel extraction unit to
Correction set in advance for the relationship between the position of the boundary in the range of the target pixel obtained from the high-definition image and the density difference between the target pixel and the reference pixel obtained from the high-definition image A correction unit that corrects a pixel value of the target pixel extracted by the pixel extraction unit using a value. A first image generation device that generates a distance image whose pixel value is a distance value and a grayscale image that is a density value for a target space, and a second that generates a high-definition image that is higher in definition than the distance image for the target space A processing device for obtaining a distance to a target object in the target space, and extracting positions of two boundary pixels sandwiching a boundary of the target object in the high-definition image. A boundary extraction unit;
Extracting the target pixel and the reference pixel based on the position of the boundary pixel using the pixel including the boundary of the object in the distance image as a target pixel, a pixel adjacent to the target pixel and positioned on the background side as a reference pixel A pixel extraction unit to
The position of the boundary in the range of the target pixel determined from the high-definition image, the position of the boundary in the range of the target pixel, and the density difference at the position of the target pixel and the reference pixel determined from the grayscale image A correction unit that corrects the pixel value of the target pixel extracted by the pixel extraction unit using a correction value set in advance for the relationship.   The program for functioning a computer as a processing apparatus of Claim 10 or 11.

Images