JP2006048156A - Image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor for extracting an object without being affected by light quantity change in an object space. <P>SOLUTION: A light emitting source 2 irradiates an object space with light, and a light detecting element 1 images the object space. An image generating part 4 calculates a distance to an object Ob according to a corresponding relation between the rays of light with which the object space has been irradiated from the light emitting source 2 and reflected light reflected by an object in the object space, and received by the light detecting element 1, and generates a distance image whose pixel value corresponding to each direction in the object space is a distance value. A differentiation processing part 5 generates a distance differential image whose distance differential value is a pixel value from the distance image. Furthermore, an object deciding part 6 extracts the neighborhood of a boundary between the object and the background by using the distance differential value. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus that extracts an object existing in a target space using an image of the target space.

Conventionally, various techniques for monitoring a target space using an image obtained by imaging the target space and extracting the appearance of an object in the target space have been proposed (for example, see Patent Document 1).
JP-A-11-284997

  By the way, the image used in the above-described technique is a grayscale image reflecting the lightness and darkness of the target space, and is affected by a change in the amount of external light, so that it can be used only in an environment where the amount of light hardly changes. There is.

  The present invention has been made in view of the above reasons, and an object of the present invention is to provide an image processing apparatus that can extract an object without being affected by a change in the amount of light in the object space.

  The invention of claim 1 corresponds to a light emitting source that irradiates light to a target space, a light detection element that images the target space, and reflected light that is irradiated from the light source to the target space and reflected by an object in the target space. An image generation unit for determining a distance to an object based on an output of a light detecting element that generates a distance image having a pixel value as a distance value, and a distance differential value as a differential intensity value of each pixel obtained from the distance value of the distance image A differential processing unit that generates a distance differential image having a pixel value as a pixel value, and an object determination unit that extracts an object using the distance differential value obtained by the differential processing unit.

  In this configuration, a distance differential image is generated with a distance differential value, which is a differential intensity value, for the distance image as a pixel value, so an area with a large distance change rate and an area with a small distance change rate such as a step can be easily obtained Can be identified. For example, when the distance difference between the human body as the target object and the background is relatively large in the target space and the rate of change of the distance around the target object is large, the distance differential value increases, so the contour of the human body is extracted. Easy to do. Moreover, if the distance image is within the dynamic range of the light detection element, the same distance differential value can be obtained without being affected by the change in the amount of external light, so that the object can be accurately detected. Become.

  According to a second aspect of the present invention, in the first aspect of the invention, the image generation unit generates a distance image in time series based on an output of the light detection element that images the target space with the passage of time. The object determination unit generates a difference image that is a difference between two distance differential images generated from two distance images obtained at different times, and an area in which the pixel value is equal to or greater than a predetermined threshold in the difference image. It is characterized by extracting as an object.

  According to this configuration, it is possible to extract only the region in which the distance has changed within the field of view of the light detection element due to the difference between the two differential images, and the region having a small difference is removed by the threshold. It is possible to extract the region of the object that has moved between the time when the two distance images were obtained. In addition, even if the object is almost stationary, the region near the boundary of the object is extracted if the object moves slightly like a human body or if the object is fluctuated by the wind. .

  According to a third aspect of the present invention, in the first aspect of the invention, the image generation unit generates a distance image in time series based on an output of the light detection element that images the target space as time passes. The target object determination unit generates a difference image that is a difference between each of two or more distance differential images generated from three or more distance images obtained at different times, and pixel values of the plurality of difference images And generating a plurality of binary images by extracting the region of the moving object at the time of interest by a logical operation combining pixel values at the same position in each binary image. And

  According to this configuration, the background can be removed almost completely by removing the background using the difference image and generating a binary image. In addition, since a logical operation that combines the pixel values of the binary image is performed, when the moving object is included in the distance image, the background of the object is removed by removing the background almost completely at the time of interest. Can be extracted. That is, a silhouette is extracted for the moving object.

  According to a fourth aspect of the present invention, in the first aspect of the invention, a background for storing a distance differential image for a background generated by the differential processing unit from the distance image obtained in a state where there is no object moving in the target space. A storage unit, wherein the object determination unit generates a difference image that is a difference between the distance differential image generated by the differential processing unit and the background distance differential image at a time different from the background distance differential image. In the difference image, a region where the pixel value is equal to or greater than a predetermined threshold is extracted as an object.

  According to this configuration, the distance differential image related to the background is stored in advance, and the object is extracted by comparing the difference between the distance differential image obtained by imaging the target space and the distance differential image related to the background with the threshold value. Therefore, it is possible to easily detect a region changed with respect to the background as an object.

  According to the configuration of the present invention, a distance differential image having a differential distance value that is a differential intensity value as a pixel value is generated for the distance image. Therefore, an area having a large distance change rate such as a step and a small distance change rate are generated. There is an advantage that it is possible to easily distinguish the region, and the same differential distance value can be obtained without being affected by the change in the amount of external light as long as it is within the dynamic range of the light detection element. Therefore, there is an advantage that the object can be accurately detected.

  In describing the embodiment of the present invention, the configuration of the distance image sensor used in the present embodiment will be described first.

  As shown in FIG. 1, the distance image sensor used in the present embodiment includes a light emitting source 2 that irradiates light to a target space, and receives an electric value of an output value that receives light from the target space and reflects a received light amount. The obtained light detection element 1 is provided. The distance to the object Ob existing in the object space is the time from when the light is emitted from the light source 2 to the object space until the reflected light from the object Ob enters the light detection element 1 (“flight time”). Call). However, since the flight time is very short, use the intensity-modulated light that is modulated so that the intensity of the light irradiating the target space changes periodically at a constant period, and use the phase when the intensity-modulated light is received. Ask for time. The technical idea of the present invention can be employed as a distance image sensor in a configuration in which a distance image is generated based on the principle of triangulation in addition to a configuration in which a distance image is generated based on a flight time. However, the distance image sensor with the configuration described below adopts the principle of triangulation because it can generate a distance image in a shorter time (almost real time) compared to the distance image sensor that uses the principle of triangulation. It is preferable to the distance image sensor.

As shown in FIG. 2A, if the intensity of light radiated from the light source 2 into the space changes as shown by curve A, and the amount of received light received by the light detecting element 1 changes as shown by curve B, Since the phase difference ψ corresponds to the flight time, the distance to the object Ob can be obtained by obtaining the phase difference ψ. Further, the phase difference ψ can be calculated using the received light amount of the curve B obtained at a plurality of timings of the curve A. For example, it is assumed that the received light amounts of curve B obtained with the phases of curve A at 0, 90, 180, and 270 degrees are A0, A1, A2, and A3 (received light amounts A0, A1, A2,. A3 is indicated by hatching). However, the received light amount A0, A1, A2, A3 in each phase uses the received light amount integrated at a predetermined time Tw instead of the instantaneous value. Now, while obtaining the received light amounts A0, A1, A2, and A3, the phase difference ψ does not change (that is, the distance to the object Ob does not change), and the reflectance of the object Ob does not change. Shall. Further, it is assumed that the intensity of light emitted from the light emitting source 2 is modulated by a sine wave, and the intensity of light received by the light detection element 1 at time t is represented by A · sin (ωt + δ) + B. Here, A is the amplitude, B is the external light component, ω is the angular frequency, and δ is the phase. If the received light amounts A0, A1, A2, and A3 received by the light detecting element 1 are instantaneous values instead of the integrated values of the time Tw, the received light amounts A0, A1, A2, and A3 can be expressed as follows.
A0 = A · sin (δ) + B
A1 = A · sin (π / 2 + δ) + B
A2 = A · sin (π + δ) + B
A3 = A · sin (3π / 2 + δ) + B
Since δ = −ψ, A0 = −A · sin (ψ) + B, A1 = A · cos (ψ) + B, A2 = A · sin (ψ) + B, A3 = −A · cos (ψ ) + B, and as a result, the relationship between the received light amounts A0, A1, A2, A3 and the phase difference ψ is expressed as follows.
ψ = tan ⁻¹ {(A2−A0) / (A1−A3)} (1)
Although the instantaneous values of the received light amounts A0, A1, A2, and A3 are used in the equation (1), the phase difference ψ in the equation (1) even if the integrated values at the time Tw are used as the received light amounts A0, A1, A2, and A3. Can be requested.

  As described above, in order to modulate the intensity of the light irradiated to the target space, the light source 2 includes, for example, a structure in which a large number of light emitting diodes are arranged on one plane, a combination of a semiconductor laser and a diverging lens, or the like. Is used. The light source 2 is driven by a modulation signal having a predetermined modulation frequency output from the control circuit unit 3, and the intensity of the light emitted from the light source 2 is modulated by the modulation signal. The control circuit unit 3 modulates the intensity of light emitted from the light source 2 with, for example, a 20 MHz sine wave. The intensity of the light emitted from the light source 2 may be modulated by a triangular wave, a sawtooth wave or the like in addition to the modulation by a sine wave. In short, any configuration is acceptable as long as the intensity is modulated at a constant period. May be adopted.

  The light detection element 1 includes a plurality of photosensitive portions 11 regularly arranged. A light receiving optical system 8 is disposed in the light incident path to the photosensitive portion 11. The photosensitive portion 11 is a portion where light from the target space is incident through the light receiving optical system 8 in the light detection element 1, and the photosensitive portion 11 generates an amount of charge corresponding to the amount of received light. Further, the photosensitive portions 11 are arranged on the lattice points of the planar lattice, and are arranged in a matrix in which, for example, a plurality are arranged at equal intervals in the vertical direction (that is, the vertical direction) and the horizontal direction (that is, the horizontal direction). Is done.

  The light receiving optical system 8 associates the line-of-sight direction when viewing the target space from the light detection element 1 with each photosensitive portion 11. That is, the range in which light enters each photosensitive portion 11 through the light receiving optical system 8 can be regarded as a conical field of view having a small apex angle set for each photosensitive portion 11 with the center of the light receiving optical system 8 as the apex. . Therefore, if the reflected light emitted from the light emitting source 2 and reflected by the object Ob existing in the target space is incident on the photosensitive portion 11, the optical axis of the light receiving optical system 8 depends on the position of the photosensitive portion 11 that has received the reflected light. Can be known as the reference direction.

  Since the light receiving optical system 8 is generally arranged so that the optical axis is orthogonal to the plane on which the photosensitive portion 11 is arranged, the center of the light receiving optical system 8 is the origin, and the vertical and horizontal directions of the plane on which the photosensitive portion 11 is arranged If an orthogonal coordinate system is set in which the optical axis of the light receiving optical system 8 is in the direction of the three axes, the angle (so-called azimuth and elevation angle) when the position of the object Ob existing in the object space is expressed in spherical coordinates. It corresponds to each photosensitive portion 11. The light receiving optical system 8 can also be arranged so that the optical axis intersects at an angle other than 90 degrees with respect to the plane on which the photosensitive portions 11 are arranged.

  In the present embodiment, as described above, in order to obtain the distance to the object Ob, the received light amounts A0, A1, and A2 are synchronized at four timings synchronized with the intensity change of the light emitted from the light source 2 to the target space. , A3. Therefore, it is necessary to control the timing to obtain the desired received light amount A0, A1, A2, A3. In addition, since the amount of charge generated in the photosensitive portion 11 is small in one cycle of intensity change of light irradiated from the light source 2 to the target space, it is desirable to accumulate the charge over a plurality of cycles. Therefore, as shown in FIG. 1, a plurality of charge accumulating units 13 for accumulating the charges generated in the respective photosensitive units 11 are provided, and the areas of the regions for generating the charges that can be used in the respective photosensitive units 11 are changed to change the respective areas. A plurality of sensitivity control units 12 for adjusting the sensitivity of the photosensitive unit 11 are provided.

  In each sensitivity control unit 12, the sensitivity of the photosensitive unit 11 corresponding to the sensitivity control unit 12 is increased at any one of the four points described above, and in the photosensitive unit 11 with increased sensitivity, the received light amount A0, Since charges corresponding to A1, A2, and A3 are mainly generated, charges corresponding to the received light amounts A0, A1, A2, and A3 can be accumulated in the charge accumulating unit 13 corresponding to the photosensitive unit 11.

  By the way, the sensitivity control unit 12 changes the amount of charge generated in each period by changing the area (substantial light receiving area) of the region that generates the charge that can be used in the photosensitive unit 11. The charges accumulated in 13 include not only the charges generated during the period in which the received light amounts A0, A1, A2, and A3 are obtained, but also the charges generated during other periods. Now, in the sensitivity control unit 12, the sensitivity in the period for generating the charges corresponding to the received light amounts A0, A1, A2, A3 is α, the sensitivity in the other periods is β, and the photosensitive unit 11 is a charge proportional to the received light amount. Is generated. Under this condition, the charge accumulating unit 13 that accumulates charges corresponding to the received light amount A0 is proportional to αA0 + β (A1 + A2 + A3) + βAx (Ax is the received light amount other than the period during which the received light amounts A0, A1, A2, and A3 are obtained). In the charge accumulating unit 13 that accumulates charges and accumulates charges corresponding to the received light quantity A2, charges proportional to αA2 + β (A0 + A1 + A3) + βAx are accumulated. As described above, when obtaining the phase difference ψ, (A2−A0) is obtained, and A2−A0 = (α−β) (A2−A0), and similarly, A1−A3 = (α−). β) Since (A1-A3), (A2-A0) / (A1-A3) theoretically has the same value regardless of the presence or absence of charge mixing. The phase difference ψ has the same value.

  The photodetecting element 1 including the photosensitive unit 11, the sensitivity control unit 12, and the charge accumulating unit 13 is configured as one semiconductor device, and the photodetecting element 1 transmits charges accumulated in the charge accumulating unit 13 to the outside of the semiconductor device. A charge extraction unit 14 is provided for extraction. The charge extraction unit 14 has the same configuration as the vertical transfer unit and horizontal transfer unit in the CCD image sensor.

  The electric charge extracted from the electric charge extraction unit 14 is given to the image generation unit 4 as an image signal, and the distance to the object Ob in the target space in the image generation unit 4 is calculated based on the above-described equation (1). Calculated from A1, A2, and A3. That is, the image generation unit 4 calculates the distance to the object Ob in each direction corresponding to each photosensitive unit 11, and calculates the three-dimensional information of the target space. By using this three-dimensional information, it is possible to generate a distance image in which the pixel values of the pixels matching each direction of the target space are distance values.

  Hereinafter, a specific structural example of the light detection element 1 will be described. The photodetecting element 1 shown in FIG. 3 includes a plurality of (for example, 100 × 100) photosensitive portions 11 arranged in a matrix, and is formed on, for example, a single semiconductor substrate. Each column in the vertical direction in the photosensitive portion 11 shares the semiconductor layer 21 that is integrally continuous, and the semiconductor layer 21 is used as a transfer path for charges in the vertical direction (electrons are used in this embodiment). It is possible to employ a configuration in which a semiconductor substrate is provided with a horizontal transfer portion Th that is a CCD that receives charges from one end of the semiconductor layer 21 and transfers the charges in the horizontal direction.

  That is, as shown in FIG. 4, the semiconductor layer 21 serves as the photosensitive portion 11 and the charge transfer path, and is similar to a frame transfer (FT) type CCD image sensor. Similarly to the FT type CCD image sensor, a light-shielded accumulation region Db is provided adjacent to the imaging region Da in which the photosensitive portions 11 are arranged, and charges accumulated in the accumulation region Db are transferred to the horizontal transfer unit Th. To do. The transfer of charges from the imaging area Da to the storage area Db is performed at once in the vertical blanking period, and the horizontal transfer unit Th transfers charges for one horizontal line in one horizontal period. The charge extraction unit 14 illustrated in FIG. 1 represents a function including a horizontal transfer unit Th along with a function as a charge transfer path in the vertical direction in the semiconductor layer 21. However, the charge accumulation unit 13 does not mean the accumulation region Db, but represents a function of accumulating charges in the imaging region Da. In other words, the accumulation region Db is included in the charge extraction unit 14.

  The semiconductor layer 21 is doped with impurities, the main surface of the semiconductor layer 21 is covered with an insulating film 22 made of an oxide film, and a plurality of control electrodes 23 are arranged on the semiconductor layer 21 via the insulating film 22. . This light detection element 1 has a structure known as a MIS element, but a normal MIS element is that a plurality of (five in the illustrated example) control electrodes 23 are provided in a region functioning as one light detection element 1. Is different. A material is selected for the insulating film 22 and the control electrode 23 so that light having the same wavelength as the light emitted from the light source 2 to the target space can be transmitted. When light enters the semiconductor layer 21 through the insulating film 22, the semiconductor layer 21. A charge is generated inside the. The conductivity type of the semiconductor layer 21 in the illustrated example is n-type, and electrons e are used as charges generated by light irradiation. FIG. 3 shows only a region corresponding to one photosensitive portion 11, and a plurality of regions having the structure shown in FIG. 3 are arranged on the semiconductor substrate (not shown) and the charge extraction is performed. A structure to be part 14 is provided. The vertical transfer unit provided as the charge extraction unit 14 is assumed to transfer charges in the left-right direction in FIG. 3, but it is also possible to adopt a configuration in which charges are transferred in a direction orthogonal to the plane in FIG. is there. In addition, when transferring charges in the horizontal direction in the figure, it is desirable to set the width dimension of the control electrode 23 in the horizontal direction to about 1 μm.

  In the light detection element 1 having this structure, when a positive control voltage + V is applied to the control electrode 23, a potential well (depletion layer) 24 that accumulates electrons e in a portion corresponding to the control electrode 23 is formed in the semiconductor layer 21. The That is, when light is applied to the semiconductor layer 21 with a control voltage applied to the control electrode 23 so as to form the potential well 24 in the semiconductor layer 21, a part of the electrons e generated in the vicinity of the potential well 24. Are captured in the potential well 24 and accumulated in the potential well 24, and the remaining electrons e disappear due to recombination in the deep part of the semiconductor layer 21. Further, the electrons e generated at a location away from the potential well 24 are also extinguished by recombination in the deep part of the semiconductor layer 21.

  Since the potential well 24 is formed at a portion corresponding to the control electrode 23 to which the control voltage is applied, the potential well 24 along the main surface of the semiconductor layer 21 is changed by changing the number of the control electrodes 23 to which the control voltage is applied. (In other words, the area of a region that generates a charge that can be used on the light receiving surface) can be changed. That is, changing the number of control electrodes 23 to which the control voltage is applied means adjusting sensitivity in the sensitivity control unit 12. For example, when the control voltage + V is applied to three control electrodes 23 as shown in FIG. 3A and when the control voltage + V is applied to one control electrode 23 as shown in FIG. Since the area occupied by the potential well 24 on the light receiving surface changes, and the area of the potential well 24 is larger in the state of FIG. 3A, the same light quantity is obtained compared to the state of FIG. As a result, the ratio of the charge that can be used increases and the sensitivity of the photosensitive portion 11 is substantially increased. Thus, it can be said that the photosensitive portion 11 and the sensitivity control portion 12 are constituted by the semiconductor layer 21, the insulating film 22, and the control electrode 23. The potential well 24 functions as the charge accumulation unit 13 because it holds charges generated by light irradiation.

  In order to extract charges from the potential well 24, a technique similar to that of the FT type CCD may be employed. After the electrons e are accumulated in the potential well 24, a control voltage of an applied pattern different from that during charge accumulation is controlled. By applying the voltage to the electrode 23, the electrons e accumulated in the potential well 24 can be transferred in one direction (for example, the right direction in the figure). That is, the semiconductor layer 21 can be used as a charge transfer path in the same manner as the vertical transfer portion of the CCD. Further, the electric charge is transferred through the horizontal transfer portion Th shown in FIG. 4 and is taken out of the photodetecting element 1 from an electrode (not shown) provided on the semiconductor substrate. In short, by controlling the application pattern of the control voltage to the control electrode 23, the sensitivity of each photosensitive portion 11 is controlled, charges generated by light irradiation are integrated, and the integrated charges are transferred. Can do.

  The sensitivity control unit 12 in the present embodiment switches the sensitivity of the photosensitive unit 11 to two levels of high and low by switching the area for generating available charges into two levels of large and small, and the received light amount A0, A1, A2, A3. High sensitivity is set only during a period in which the charge corresponding to any of the photosensitive portions 11 is to be generated (the area for generating charges is increased), and low sensitivity is set in other periods. The period of high sensitivity and the period of low sensitivity are set in synchronization with the modulation signal that drives the light emitting source 2. In addition, after the charges are accumulated in the potential well 24 over a plurality of periods of the modulation signal, the charges are extracted to the outside of the light detection element 1 through the charge extraction unit 14. Charges are accumulated over a plurality of periods of the modulation signal because the period during which the photosensitive unit 11 generates usable charges within one period of the modulation signal is short (for example, if the frequency of the modulation signal is 20 MHz). This is because less than a quarter of 50 ns is generated. That is, by integrating charges for a plurality of periods of the modulation signal, signal charges (charges corresponding to light emitted from the light emission source 2) and noise charges (external light components and shots generated inside the light detection element 1). A large signal-to-noise ratio can be obtained, and a large SN ratio can be obtained.

  By the way, if the charges corresponding to the four kinds of received light amounts A0, A1, A2, and A3 necessary for obtaining the phase difference ψ are generated by one photosensitive portion 11, the resolution in the line-of-sight direction is increased. There arises a problem that the time difference for obtaining the charges corresponding to the received light amounts A0, A1, A2, A3 becomes large. On the other hand, if the charges corresponding to the received light amounts A0, A1, A2, A3 are generated by the four photosensitive portions 11, respectively, the time difference for obtaining the charges corresponding to the received light amounts A0, A1, A2, A3 becomes small. However, a shift occurs in the line-of-sight direction for obtaining the four types of charges, and the resolution in the line-of-sight direction decreases. Therefore, in the present embodiment, a configuration is adopted in which two types of charges corresponding to the received light amounts A0, A1, A2, and A3 are generated in one cycle of the modulation signal by using the two photosensitive portions 11. . That is, two photosensitive portions 11 are used as a set, and light from the same line-of-sight direction is incident on the two photosensitive portions 11 in the set.

  By adopting the above-described configuration, the resolution in the line-of-sight direction can be made relatively high, and the time difference for generating charges corresponding to the received light amounts A0, A1, A2, and A3 can be reduced. That is, by reducing the time difference for generating charges corresponding to the received light amounts A0, A1, A2, and A3, the distance detection accuracy is kept relatively high even for the object Ob moving in the target space. be able to. In the configuration of this embodiment, the resolution in the line-of-sight direction is lower than that in the case where charges corresponding to four types of received light amounts A0, A1, A2, and A3 are generated by one photosensitive unit 11, but the line-of-sight direction The resolution can be improved by downsizing the photosensitive unit 11 or designing the light receiving optical system 8.

  The operation will be specifically described below. In the example shown in FIG. 3, an example in which five control electrodes 23 are provided for one photosensitive portion 11 is shown. However, two control electrodes 23 on both sides are charged (electrons e) by the photosensitive portion 11. In the case where two photosensitive portions 11 are used as a pair, the adjacent photosensitive portions are formed. Since any one of the photosensitive portions 11 forms a barrier between the 11 potential wells 24, it is sufficient to provide three photosensitive electrodes 11 for each of the photosensitive portions 11. With this configuration, the occupation area per one photosensitive portion 11 is reduced, and it is possible to suppress a decrease in resolution in the line-of-sight direction while using the two photosensitive portions 11 as a set.

  Here, as shown in FIG. 5, the numbers (1) to (6) are assigned to the control electrodes 23 in order to distinguish the three control electrodes 23 provided in the two photosensitive sections 11 respectively. Attached. The two photosensitive portions 11 having the control electrodes 23 assigned with the numbers (1) to (6) correspond to one line-of-sight direction and constitute pixels in the image sensor. In addition, it is desirable to provide an overflow drain in association with the photosensitive portion 11 for each pixel.

  5A and 5B show a state in which a control voltage + V is applied to the control electrode 23 in a different application pattern (a state in which a control voltage + V is applied between a substrate electrode (not shown) provided on the semiconductor substrate and the control electrode 23). As can be seen from the shape of the potential well 24, in FIG. 5A, a positive control voltage + V is applied to the control electrodes (1) to (3) of the two photosensitive portions 11 serving as one pixel. In addition, a positive control voltage + V is applied to the central control electrode (5) among the remaining control electrodes (4) to (6). In FIG. 5B, a positive control voltage + V is applied to the central control electrode (2) among the control electrodes (1) to (3), and the remaining control electrodes (4) to (6) are applied. A positive control voltage + V is applied. In other words, the application pattern of the control voltage + V applied to the two photosensitive portions 11 constituting one pixel is alternately replaced. The timing of switching the application pattern of the control voltage + V applied to the two photosensitive portions 11 is the timing of the opposite phase (the phase is 180 degrees different) in the modulation signal. In addition, except for the period in which the control voltage + V is simultaneously applied to the three control electrodes 23 provided in each photosensitive portion 11, one central control electrode 23 (that is, the control electrode) provided in each photosensitive portion 11 is provided. (2) The control voltage + V is applied only to (5)), and the other control electrodes 23 are kept at 0V.

  For example, when the charges corresponding to the received light amounts A0 and A2 are alternately generated in the two photosensitive portions 11 constituting one pixel, the charges corresponding to the received light amount A0 in one photosensitive portion 11 as shown in FIG. While the control voltage + V is applied to the three control electrodes (1) to (3) in order to generate the signal, the other photosensitive portion 11 has one charge to hold the charge corresponding to the received light quantity A2. A control voltage + V is applied only to the control electrode (5). Similarly, while the control voltage + V is being applied to the three control electrodes (4) to (6) in order to generate a charge corresponding to the received light amount A2 in one photosensitive unit 11, the other photosensitive unit 11 is exposed. The unit 11 applies the control voltage + V only to one control electrode (2) in order to hold the charge corresponding to the received light quantity A0. In addition, the control voltage + V is applied only to the control electrodes (2) and (5) except for the period in which charges corresponding to the received light amounts A0 and A2 are generated. FIGS. 2B and 2C show the application timing of the control voltage + V to the control electrodes (1) to (6) when accumulating charges corresponding to the received light amounts A0 and A2. In the figure, the hatched portion indicates a state where the control voltage + V is applied, and the blank portion indicates a state where no voltage is applied to the control electrodes (1) to (6).

  The same applies to the case where the charges corresponding to the received light amounts A1 and A3 are generated in the two photosensitive portions 11 constituting one pixel, and the case where the charges corresponding to the received light amounts A0 and A2 are generated is different from the control electrode 23. The only difference is that the timing at which the control voltage + V is applied differs by 90 degrees in the phase of the modulation signal. In addition, the charge is transferred from the imaging region to the accumulation region between a period for generating charges corresponding to the received light amounts A0 and A1 and a period for generating charges corresponding to the received light amounts A1 and A3. That is, charges corresponding to the received light amount A0 are accumulated in the potential well 24 corresponding to the control electrodes (1) to (3), and charges corresponding to the received light amount A2 correspond to the control electrodes (4) to (6). When accumulated in the potential well 24, the charges corresponding to the received light amounts A0 and A2 are taken out. Next, charges corresponding to the received light amount A1 are accumulated in the potential well 24 corresponding to the control electrodes (1) to (3), and charges corresponding to the received light amount A3 are applied to the control electrodes (4) to (6). When accumulated in the corresponding potential well 24, charges corresponding to these received light amounts A1 and A3 are taken out to the outside. By repeating such an operation, the charges corresponding to the received light amounts A0, A1, A2, and A3 of the four sections can be taken out of the light output element 1 by two reading operations, and the extracted charges are used. The phase difference ψ can be obtained. For example, in order to obtain an image of 30 frames per second, the period for generating charges corresponding to the received light amounts A0 and A1 and the period for generating charges corresponding to the received light amounts A1 and A3 are shorter than 1/60 second. A short period.

  In the above example, the control voltage applied simultaneously to the three control electrodes 23 ((1) to (3) or (4) to (6)) and one control electrode 23 ((2) or (5)) Since the control voltage applied only to is equal, the area of the potential well 24 changes, but the depth of the potential well 24 is equal. In this case, the charges generated at the control electrode 23 ((1) (3) or (4) (6)) to which no control voltage is applied flow into the potential well 24 with a similar probability. That is, by applying the control voltage + V to only one of the three control electrodes 23 constituting the photosensitive portion 11, the region functioning as the charge accumulation portion 13 and all the three control electrodes 23 are applied. A similar amount of charge flows into both the region to which the control voltage + V is applied. That is, since the noise component flowing into the potential well 24 holding the charge is relatively large, the dynamic range is lowered.

  Therefore, as shown in FIG. 6, the control voltage applied simultaneously to each of the three control electrodes (1) to (3) or (4) to (6) provided in the two photosensitive portions 11 in the set is 1 It is desirable to set the potential well 24 to be higher than the control voltage applied only to the individual control electrodes (2) or (5), and to set the large-area potential well 24 deeper than the small-area potential well 24. As described above, the potential well 24 that mainly generates charges (electrons e) is made deeper than the potential well 24 that mainly holds charges, so that the control electrode (1) to which no control voltage is applied is applied. Charges generated at the sites corresponding to (3) or (4) and (6) are likely to flow into the deeper potential well 24. That is, as compared with the case where a constant control voltage + V is applied to the control electrode 23, the noise component flowing into the potential well 24 holding the charge can be reduced.

  In the configuration example of the distance image sensor described above, four periods corresponding to the received light amounts A0, A1, A2, and A3 are set so that the phase interval is 90 degrees in one cycle of the modulation signal. If the phase with respect to the modulation signal is known, the four periods can be set at appropriate intervals other than 90 degrees. However, the formula for obtaining the phase difference ψ differs if the interval is different. Also, the period of taking out the charge corresponding to the amount of received light in the four periods is within one period of the modulation signal as long as the reflectance of the object Ob and the external light component do not change and the phase difference ψ does not change. Thus, it is not essential to take out four signal charges. Furthermore, when there is an influence of disturbance light such as sunlight or illumination light, it is desirable to dispose an optical filter that transmits only the wavelength of light emitted from the light source 2 in front of the photosensitive portion 11. In the configuration example described with reference to FIGS. 5 and 6, three control electrodes 23 are associated with each photosensitive portion 11, but four or more control electrodes 23 may be provided. In the above example, the same configuration as the FT type CCD image sensor is adopted, but the same configuration as the interline transfer (IT) method and the frame interline transfer (FIT) method is adopted. Is also possible.

  Next, a technique for extracting an object existing in the target space using the above-described distance image sensor will be described. Here, in order to reduce the incidence of external light components on the light detection element 1 when generating a distance image, the target space is irradiated with infrared light from the light source 2 and an infrared transmission filter is provided in front of the light detection element 1. Assume that it is arranged.

As shown in FIG. 1, the distance image generated by the image generation unit 4 is given to the differentiation processing unit 5, and the differentiation processing unit 5 is a distance differentiation that is a differential intensity value of each pixel obtained from the distance value of the distance image. A distance differential image with the value as the pixel value is generated. The differential intensity value is a value obtained using, for example, 8 neighborhoods of the pixel of interest, and in this embodiment, when the pixel p5 in the center of the 3 × 3 square area as shown in FIG. By using the pixel values B1 to B4 and B6 to B9 of the other pixels p1 to p4 and p6 to p9, they are expressed by equation (2).
(ΔX ² + ΔY ² ) ^1/2 (2)
However,
ΔX = (B1 + B4 + B7) − (B3 + B6 + B9)
ΔY = (B1 + B2 + B3) − (B7 + B8 + B9)
In this embodiment, since the distance differential value is obtained, the pixel values B1 to B4 and B6 to B9 are distance values. In a distance differential image having a distance differential value obtained by the equation (2) as a pixel value, the distance differential value increases as the distance difference in the distance image increases.

  The distance differential image is given to the object determination unit 6, and the object Ob existing in the object space is detected. Below, the technique which detects the target object Ob from a distance differential image in the target object determination part 6 is demonstrated.

  In general, an object Ob such as a person has a large rate of change in the distance value near the boundary with the background. Therefore, if an area where the distance differential value is a maximum value is extracted from the distance differential image, the contour of the object Ob is included. A region having a width of about a pixel can be extracted. If the background information excluding the target object Ob to be extracted in the target space is known, only the target object Ob can be obtained by removing the background using the known information from the region where the distance differential value becomes the maximum value. Can be extracted. Although the maximum value is used here, the same processing can be performed by binarizing with an appropriate threshold. If the amount of external light is within the dynamic range of the light detection element 1, the distance differential value is not affected by the change in the amount of external light, and therefore is affected by the amount of external light as in the case of using a grayscale image. The object Ob can be detected accurately.

  By the way, in order to obtain the background information, the distance differential image for the background generated by the differential processing unit 5 from the distance image obtained in the state where the object Ob moving in the target space does not exist is stored in advance. desirable. Therefore, the object determination unit 6 is provided with a background storage unit 6a for storing a distance differential image obtained in a state where the object Ob does not exist in the object space. The object determination unit 6 is a difference that is a difference between the distance differential image generated by the differential processing unit 5 at a time different from the background distance differential image and the background distance differential image stored in the background storage unit 6a. An image is generated, and a region where the pixel value is equal to or greater than a predetermined threshold in the difference image is extracted as the object Ob. That is, in the difference image, since the region changed with respect to the background is extracted, the object Ob can be easily extracted by separating from the background.

  As described above, in order to use the differential image of the background, it is necessary to generate a distance image of the target space in advance without the object Ob. On the other hand, in the case of a moving object Ob, a stationary background excluding the object Ob can be removed by generating a difference image that is a difference between two distance differential images.

  When extracting the region corresponding to the moving object Ob, for example, the charge is taken out from the photodetecting element 1 so that a distance image of 30 frames per second can be generated as in the case of a TV camera, and the generated distance image is used. The differential processing unit 5 generates a distance differential image. The two frames for generating the difference image may employ either a pair of adjacent frames or a pair of frames separated from each other by a plurality of frames. However, when the object Ob moves relatively quickly in the target space, it is desirable to use a pair of adjacent frames.

  When a difference image is generated from a distance differential image, a pixel value other than 0 is obtained in a pixel in which the distance differential value has changed in the difference image. By binarizing the difference image with an appropriate threshold value, the difference image is obtained. It is possible to extract a region in which the difference in the distance differential value between the two frames that have generated is greater than or equal to the threshold value. Since such an area is generated by the movement of the object Ob, the area of the object moved between the times when two distance images are obtained by removing noise other than the object Ob is extracted. Can do. Here, even when the object Ob is almost stationary, the object Ob is a human body and moves slightly, or when the object Ob is a curtain and fluctuates in the wind or the like, the object A region near the boundary of Ob can be extracted.

  An image obtained by binarizing the difference image is almost identical to an image obtained by binarizing two distance differential images that generate the difference image and using the exclusive OR of each pixel in the binarized image as a pixel value. . Since the region of the object Ob in both frames is extracted in this image, two regions corresponding to the object Ob are extracted when the object Ob moves.

  As described above, when the object Ob is extracted from the difference image generated from the two frames, the object Ob is extracted at two locations, and the object Ob included in each frame cannot be separated. When extracting only the area of the object Ob at a specific time (specific frame), the object extraction unit 6 performs the following process using three or more frames. The specific procedure is different between the case of using three frames and the case of using four or more frames, but since the concept is the same, the processing procedure will be described below in the case of using three frames.

  Now, it is assumed that three distance differential images respectively obtained from the distance images at times T−ΔT, T, and T + ΔT are used. The distance differential images at each time T−ΔT, T, and T + ΔT are binarized with appropriate threshold values, respectively, and three binary images E (T−ΔT), E from the distance differential image as shown in FIG. (T) and E (T + ΔT) are generated. In such binary images E (T−ΔT), E (T), and E (T + ΔT), the region including the contour of the object Ob has a pixel value different from the background. Here, the pixel value of the region including the contour of the object Ob is set to “1”.

  The object extraction unit 6 obtains a difference between each pair of binary images adjacent to each other in time series (that is, E (T−ΔT) and E (T), E (T) and E (T + ΔT)). In calculating the difference, a logical operation for obtaining an exclusive OR (XOR) for each pair of pixels at the same position for each pair of binary images is performed. The image thus obtained is hereinafter referred to as a binary difference image. In the binary difference image obtained as the difference between each pair of binary images, the object Ob appears twice each as shown in FIG. 8B. This is the same as when a differential image of two differential distance images is generated.

  The three distance differential images are used to extract the object Ob included in the distance differential image at a specific time (time T), and therefore the target included in the binary image E (T). In order to extract Ob, a logical operation for obtaining a logical product for each pair of pixels at the same position is performed on two binary difference images. That is, since the background is almost removed in the two binary difference images, when the logical product operation is performed on the two binary difference images, the common part moves at time T as shown in FIG. It becomes an area | region corresponding to the target object Ob to be performed. In this way, by generating a plurality of binary images and appropriately performing a logical operation that combines pixel values at the same position in each binary image, it is possible to extract the region of the object at the time of interest (time T). It becomes possible. Since the background of the extracted region is almost completely removed and the boundary of the object Ob is extracted, the silhouette of the moving object Ob at the time T is extracted.

It is a block diagram which shows embodiment of this invention. It is operation | movement explanatory drawing same as the above. It is operation | movement explanatory drawing of the principal part of the photon detection element used for the same as the above. It is a top view of the photon detection element used for the same as the above. It is operation | movement explanatory drawing of the principal part of the photon detection element used for the same as the above. It is operation | movement explanatory drawing of the principal part of the photon detection element used for the same as the above. It is operation | movement explanatory drawing of a differentiation processing part same as the above. It is operation | movement explanatory drawing of the process which extracts a target object in the same as the above.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Photodetection element 2 Light emission source 3 Control circuit part 4 Image generation part 5 Differentiation processing part 6 Object determination part 6a Background memory | storage part 11 Photosensitive part Ob Object

Claims (4)

  A light source that emits light to the target space, a light detection element that images the target space, and an output of the light detection element that corresponds to the reflected light that is irradiated from the light source to the target space and reflected by the target in the target space. An image generation unit that calculates the distance to the object and generates a distance image whose pixel value is the distance value, and a distance differentiation that uses the distance differential value that is the differential intensity value of each pixel determined from the distance value of the distance image as the pixel value An image processing apparatus comprising: a differential processing unit that generates an image; and an object determination unit that extracts an object using a distance differential value obtained by the differential processing unit.   The image generation unit is configured to generate a distance image in time series based on an output of the light detection element that images the target space over time, and the object determination unit includes two images obtained at different times. The difference image which is the difference of the two distance differential images produced | generated from the distance image of this is produced | generated, and the area | region where a pixel value is more than a predetermined | prescribed threshold value in a difference image is extracted as a target object. Image processing apparatus.   The image generation unit is configured to generate a distance image in time series based on an output of the light detection element that images the target space over time, and the object determination unit includes three images obtained at different times. A difference image that is a difference between two of the three or more distance differential images generated from the above distance images is generated, and a plurality of binary values are compared by comparing pixel values of the plurality of difference images with a prescribed threshold value. The image processing apparatus according to claim 1, wherein the region of the moving object at the time of interest is extracted by a logical operation that generates an image and combines pixel values at the same position in each binary image.   A background storage unit that stores a background differential image generated by the differential processing unit from the distance image obtained in a state where there is no target object moving in the target space, and the target determination unit A differential image that is a difference between the differential differential image generated by the differential processing unit and the differential differential image for background at a time different from the differential differential image of The image processing apparatus according to claim 1, wherein the image processing device is extracted as an object.

Images