WO2022264576A1

WO2022264576A1 - Distance measurement device and distance measurement method

Info

Publication number: WO2022264576A1
Application number: PCT/JP2022/011891
Authority: WO
Inventors: 和彦村岡
Original assignee: ソニーセミコンダクタソリューションズ株式会社
Priority date: 2021-06-15
Filing date: 2022-03-16
Publication date: 2022-12-22
Also published as: JP2022191058A

Abstract

[Problem] To provide a distance measurement device and a distance measurement method in which it is possible to suppress the effect of the color of a measurement object. [Solution] According to the present disclosure, this distance measurement device comprises a projection unit for projecting a prescribed first color pattern and a prescribed second color pattern onto a measurement object in the stated order, and an information processing unit for measuring the distance to the measurement object on the basis of the first and second color patterns, the distance measurement device being such that lights projected in the same projection region in the first and second color patterns are of different wavelength bands.

Description

Ranging device and ranging method

The present disclosure relates to a rangefinder and a rangefinder method.

As an active method that applies the principle of triangulation, a method is generally known in which a light source image projected onto the object surface from a light source as one viewpoint is observed by a camera set at the other viewpoint and the shape is measured. Such active methods project two-dimensional color patterns such as stripe patterns.

Japanese Unexamined Patent Application Publication No. 2002-191058 JP 2019-153822 A

However, imaging a color pattern requires reflection for the wavelength of each color of the color pattern. Therefore, it may be difficult to measure a colored subject. In addition, since the surface of the object is illuminated with light of a specific color, the accuracy of observing the original color of the object may decrease.
Therefore, the present disclosure provides a distance measuring device and a distance measuring method capable of suppressing the influence of the color of the object to be measured.

In order to solve the above problems, according to the present disclosure, a projection unit that sequentially projects a predetermined first color pattern and a predetermined second color pattern onto a measurement target;
an information processing unit that measures the distance to the measurement object based on the first color pattern and the second color pattern;
with
The distance measuring device, wherein light projected onto the same projection area of the first color pattern and the light of the second color pattern are light of different wavelength bands.

The light projected onto the same projection area of the first color pattern and the second color pattern is composed of a combination of light of three wavelength bands. At least one of the three wavelength bands of light may be projected, and the remaining wavelength bands of the three wavelength bands of light may be projected in the second color pattern.

Each of the light in the three wavelength bands may be light in the red band, the green band, or the blue band.

an imaging unit that sequentially images the measurement object onto which the first color pattern and the second color pattern are projected;
You may also prepare.

The imaging unit is
A timing generation unit may be provided for generating a control signal for controlling projection timing of the first color pattern and the second color pattern in the projection unit and imaging timing of the imaging unit.

The imaging unit is
A red pixel having sensitivity to light in the red band and having less sensitivity to green and blue bands than light in the red band, and a pixel having sensitivity to light in the green band and having sensitivity to red and blue bands Green pixels whose sensitivity is suppressed from light in the green band, and blue pixels which have sensitivity to light in the blue band and whose sensitivity to red and green bands is suppressed from light in the blue band. It may have a pixel array in which imaging pixels are arranged two-dimensionally.

Each of the red pixel, the green pixel, and the blue pixel within the imaging pixel may generate more pixel signals for either the first color pattern or the second color pattern. .

The imaging unit is
A first accumulated charge amount accumulated according to the first color pattern and accumulated according to the second color pattern for each of the red pixel, the green pixel, and the blue pixel in the imaging pixel A voltage corresponding to the difference between the accumulated charge amount and the accumulated charge amount may be converted into a digital pixel signal.

Based on the control signal generated by the timing generator,
Each of the red pixel, the green pixel, and the blue pixel in the imaging pixel may be controlled such that the imaging time for the first color pattern and the imaging time for the second color pattern are equal. good.

The information processing unit
An image generation unit may be provided that generates a color image of the measurement object based on the output signal of the imaging pixel for the first color pattern and the output signal of the imaging pixel for the second color pattern.

The information processing unit
Based on the digital pixel signal, patterns are sequentially detected from a digital image of the first color pattern and the second color pattern, and the detected pattern, the first color pattern projected by the projection unit, and the second color pattern are detected. It may have a measuring unit that associates at least one of the two color patterns and generates a distance to each portion of the measurement target.

The first color pattern and the second color pattern are composed of a plurality of partitioned regions, and the color of light in each partitioned region is any one of the red band, the green band, and the blue band. or any combination of two of the red band, the green band and the blue band of light.

The plurality of partitioned areas may be grouped, and the color of the light for each partitioned area within the group may be different.

In the plurality of partitioned areas, combinations of two adjacent colors may all be different.

The colors of the plurality of partitioned areas may be a cyclic pattern based on a De Brown sequence.

The plurality of partitioned areas may be a tile pattern that is two-dimensionally arranged in a matrix.

The imaging unit is
For each of the red pixel, the green pixel, and the blue pixel in the imaging pixel, a voltage corresponding to a first accumulated charge amount accumulated according to the first color pattern is converted into a first digital signal. Then, a voltage corresponding to the accumulated charge amount accumulated according to the second color pattern may be converted into a second digital signal, and the difference between them may be used as a digital pixel signal.

The information processing unit
A pattern generation unit that generates the first color pattern and the second color pattern may be provided, and a signal having information on the first color pattern and the second color pattern may be supplied to the projection unit.

The imaging unit is
The infrared wavelength band is divided into three wavelength bands, and consists of pixels that are sensitive only to each wavelength band,
The first color pattern and the second color pattern may be configured in each of the three wavelength bands of the infrared region.

The imaging unit is
At least the wavelength band may be divided into two wavelength bands, and pixels having sensitivity only to each wavelength band may be configured.

In order to solve the above problems, according to the present disclosure, a projecting step of sequentially projecting a predetermined first color pattern and a predetermined second color pattern onto a measurement object;
an information processing step of measuring a distance to the measurement target based on the first color pattern and the second color pattern;
with
Light projected onto the same projection area of the first color pattern and the second color pattern has different wavelengths from light projected onto the same projection area of the first color pattern and the second color pattern. A ranging method is provided, which is a band of light.

The figure which shows the structural example of the distance measuring device which concerns on embodiment. FIG. 4 is a diagram for explaining the principle of measurement up to a measurement target; FIG. 2 is a block diagram showing a more detailed configuration example of the distance measuring device according to the embodiment; FIG. 3 is a schematic diagram showing a configuration example of a pixel array; FIG. 4 is a diagram schematically showing a color arrangement example of a first color pattern and a second color pattern; A table showing combinations of red light, green light, and blue light in color patterns. Table showing color code examples. FIG. 4 is a diagram schematically showing a projection example of a first color pattern; FIG. 4 is a diagram showing an example of projection onto a non-reflective area; The table shown in FIG. 6A further shows the reaction of each pixel by imaging R/G/B. The table when only the first color pattern is projected. FIG. 4 is a diagram showing a configuration example of imaging pixels; FIG. 2 is a block diagram showing an example of an ADC circuit forming an AD converter; Schematic diagram of a differential exposure process. FIG. 4 is a diagram showing an example of a shutter timing chart; FIG. 4 is a diagram showing an example of a transfer timing chart of floating diffusion; FIG. 4 is a diagram showing an example of a timing chart of reading; FIG. 4 is a diagram showing a case where a pixel has sensitivity to the first color pattern; FIG. 10 is a diagram showing a case where a pixel has sensitivity to a second color pattern; A table showing the luminance when a pattern is projected onto a reflective object. A table showing the result of performing differential AD conversion on an object having reflectance (R: 50%, G: 100%, B: 75%). A table showing the results of conversion to significant color components. FIG. 6B is a table showing results when the pattern shown in FIG. 6A is subjected to differential AD conversion according to the present embodiment; FIG. A table showing the results of measuring an object that does not reflect blue light. Table showing color stripe patterns generated by De Brown. FIG. 10 is a diagram showing an example of extension of a cyclic pattern by the de Brownian sequence algorithm; A table showing a color tile pattern with color variations extended in the column direction. FIG. 27 is a diagram showing the arrangement of the color tile pattern example of FIG. 26; FIG. 10 is a diagram showing a timing chart for reading a counter difference and a digital difference; The figure which shows the waveform of the comparator input of ADC. FIG. 10 is a diagram showing a case where a specific pixel has sensitivity to a second color pattern;

Hereinafter, embodiments of the distance measuring device and the distance measuring method will be described with reference to the drawings. Although the main components of the rangefinder and the rangefinder method will be mainly described below, the rangefinder and the rangefinder method may include components and functions that are not shown or described. The following description does not exclude components or features not shown or described.

(one embodiment)
FIG. 1 is a diagram showing a configuration example of a distance measuring device 1 according to this embodiment. As shown in FIG. 1 , the distance measuring device 1 includes a projection section 100 , an imaging section 200 and an information processing section 300 . The projection unit 100 according to the present embodiment is, for example, a projector, and sequentially projects a plurality of types of color patterns onto the measurement object T100. A region T100a indicates a convex region of the measurement object T100.

The information processing section 300 includes, for example, a CPU (Central Processing Unit) and an MPU (Micro Processor), and configures each processing section by executing a program stored in the storage section. The information processing section 300 generates a plurality of types of color patterns to be projected from the projection section 100 and supplies them to the projection section 100 . Further, the information processing section 300 processes images of a plurality of types of color patterns captured by the imaging section 200, and generates distance information and color information of the measurement target T100. The program used by the information processing section 300 may be stored in the storage section, or may be stored in a storage medium such as a DVD (Digital Versatile Disc), a cloud computer, or the like. In the information processing unit 300, the program may be executed by a CPU (Central Processing Unit) or an MPU (Micro Processor) using a RAM (Random Access Memory) or the like as a work area, or may be executed by an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate), or the like. may be implemented by an integrated circuit of

The imaging unit 200 is, for example, a camera, and captures images of multiple types of color patterns projected onto the measurement object T100.

FIG. 2 is a diagram for explaining the measurement principle up to the measurement object T100. As shown in FIG. 2, the information processing section 300 generates distance information based on the principle of triangulation. Let A be the viewpoint of the projection unit 100, B be the viewpoint of the imaging unit 200, and L be the length of a line segment AB connecting A and B. FIG. In this case, the distance L and the angles of the projection unit 100 and the imaging unit 200 with respect to the line segment AB are set in the information processing unit 300 in advance.

Thereby, the distance to the point P of the measurement object T100 is calculated by the formula (1). The angle α from the imaging unit 200 is set for each projection angle of color patterns, which will be described later. Also, the angle β is calculated by using the positional information of the color pattern on the image captured by the imaging unit 200 . That is, the distance and surface shape of the measurement object T100 are measured by calculating the angles α and β of each point P at the edge of each color pattern over the entire surface of the measurement object T100.

FIG. 3 is a block diagram showing a more detailed configuration example of the distance measuring device 1 according to this embodiment. As described above, the projection unit 100 is, for example, a projector. This projector is composed of, for example, a light source, a liquid crystal panel, and a lens. The projection unit 100 sequentially projects the first color pattern (pattern1) and the second color pattern (pattern2) onto the measurement object T100 according to the synchronization signal generated by the imaging unit 200 .

The liquid crystal panel is transmissive and has elements that transmit or block the light emitted from the light source. These elements are arranged in a matrix to generate an image input to the projector. The liquid crystal panel has red (R)/green (G)/blue (B) pixels arranged in a matrix, and is sequentially driven by the image input to the projector, that is, the first color pattern and the second color pattern. be done. Light emitted from the light source passes through the liquid crystal panel and is projected onto the screen by the lens. As described above, in this embodiment, the first color pattern and the second color pattern are sequentially projected onto the measurement object T100 in accordance with the synchronization signal generated by the imaging unit 200. FIG. In addition, in this embodiment, red, green, and blue may be simply described as R, G, and B.

Also, in this projector, by turning on the light source, the difference in scanning speed between the pixel array 206 of the imaging unit 200 and the projecting unit 100 is suppressed, and the pixel array 206 and the projecting unit 100 can operate in synchronization. For example, in addition to the light source control function, the projector according to this embodiment can update the first color pattern and the second color pattern of the liquid crystal panel before the light emission timing. Details of the update timing will be described later. Further, digital light processing (DLP) or the like may be used instead of the liquid crystal panel for the image generation element used in the projector. In this case, light emitted from the light source is reflected by digital light processing (DLP) and projected onto a screen through a lens. Projection of the projector can be controlled by light emission of the light source, and can be used in the projection unit 100 according to the present embodiment.

The imaging unit 200 has a lens 202 and an image sensor 204 . An image of the object T100 to be measured is formed on the image sensor 204 via the lens 202, and captured by the principle of photoelectric conversion. That is, the image sensor 204 includes a pixel array 206, an AD converter 208, a signal processor 210, an image output interface 212, a timing controller 214, a pixel controller 216, and a timing output interface 218. have.

FIG. 4 is a schematic diagram showing a configuration example of the pixel array 206. As shown in FIG. As shown in FIG. 4, the pixel array 206 is configured by arranging a plurality of imaging pixels P206 in a matrix. Also, each imaging pixel P206 has an RGB pixel. For example, the imaging pixel P206 includes a red (R) pixel photoelectrically converted via a red filter, a green (G) pixel photoelectrically converted via a green filter, and a blue (B) pixel photoelectrically converted via a blue filter. ) color pixels.

For example, each pixel uses a type that transfers accumulated charges (here, photoelectrons) accumulated by photoelectric conversion of the photodiode PD to the floating diffusion FD and accumulates them (see Patent Document 2). By using such a type pixel in which a phototransistor (PD) and a floating diffusion FD form a pair, a voltage corresponding to the first accumulated charge amount corresponding to the first color pattern and the second A voltage corresponding to the second accumulated charge amount corresponding to the second color pattern can be AD-converted at once to generate a digital pixel value. The details of the pixels of the pixel array 206 and the AD conversion operation will be described later.

As will be described later, the color pattern according to the present embodiment includes red light corresponding to the transmission band of the red filter, green light corresponding to the transmission band of the green filter, and green light corresponding to the transmission band of the blue filter. and a corresponding blue light. For this reason, the red (R) color pixel is configured such that its sensitivity is suppressed and does not react to blue light and green light in the color pattern. Similarly, the green (G) color pixel is configured to have reduced sensitivity and not react to red and blue light in the color pattern. Similarly, the blue (B) color pixel is configured such that it has reduced sensitivity and does not react to red and green light in the color pattern. That is, each imaging pixel P206 has sensitivity to light in the red band, and a red (R) pixel whose sensitivity to green and blue bands is suppressed more than light in the red band, and sensitivity to light in the green band. and a green (G) pixel in which the sensitivity to the red band and the blue band is suppressed from the light in the green band, and a pixel having sensitivity to the light in the blue band and having sensitivity to the red band and the green band and blue (B) pixels that are suppressed from light in the blue band.

It should be noted that in this embodiment, in order to save power and speed up processing, a pixel of a type in which a transistor (PD) and a rolling diffusion FD are paired is used, but the present invention is not limited to this. For example, when sequentially performing AD conversion on the image signals for the first color pattern and the second color pattern, general pixels configured by photoelectric conversion elements may be used.

Also, in the present embodiment, the pixels are composed of red (R) color pixels, green (G) color pixels, and blue (G) color pixels, but are not limited to this. For example, the infrared wavelength band may be divided into three separate wavelength bands, and a pixel having sensitivity only to each wavelength band may be configured. In this case, the color pattern consists of each of three separate wavelength bands. In this case, since the wavelength band to which the human eye has no visual sensitivity is used for distance measurement, the distance can be measured without being detected by human vision. In the present embodiment, the red (R) color pixel, green (G) color pixel, and blue (G) color pixel may simply be referred to as R pixel, G pixel, and B pixel, respectively.

As shown in FIG. 4 again, the pixel control unit 216 controls the shutter/rotating diffusion for each of the R/G/B pixels that make up each imaging pixel P206 by horizontally wired control wires. (FD) Controls transfer/reading. The control lines (control wires 8-1) for control are shared in the horizontal direction, and row addresses (vertical addresses 8-1) are assigned from top to bottom.

Wirings VSL1 to VSLn arranged in the vertical direction are signal lines for supplying R/G/B pixel data to the AD converter 208 (see FIG. 3), and are accumulated from pixels selected by control wires. A signal charge is input to the AD converter 208 .

A voltage corresponding to the accumulated signal charge accumulated in each pixel of the pixel array 206 is supplied to the AD converter 208 via the wirings VSL1 to VSLn. The AD conversion unit 208 AD-converts the voltage corresponding to the accumulated signal charge and converts it into a digital pixel signal, which is digital data. The AD converter 208 performs, for example, differential AD conversion. The AD conversion unit 208 linearly converts a voltage corresponding to the accumulated signal charge (photoelectric charge) of each pixel accumulated by exposure, and calculates the first accumulated charge amount of the first frame corresponding to the imaging of the first color pattern. and the second accumulated charge amount of the second frame corresponding to imaging of the second color pattern. Details of an operation example of the AD conversion unit 208 will also be described later.

The signal processing unit 210 supplies the digital image data generated by the AD conversion unit 208 to the image output interface 212 . In this case, the signal processing unit 210 can perform various signal processing such as noise reduction processing and threshold processing.

The image output interface 212 performs level conversion to the signal line. The image output interface 212 then supplies the image data generated by the AD converter 208 to the information processor 300 .

The timing control unit 214 controls the timing of the first color pattern and the second color pattern projected from the projection unit 100, the imaging timing of the first color pattern and the second color pattern in each pixel of the pixel array 206, and AD conversion. Control is performed to synchronize with the AD conversion timing of the unit 208 . More specifically, the timing control section 214 controls the drive timing of the differential AD conversion for the AD conversion section 208 . The timing control unit 214 controls the imaging timing of the first color pattern and the second color pattern in each pixel of the pixel array 206 for the pixel control unit 216 .

The pixel control unit 216 controls the imaging timing of the first color pattern and the second color pattern in each pixel of the pixel array 206 according to the timing control of the timing control unit 214 as described above. More specifically, it controls control timing of initialization of each pixel of the pixel array 206, photoelectric conversion in the phototransistor (PD), transfer from the phototransistor (PD) to the rolling diffusion FD, and the like.

The timing output interface 218 converts the level of the control signal including the synchronization timing information generated by the timing control unit 214 to the signal line. Then, the timing output interface 218 supplies the information processing section 300 with a control signal including information on the synchronization timing.

The information processing unit 300 includes a timing input interface 302, a pattern generation unit 304, a projector output interface 306, a light emission control unit 308, an image input interface 310, a second signal processing unit 312, an

image generation unit

314, 3 It has a dimension measurement calculator 316 and a data output interface 318 . The timing input interface 302 inputs a control signal including synchronization timing information generated by the timing control section 214 .

The pattern generator 304 generates the first color pattern and the second color pattern according to the synchronization timing generated by the timing controller 214 .

The projector output interface 306 supplies an image signal including information on the first color pattern and the second color pattern generated by the pattern generation unit 304 to the projection unit 100 . The light emission control unit 308 controls the light emission of the light source of the projection unit 100 in correspondence with the projection timing of the first color pattern and the second color pattern according to the synchronization timing generated by the timing control unit 214 . That is, the light emission control unit 308 supplies a light emission control signal including information on light emission timing to the projection unit 100 via the projector output interface 306 .

The image input interface 310 inputs image data captured by the imaging unit 200 and supplies the data to the second signal processing unit 312 . The second signal processing unit 312 performs RGB normalization and binarization processing of the image data captured by the imaging unit 200 . Details of these processes will be described later.

The image generation unit 314 generates normal RGB color captured image data based on the RGB-normalized image data. The image generator 314 then supplies the captured image data to the data output interface 318 . In this case, the image generator 314 can also perform gamma conversion, frequency processing, noise reduction processing, and the like on the captured image data.

The three-dimensional measurement calculation unit 316 uses the binarized image data to obtain the projection angle α of the first color pattern and the second color pattern and β at the corresponding location in the image captured by the imaging unit 200. do. Then, the three-dimensional measurement calculation unit 316 calculates the distance d of each point based on the formula (1), generates three-dimensional measurement data, and supplies the data output interface 318 with the three-dimensional measurement data. In this way, the three-dimensional measurement calculation unit 316 detects patterns from digital images obtained by imaging the first color pattern and the second color pattern, and detects the detected patterns, the first color pattern projected by the projection unit 100, and the second color pattern. At least one of the patterns is associated to generate the distance to each portion of the measurement object T100. Note that the three-dimensional measurement calculation unit according to this embodiment corresponds to the measurement unit.

The data output interface 318 supplies captured image data and three-dimensional measurement data supplied from the image generation unit 314 to a storage unit (storage), a screen, and the like. The three-dimensional measurement data means the distance to each part of the measurement target T100.

Here, examples of the first color pattern and the second color pattern generated by the pattern generation unit 304 will be described in detail with reference to FIGS. 5, 6A, and 6B. FIG. 5 is a diagram schematically showing a color arrangement example of the first color pattern and the second color pattern. As described above, the first color pattern and the second color pattern are sequentially repeated and projected onto the measurement target T100 by the projection unit 100. FIG. In FIG. 5, the first color pattern and the second color pattern are shown on separate lines, but the first color pattern and the second color pattern are projected in order to the same position. For example, after the projection of the stripe designated C1 in the first color pattern, the stripe designated C6 in the second color pattern is projected at the same position as the stripe C1. C1 to C6 correspond to color codes (see FIG. 6B), which will be described later.

FIG. 6A shows a combination of red (R: red) light, green (G: green) light, and blue (B: blue) light of the first color pattern (patter1) and the second color pattern (patter2). is a table showing FIG. 6B is a table showing color code examples. For example, color number C1 is red, color number C2 is pink, color number C3 is green, color number C4 is yellow, color number C5 is corresponds to blue, and color number C6 corresponds to sky blue.

As shown in FIG. 6A again, the stripe number (color bar number) in FIG. 6A corresponds to the position of each stripe in FIG. Projected colors (colors) indicate colors corresponding to the color codes (C1 to C6) in FIG. 6B.

Also, R/G/B indicates a combination of red (R: red) light, green (G: green) light, and blue (B: blue) light. For example, red (red) light is indicated by 1/0/0 and consists of red (R:red) light. Similarly, pink light is indicated by 1/0/1 and is composed of red (R) light and blue (B) light. Similarly, green light is indicated by 0/1/0 and is composed of green (G) light. Similarly, yellow light is indicated by 1/1/0 and is composed of red (R) light and green (G) light. Similarly, blue light is indicated by 0/0/1 and is composed of blue (B) light. Similarly, sky blue light is indicated by 0/1/1 and is composed of green (G) light and blue (B) light.

That is, after the red light of color number C1 in which R/G/B is 1/0/0 in the first color pattern, R/G/B in the second color pattern is 0/1/1. A sky blue color with color number C6 shown is illuminated. In this way, the light projected onto the same projection area of the first color pattern and the second color pattern includes light of three wavelength bands (red (R) light and green (G) light). , blue (B) light), and for the same projection area, in the first color pattern, light of three wavelength bands (red (R) light, green (G) light, and , blue (B) light) at least one wavelength band (for example, red) is projected, and the second color pattern projects light of three wavelength bands (red (R: red) light and green ( G (green) light and light in the remaining wavelength band in blue (B) light (eg, green (G) light and blue (B) light) are projected. In other words, light composed of 1/0/0 [R/G/B] and light composed of exclusive 0/1/1 [R/G/B] are projected in order. Here, exclusive means 1/0/0 [R/G/B] constituent colors in the first color pattern and 0/1/1 [R/G/B] constituent colors in the second color pattern. In the present embodiment, the relationship of 1/1/1 [R/G/B] when added to the color is referred to as exclusive or complementary.

Here, the exclusive combination will be explained more specifically. As shown in FIG. 4, among the RGB pixels shown in the image pickup pixel P206, the R pixel is red light of color number C1 composed of 1/0/0 [R/G/B] in the first color pattern. When is projected, it reacts, that is, photoelectrically converted, but in the second color pattern, sky blue light of color number C6 composed of 0/1/1 [R/G/B] is projected. photoelectric conversion does not occur.

On the other hand, in the first color pattern, the G pixel does not perform photoelectric conversion when red light having a color number C1 composed of 1/0/0 [R/G/B] is projected. In the color pattern, photoelectric conversion is performed when sky blue light of color number C6 composed of 0/1/1 [R/G/B] is projected. Similarly, in the first color pattern, the B pixel does not perform photoelectric conversion when red light having a color number C1 composed of 1/0/0 [R/G/B] is projected. In the color pattern, photoelectric conversion is performed when sky blue light of color number C6 composed of 0/1/1 [R/G/B] is projected.

That is, in the first color pattern, R pixels are photoelectrically converted, but the remaining G and B pixels are not photoelectrically converted. On the other hand, in the second color pattern, R pixels are not photoelectrically converted, but the remaining G and B pixels are photoelectrically converted. Thus, in the exclusive relationship, the R/G/B pixels to be photoelectrically converted are exclusively exchanged between the first color pattern and the second color pattern. As can be seen, each of the red, green, and blue pixels in imaging pixel P206 (see FIG. 4) provides more pixel signals for either the first color pattern or the second color pattern. Generate.

Similarly, among the RGB pixels indicated by the imaging pixel P206, the R pixel projects pink light of color number C2 composed of 1/0/1 [R/G/B] in the first color pattern. However, in the second color pattern, when green light of color number C3 composed of 0/1/0 [R/G/B] is projected, photoelectric conversion is not performed. On the other hand, the G pixel does not perform photoelectric conversion when pink light having a color number C2 composed of 1/0/1 [R/G/B] is projected, but in the second color pattern, 0 /1/0 [R/G/B], and photoelectric conversion is performed when green light of color number C3 is projected. On the other hand, the B pixel photoelectrically converts when pink light of color number C2 composed of 1/0/1 [R/G/B] is projected, but in the second color pattern, 0 /1/0 [R/G/B], when green light with color number C3 is projected, photoelectric conversion is not performed.

In this way, in each projection area corresponding to stripe numbers (Color bar No.) 1 to 6, the first color pattern and the second color pattern are (Color No. C1, Color No. C6), ( Color No. C2, Color No. C3), (Color No. C3, Color No. C2), (Color No. C4, Color No. C5), (Color No. C5, Color No. C4), (Color No. .C6, Color No. C1), the exclusively combined projection light is projected. Note that the combinations are not limited to these combinations as long as they are exclusive.

FIG. 7 is a diagram schematically showing a projection example of the first color pattern. The observation point P is the part where the color changes. As shown in FIG. 7, the pattern generator 304 assigns IDs R1, R2, R3, R4, . Similarly, the pattern generator 304 assigns IDs such as pi1 to pin, G1 to Gn, Y1 to Yn, B1 to Bn, sb1 to sbn, etc. to the color numbers C2 to C6.

As shown in FIG. 7, since the line of interest on the object is closer to R4 than to R3, it is possible to easily identify the line of interest even if the imaging unit 200 captures an image. It can be seen that R4 is correct even in the projection view T100c from above. On the other hand, with black and white stripes that have been generally used, it is difficult to identify which line is the line of interest. By using color stripes in this way, it is possible to further improve the measurement accuracy in the convex regions T100a and the like.

Here, based on FIGS. 8 and 9, the processing effect for the region T100a having a color tone that does not reflect the specific light will be described. FIG. 8 is a diagram showing an example of projection onto a region T100a where the specific light is not reflected. FIG. 8A shows a front view and a top view of the convex region T100a of the measurement object T100. The convex region T100a has a yellow color. Therefore, blue (B) light is absorbed by the yellow of the convex region T100a and is hardly reflected.

FIG. 8B shows a state in which the first color pattern is projected onto the convex region T100a of the measurement object T100. FIG. 8C shows a state in which the second color pattern is projected onto the convex region T100a of the measurement target T100. Areas B1 and B2 that are imaged as black in the convex area T100a are also shown.

FIG. 9 is a table showing the reaction of each pixel RGB as imaging R/G/B in addition to the table shown in FIG. 6A. As shown in FIG. 9, blue (B) light is absorbed by the yellow of the convex region T100a and is hardly reflected. For this reason, the reaction of the B pixel indicated by imaging R/G/B is shown as 0. In particular, with blue (B: blue) light of color number C5 (Color No. 5) composed of projection 0/0/1 [R/G/B], imaging 0/0/0 [R/G/B] ], resulting in a state in which almost no optical signal is obtained from each of the RGB pixels. For this reason, the imaging pixel P206 (see FIG. 4) indicated by imaging 0/0/0 [R/G/B] is imaged as a black area B1 (see FIG. 8).

On the other hand, as described above, the first color pattern and the second color pattern are exclusive so that (Color No. C4, Color No. C5), (Color No. C5, Color No. C4) are combined objectively (or complementary). Therefore, yellow light of color number C4 (Color No. C4) composed of 1/1/0 [R/G/B] has a reflection component.

FIG. 10 is a table in the case of projecting only the first color pattern as a comparative example. As shown in FIG. 10, when only the first color pattern is projected, blue (B: With blue) light, the imaging becomes 0/0/0 [R/G/B], and almost no optical signal is obtained from each RGB pixel. For this reason, it becomes impossible to distinguish whether the object is a black area or a yellow area in the area where the blue (B: blue) light of stripe number 5 is projected. On the other hand, as shown in FIG. 9, when the first color pattern and the second color pattern that are exclusively combined are projected, even if the convex region T100a is colored, the first color pattern , or the second color pattern. Therefore, in the case of projecting the first color pattern and the second color pattern that are exclusively combined, the object is either a black area (or the object is at infinity) or a yellow area (a specific color is absorbed). It is possible to determine whether the

In this way, the light projected onto the same projection area for the first color pattern and the second color pattern are lights of different wavelength bands. Thus, a color pattern can be configured such that even if light in one wavelength band is absorbed by the measurement target T100, light in the other wavelength band is reflected. In this embodiment, the projection light is a combination of red (R: red) light, green (G: green) light, and blue (B: blue) light, but the present invention is not limited to this. For example, the light projected onto the same projection area for the first color pattern and the second color pattern may be light of different wavelength bands. That is, the light projected onto the same projection area of the first color pattern and the second color pattern is complementary so that even if the light of one wavelength band is absorbed, the light of the other wavelength band is reflected. should be configured to In this case, the wavelength band may be divided into at least two wavelength bands, and the pixels forming the imaging unit 200 (see FIG. 2) may be configured to have sensitivity only in each wavelength band.

FIG. 11 is a diagram showing a configuration example of the imaging pixel P206. As shown in FIG. 11, each pixel forming the imaging pixel P206 has a photodiode PD, transistors SW1, SW2, SW3, and AMP. The control wires consist of three control lines RST, TRG and SEL.

The photodiode PD has one end connected to the ground GND and the other end connected to one end of the transistor SW1. The other end of this transistor SW1 is connected to the floating diffusion FD. One end of the transistor SW2 is connected to the floating diffusion FD, and the other end is connected to one end of the transistor AMP. One end of the transistor AMP is connected to the VSS line, and the other end is connected to one end of the transistor SW3. The other end of transistor SW3 is connected to the VSL line. The gate of the transistor SW1 is connected to the control line TRG, the gate of the transistor SW2 is connected to the control line RST, the gate of the transistor SW3 is connected to the control line SEL, and the gate of the transistor AMP is connected to the floating diffusion FD. Note that, hereinafter, the photodiode PD may be simply referred to as PD, and the floating diffusion FD may be simply referred to as FD.

FIG. 12 is a block diagram showing an example of an ADC circuit forming the AD conversion section 208. As shown in FIG. As shown in FIG. 12, the ADC circuit (see FIG. 4) has multiple AZ circuits, comparators, and counters. The DAC circuit has an internal counter (not shown), and outputs a voltage value corresponding to the counter value while decrementing.

The AZ circuit adjusts the voltage of the input signal to the AZ reference voltage during the period (not shown) indicated by the auto-zero circuitry, and holds the voltage offset between the input and the output. This AZ circuit can maintain the voltage offset beyond the indicated period. This ADC circuit has two AZ circuits for the input from the DAC circuit and the input from the VSL line.The AZ circuits output the voltage offset to the comparator.

FIG. 13 is a schematic diagram of differential exposure processing. An example of differential exposure processing will be described with reference to FIG. As shown in FIG. 13, the image Im130 indicates the driving timing of the image sensor 204. As shown in FIG. The horizontal axis indicates time, and the vertical axis indicates the vertical address of the pixel array 206 (see FIG. 4). The image sensor 204 sequentially processes each vertical address. That is, the oblique graph shows how the vertical address is incremented and processed.

An image Im132 shows a flash strobe signal, which is a control signal for the timing of projector light emission. An image Im134 indicates generation and update timings of the first color pattern and the second color pattern, respectively. An image Im136 indicates the timing of projector light emission. Image Im138 shows the projection timings of the first color pattern and the second color pattern.

As shown in the image Im130, in the pixel array 206 (see FIG. 4), pixel reset (shutter), FD transfer, readout "read", and AD conversion are sequentially performed for each row. The left line L130 indicates the pixel reset "shutter" timing, the center line L132 indicates the "FD transfer" timing, and the right line L134 indicates the "read" timing. Below, pixel reset may be described as a shutter.

FD transfer is performed at the central timing of pixel reset (shutter) and readout "read". As a result, the FD transfer from the pixel reset (shutter) and the reading "read" from the FD transfer form a congruent parallelogram. This indicates that the exposure time from pixel reset (shutter) to FD transfer is equal to the exposure time from FD transfer to readout "read". The exposure from pixel reset (shutter) to FD transfer is defined as the first frame, and the exposure from FD transfer to readout "read" is defined as the second frame. That is, the first frame corresponds to imaging with the first color pattern, and the second frame corresponds to imaging with the second color pattern.

A timing generation unit 214 (see FIG. 3) in the image sensor 204 generates a pixel reset (shutter) startup timing, an FD transfer startup timing, a read startup timing, and a flash strobe timing that is a projector emission timing. . A control signal including information on pixel reset (shutter) activation timing, FD transfer activation timing, read activation timing, and projector light emission timing is supplied to the pixel control unit 216 and controlled as a trigger for pixel control. A flash strobe signal, which is a control signal for projector light emission timing, is sent from the timing output I/F 212 of the image sensor in the image sensor 204 to the pattern generation unit 304 via the timing input I/F 302 of the information processing unit 300. It is supplied to the light emission control unit 308 .

In order to synchronize the first color pattern and the second color pattern projected from the projection unit 100 with the photographing operation of the image sensor 204, a flash strobe signal, which is a control signal for the timing of projector light emission, is received from the image sensor 204. to be issued. The flash emission period is the timing at which the pixel reset (shutter), FD transfer, and readout "read" operations are stopped in each of the two exposures, and the lengths of the two flash emission (Flash strobe) periods are made equal.

Since it takes time to update the screen in the projection unit 100, the first color pattern and the second color pattern are updated during the period in which the light source does not emit light. Screen update is triggered by the timing at which the flash strobe signal received from the imaging unit 200 changes from high to low. The screen of the first color pattern (Pattern 1) is displayed in advance, and the screen of the second color pattern (Pattern 2) is updated at the changing point of the first flash strobe signal from High to Low. . The screen is updated with the first color pattern (Pattern 1) at the point where the second flash strobe signal changes from High to Low, and this is repeated thereafter. In this manner, the projection unit 100 emits light by controlling the light source of the projection unit 100 . A light emitting diode (LED), a halogen lamp, a xenon flash, or the like can be used as the light source.

Returning to FIG. 11 again, each pixel of the imaging pixel P206 sequentially accumulates signal charges corresponding to the first color pattern and signal charges corresponding to the second color pattern, and supplies them to the VSL line. In other words, these pixels sequentially drive the shutter "shutter", "FD transfer", and readout "read".

Here, the timing of the shutter "shutter" will be described based on FIG. 14 while referring to FIG. FIG. 14 is a diagram illustrating an example of a shutter timing chart. Signal levels of the control lines RST, TRG, and SEL are shown in order from the top. When the control line TRG becomes high (High), the transistor SW1 becomes conductive, and the charge of the photodiode PD is conducted to the FD. At this time, by setting the control line RST to High, the transistor SW2 is brought into a conductive state, and the charge of the floating diffusion FD is conducted to the VSS line. By this control, the charges in the photodiode PD and the floating diffusion FD flow to the VSS line and the charges are cleared. Then, by setting the control lines TRG and RST to low again, the transistors SW1 and SW2 are cut off and the charge of the photodiode PD does not flow. . Below, interruption may be described as a non-conducting state.

FIG. 15 is a diagram showing an example of a transfer timing chart of the floating diffusion FD. Signal levels of the control lines RST, TRG, and SEL are shown in order from the top. By setting the control line TRG to high, the transistor SW1 becomes conductive, and the charge of the photodiode PD is transferred and held in the floating diffusion FD. After that, the control line TRG is set to low to cut off the transistor SW1, so that the charges accumulated by the exposure of the photodiode PD are no longer transferred to the FD. As a result, the charge of the photodiode PD storing the charge of the exposure of the first frame corresponding to the first color pattern is moved to the floating diffusion FD and held therein.

FIG. 16 is a diagram showing an example of a read timing chart. Signal levels of the control lines RST, TRG, and SEL are shown in order from the top. As shown in FIG. 14, in read control, auto zero "Auto zero", FD reset, FD transfer, and AD conversion are continuously performed.

In auto-zero, the transistor SW3 is brought into conduction by setting the control line SEL to High. As a result, the voltage of FD is amplified by the amplifying action of transistor AMP and conducted to VSL. As a result, a voltage corresponding to the charge amount of the first frame exposure accumulated in the FD is conducted to the VSL line.

Subsequently, the AD converter 208 records and holds the offset between the potential of the VSL line and the AZ reference voltage by means of the AZ circuit mechanism. By setting the control SEL low again, the transistor SW3 is rendered non-conducting, and the change in the charge amount of the FD is controlled not to be conducted to the VSL line.

Subsequently, the FD reset makes the transistor SW2 conductive by setting the control line RST to High. As a result, the charges in the FD are conducted to the Vss line for resetting. By resetting FD and setting the control line RST to low again, the transistor SW2 is rendered non-conductive.

Subsequently, in the FD transfer, the transistor SW1 is made conductive by setting the control line TRG to High. This transfers the charge of the photodiode PD to the FD. Next, the charge of the photodiode PD exposed and accumulated in the second frame, that is, the second color pattern is transferred to the FD and held.

Subsequently, in the AD conversion, the control line SEL is set to High to turn on the transistor SW3. As a result, the voltage of the FD is amplified by the amplifying action of the transistor AMP, and the voltage is conducted to the VSL line. In this way, it is possible to supply the information of the amount of charge accumulated in exposure for the second frame via the VSL line. The AD converter 208 AD-converts the voltage value conducted to the VSL line. As a result, the difference between the voltage proportional to the amount of accumulated exposure charge in the first frame corresponding to the first color pattern and the voltage proportional to the amount of accumulated exposure charge in the second frame corresponding to the second color pattern is AD converted. After the AD conversion is completed, SEL is set to low to bring the transistor SW3 into a non-conducting state.

17 and 18 are diagrams showing waveforms of VSL (after passing through AZ) L170 and DAC (after passing through AZ) of the comparator (FIG. 12) input when the first color pattern is presented. FIG. 17 is a diagram showing a case where pixels have sensitivity to the first color pattern, that is, a case of photoelectric conversion.

As described above, the first color pattern and the second color pattern are configured exclusively, so photoelectric conversion is performed in only one of the first color pattern and the second color pattern in a specific pixel. As indicated by VSL (after passing through AZ) L170 in FIG. 17, charges have negative energy, so the more charges, the larger the negative voltage. In FIG. 17, the voltage in the downward direction becomes negative and the energy increases.

AD conversion in which there is effective light in the first color pattern is performed during readout "read" (see FIG. 13) control. The control states in FIG. 17 correspond to the read “read” timing chart in FIG. In the auto-zero (AZ) state, the voltage of charge accumulated in FD appears at VSL (after passing through AZ) L170. Therefore, since there is a value corresponding to the accumulated charge in the first color pattern imaged in the first frame, the voltage of VSL (after passing through AZ) L170 greatly swings downward. When the voltage of VSL (after passing through AZ) L170 is stabilized, the auto-zero AZ circuit (Fig. 12) is enabled, and the voltage of VSL (after passing through AZ) L170 and the voltage of DAC (after passing through AZ) are set to the AZ reference voltage. Align and use this as a reference point. The AZ circuit (FIG. 12) then maintains this offset value and continues to output the output with the offset value added to VSL (after passing through AZ) L170 and DAC (after passing through AZ). In the FD reset and FD transfer states, VSL is in high impedance (Hi-Z), so it is maintained in a state where it does not change significantly.

In the subsequent AD conversion state, the value of the second frame is read. In the second frame, light corresponding to the second color pattern is not imaged, and exposure due to reflection of ambient light is performed. Therefore, the VSL (after passing through the AZ) voltage L180 becomes smaller than the voltage of the VSL line in the first frame and swings upward greatly. AD conversion is performed after the VSL (after passing through AZ) voltage L170 is stabilized. This shows single-slope AD conversion that linearly changes the output of the DAC circuit (see FIG. 12) from higher voltage to lower voltage. The ADC counter increments at regular intervals from 0 at the slope start of the DAC. At the intersection of the VSL (after AZ) voltage L170 and the DAC voltage, the comparator flips and the counter stops counting. The value of this counter becomes digital data and is supplied to the signal processing unit 210 . The slopes of the counter and the DAC circuit are synchronized, and the count value of the AZ reference voltage is constant and preset. Therefore, by subtracting the counter value of the AZ reference voltage from the count value when the comparator is inverted, the AD converter result can be obtained as a positive or negative value. As shown in FIG. 17, if the pixel is sensitive to the first color pattern (pattern 1), a negative value will result.

FIG. 18 is a diagram showing the case where the pixels have sensitivity to the second color pattern, that is, the case of photoelectric conversion. The AD conversion method is the same procedure as in FIG. 17 described above. The amount of light transmitted to the photoelectric conversion element of a specific pixel is large (dark) in the first frame and small (bright) in the second frame. The result of AD conversion that swings downward is a positive value. In this way, a specific pixel has a negative value when it is sensitive to the first color pattern (pattern1), and a positive value when it is sensitive to the second color pattern (pattern2).

As can be seen from these, by comprehensively using the results of the R/G/B pixels (see FIG. 4), the color codes of the first color pattern (pattern1) and the second color pattern (pattern2) can be obtained. It is possible to make the determination by one AD conversion. In addition, the combination of the first color pattern (pattern1) and the second color pattern (pattern2) and the image sensor 204 that performs differential AD conversion reduce the influence of ambient light that becomes noise, and the two colors are combined. It is possible to acquire patterns at the same time.

Referring to FIG. 13 again, the period of the first frame including the first color pattern (pattern1) and the period of the second frame including the second color pattern (pattern2) are simultaneously exposed to reflected light from ambient light. These become noise in the measurement. For this reason, a darkroom-like environment is created in order to suppress the light that becomes noise during distance measurement. The degree of freedom in the measurement environment increases.

Suppression of ambient light reflection according to this embodiment will be described in more detail below. Here, let N1 (noise) be the exposure of the reflected ambient light in the first frame period, and N2 (noise) be the exposure of the reflected ambient light in the second frame period. Assuming that the color of the first color pattern (pattern 1) is P1C and the color of the second color pattern is P2C, as shown in equation (2), P1C+N1 is exposed in the first frame and P2C+N2 is exposed in the second frame. .

Since it can be assumed that the scene to be measured is a stationary object and there is no change in ambient light, N1=N2. Therefore, by performing differential AD conversion, the exposure of the first frame is subtracted from the exposure of the second frame. becomes possible.

In this way, the exposure components N1 and N2 of the ambient light reflection disappear, and only the information of the exposure components of the color patterns P1C and P2C is extracted. Further, when the color patterns of this embodiment are separated into color components, either the first color pattern (pattern 1) or the second color pattern (pattern 2) in each color is 0, which does not emit light. For this reason, the image sensor 204 according to the present embodiment separates colors using color filters that pass any one of R, G, and B as described above. Only the values corresponding to the emission of either the first color pattern (pattern 1) or the second color pattern (pattern 2) are present. For example, as described above, when Color bar No. is number 1, the P1C color components (R, G, B) are (1, 0, 0), and the P2C color components (R, G, B) are ( 0, 1, 1). Solving this for each color component using Equation 2 yields (-1, 1, 1). Here, since P1C is a negative value, -1 is replaced with 1 and positive values are replaced with 0, resulting in (1, 0, 0), and since P2C is a positive value, 1 is replaced with 1 and a negative value is replaced with 0, it becomes (0, 1, 1), and it becomes possible to distinguish between the two colors of Color bar No.

In addition, in the case of normal operation, AD conversion can be performed twice in the first period and the second period, and arithmetic subtraction processing can be performed. However, AD conversion consumes a lot of power, the amount of data is large, and arithmetic processing also consumes power. On the other hand, in this embodiment, power consumption, data amount, and arithmetic processing can be reduced by performing AD conversion at once. In addition, in this embodiment, the difference is obtained by the analog circuit, but the corresponding processing may be performed by digital calculation.

FIG. 19 is a table showing luminance when the color pattern shown in FIG. 6A is projected onto an object with reflectance (R: 50%, G: 100%, B: 75%). That is, it represents the brightness of the surface when projected onto an object whose reflectance is R (red) 50%, G (green) 100%, and B (blue) 75%. Here, the value is a resolution of 0-255.

FIG. 20 is a table showing the result of performing differential AD conversion by projecting the color pattern shown in FIG. 6A onto an object with reflectance (R: 50%, G: 100%, B: 75%). Here, the value is a resolution of -255 to 255. The first color pattern (pattern 1) has a negative numerical value, and the second color pattern (pattern 2) has a positive numerical value. The values of (R, G, B) with resolution in the range of -255 to 255 are supplied to the second signal processing unit 312 as an image signal.

The second signal processing unit 312 sets the result to 1 when the value AD after the differential AD conversion is larger than the threshold value TH (assumed to be a positive number), that is, when AD>TH, and sets the value to the threshold value TH. If it is smaller than the result of multiplication by -1, that is, if AD<-TH, the result is -1. The result is 0 when the value AD satisfies -TH≤AD≤TH. The influence of noise can be excluded by setting the threshold value TH to a numerical value that takes noise into account. FIG. 21 shows the results when TH=10, although it depends on the capability of the device. In other words, in normal shooting conditions, the reflectance and the distance to the object are unknown values. A portion of is determined to be significant light.

FIG. 21 is a table showing the results of conversion into significant color components. That is, it is a table showing an example of the results of conversion into significant color components by the second signal processing unit 312 based on the values after minute AD conversion.

Also, FIG. 22 is a table showing the result when the pattern shown in FIG. 6A is subjected to differential AD conversion according to this embodiment. After the theoretical difference AD conversion, the calculation result obtained by subtracting the first color pattern (pattern1) from the second color pattern (pattern2) according to the equation (2) is shown. As can be seen from the results of FIGS. 21 and 22, the row values after theoretical difference AD conversion match the values generated by the second signal processing unit 312 . In this way, the pattern can be measured by the measurement by the image sensor in consideration of the reflectance.

FIG. 23 is a table showing the results of measuring an object that does not reflect blue light. As shown in FIG. 23, when 0 is included, it is possible to specify the color without reflection of the measurement object, thereby narrowing down the range of color stripe numbers (Color bar No.). For example, if the measurement shows {R, G, B}={−1, 1, 0}, then the object has no reflection of B (blue), which is the combination of color stripes of projected light ( Color bar combination) (see FIG. 19) is number 1 or number 2.

Based on this, the second signal processing unit 312 can estimate and complement the surrounding conditions of this measurement point, for example, the color result of the left and right neighbors. In the second signal processing unit 312, the left neighbor of Color bar No. 1 is No. 6, but the left neighbor of No. 2 is No. 1, and the right neighbor of No. 2 is No. 3, but 1 Complementary guessing can be performed using features such as the fact that the number to the right of the number is the number 2. For example, the second signal processing unit 312 detects all the detected change points, such as the boundary between No. 1 and No. 2, the boundary between No. 2 and No. 3, etc., because the measurement points are the color change points in the case of stripes. Image position and which Color bar No. It is possible to send information on the boundary between the two as distance measurement data to the three-dimensional measurement calculation unit 316 (see FIG. 3). The three-dimensional measurement calculation unit 316 calculates the angle between the imaging unit 200 and the measurement point from the image position, and also calculates the color bar number. The angle between the projection unit 100 and the measurement point is calculated from the information, and the distance to the measurement point is calculated by triangulation from three pieces of information, namely, the predetermined distance L between the projection unit 100 and the imaging unit 200 . The three-dimensional measurement calculation unit 316 calculates distances at all measurement points P, and stores the data in storage or displays the data on a display via the data output I/F 318 .

On the other hand, as shown in the comparative example (see FIG. 10), the result for that color is 0 when there is no reflection. For example, when {R, G, B}={1, 0, 0}, in the conventional example, whether the object to be measured reflects all colors, does not reflect G (green), or reflects B (blue). ) becomes indistinguishable from the absence of reflection. Therefore, the possibility of No. 1, the possibility of No. 2 with B (blue) missing, and the possibility of No. 4 with G (green) missing remain. For this reason, the accuracy of information when estimating from the measurement results of the surroundings is inferior to that of the present embodiment. Also, when the light is reflected and does not return, as in No. 5, the shape of the object cannot be confirmed, resulting in a measurement result in which the object appears to be missing.

In this way, since only one color pattern is projected for distance measurement in the comparative example, it is difficult to identify the color of the object. On the other hand, in this embodiment, since the three primary colors of R, G, and B are irradiated with two color patterns, it is possible to reproduce the color of the object to be measured.

Referring to FIG. 20 again, FIG. 20 shows the brightness of each color reflected by illuminating an object with reflectance of R: 50%, G: 100%, and B: 75%. Numerical values are normalized expressions in the range from 0 to 255. Thus, 50% is 127, 100% is 255, and 75% is 191. In order to perform color reproduction using this measurement result, it is possible to obtain the color of the measured object by converting the luminance of each color into an absolute value. For example, the second signal processing unit 312 (see FIG. 3) takes {−127, 255, 191} for Color bar No. 1 in FIG. , 255, 191}.

On the other hand, consider the case where white light is applied to the measurement object. The white color components are {255, 255, 255} and the reflectance of the measurement object is {50%, 100%, 75%}. When these are multiplied for each color component and truncated after the decimal point, the colors {127, 255, 191} are obtained. This matches the value calculated from Color bar No. 1 in FIG. The second signal processing unit 312 (see FIG. 3) and the image generating unit 314 (see FIG. 3) perform these series of transformations on all pixels obtained by the image sensor 204, and the image generating unit 314 (see FIG. 3) (See Fig. 3). Then, the image generation unit 314 generates two-dimensional RGB image data based on digital pixel signals for each imaging pixel P206 such as RGB {127, 255, 191}, and outputs the data to the storage via the data output I/F 318. record or display.

FIG. 24 is a table showing color stripe patterns (color bar numbers) generated by de Bruijn. As shown in FIG. 24, the color pattern according to the present embodiment can be constructed even with a de Brown color arrangement or a matrix tile pattern. FIG. 25 is a diagram showing an example of extension of a cyclic pattern by the de Brownian sequence algorithm. As shown in FIGS. 24 and 25, for example, in the first color pattern (pattern1), there is only one pattern with red on the left and green on the right. Only one combination is possible.

Also here, as in the above-described embodiment, each color component may be configured to emit light or not. That is, it is created by an exclusive combination. Therefore, the second color pattern (pattern2) is composed of color components not used in the first color pattern (pattern1). In the second color pattern (pattern2) as well, every combination of two adjacent colors is unique. These become a cycle of 30 colors, and the rightmost color, for example, sky blue of the first color pattern (pattern 1) becomes red, and the cycle of 30 colors is repeated again. Colors connected by repetition also become unique combinations. The above processing is also possible with the color stripe pattern shown in FIG. However, looking at the combination of left neighbors or right neighbors, the number (color bar no) of the color stripe pattern that matches the improved color stripe pattern is searched and identified by the de Bruijn column in FIG. Similarly, when there is no reflection of a specific color component, it is possible to perform complementation with a combination of adjacent colors. In addition, since the number increases compared to the circulation of six colors, it is possible to widen the diopter difference while maintaining the measurement granularity, or to increase the measurement granularity by narrowing the width of each color. It becomes possible.

FIG. 26 is a table showing color tile patterns in which color changes are expanded in the column direction. Color Tile No. FIG. 27 is a diagram showing the arrangement of the color tile pattern example of FIG. The numbers in FIG. 27 correspond to color tile numbers.

As shown in FIGS. 26 and 27, it is possible to leave the order of the colors in the horizontal direction unchanged and arrange them in the same order in the vertical direction. Such patterns are arranged as one unit in the horizontal and vertical directions. Here, according to the present embodiment, each color component is created by a combination of light emission and non-light emission, and the second color pattern (pattern2) is composed of the color components not used in the first color pattern (pattern1). All combinations of two adjacent colors are unique in the second color pattern (pattern 2) as well. Color tile patterns can also be processed by the method already described. However, in cases such as when there is no reflection of a specific color component, in addition to estimation based on the left or right adjacent color, estimation based on the upper or lower color can be used. As a result, when there is an obstacle such as lack of reflection, the information for the complementary calculation increases, so the accuracy of the complementary operation is further improved. Also, the color tile pattern may be de Brown's method or another circulating pattern.

FIG. 28 is a diagram showing a timing chart for reading the counter difference and the digital difference. FIG. 28 is a diagram illustrating an example of performing AD conversion twice. Signal levels of the control lines RST, TRG, and SEL are shown in order from the top. As shown in FIG. 28, in read control, auto zero "Auto zero", AD conversion, FD reset, FD transfer, and AD conversion are performed continuously. Note that the control of the shutter state and FD transfer state is the same as in the examples of FIGS. On the other hand, the read state control is different. As shown in FIG. 28, auto zero "Auto zero (2)", AD conversion (2), FD reset, FD transfer, and AD conversion (3) are performed in succession.

In the auto zero "Auto zero (2)", the transistor SW11 of the pixel (see FIG. 11) is turned off, and the AD converter (see FIG. 4) of the AD conversion unit 208 changes the voltage of the VSL line and AZ Record and store the offset from the reference voltage. In the AD conversion (2) state, by setting the control line SEL to High, the transistor SW3 is rendered conductive, and the voltage of the FD is transmitted to the VSL line with a value amplified by the transistor AMP. As a result, a voltage corresponding to the amount of charge accumulated in the first frame exposure accumulated in the FD is conducted to the VSL line. The AD converter (see FIG. 4) performs the first AD conversion on the voltage of the VSL line. This means that the voltage proportional to the amount of charge accumulated in the first frame is AD-converted. By driving the control line SEL low again, the transistor SW3 is rendered non-conductive so that the change in the amount of charge on the FD does not affect the VSL line.

The FD reset is performed by setting the control line RST to High to bring the transistor SW2 into a conducting state and to flow the charge of the FD to the Vss line for resetting. After resetting the FD, the control line RST is set to low again to turn off the transistor SW2. In the FD transfer, the transistor SW1 is made conductive by setting the control line TRG to High to transfer the charge of the PD to the FD. As a result, the charges of the PD exposed and accumulated in the second frame are transferred to the FD and held.

In AD conversion (3), the control line SEL is set to High to bring the transistor SW3 into a conductive state. Thereby, the voltage of FD is amplified by the transistor AMP and conducted to the VSL line. This allows a voltage proportional to the amount of charge accumulated in the exposure in the second frame to be conducted through the VSL line. The AD converter performs AD conversion of this value. This means that the voltage proportional to the amount of charge accumulated in exposure in the second frame is AD-converted. After AD conversion is completed, the control line SEL is set to low to bring the transistor SW3 into a non-conducting state.

FIG. 29 is a diagram showing VSL (after passing through AZ) and DAC (after passing through AZ) waveforms of ADC comparator inputs corresponding to the timing chart of FIG. FIG. 29 shows a case where a specific pixel has sensitivity to the first color pattern. As indicated by VSL (after passing through AZ) L290 in FIG. 29, since the charge has negative energy, the more the charge, the larger the negative voltage. In FIG. 29, the voltage in the downward direction becomes negative and the energy increases.

In the auto-zero (AZ) state, the voltages of VSL (after passing through AZ) L290 and DAC (after passing through AZ) are aligned with the AZ reference voltage, and this is used as the reference point. The AZ circuit then maintains this offset value and continues to output the offset-added output to VSL (after passing through AZ) L290 and DAC (after passing through AZ). In the AD conversion (2) state, the voltage of the charge accumulated in FD appears on VSL. Therefore, since the accumulated charge in the first frame has an accumulated amount corresponding to the projection light, VSL (after passing through AZ) L290 swings downward greatly. VSL (after passing through AZ) When L290 stabilizes, perform the first AD conversion and count with the ADC counter that was cleared to 0 in advance.

It shows a single-slope AD conversion that linearly changes the DAC output from higher voltage to lower voltage. However, in this embodiment, the counter is decremented because the difference processing is performed by the counter. When the voltages of VSL (after passing through AZ) and DAC (after passing through AZ) intersect, the comparator is inverted and the ADC counter stops and holds the value.

In the FD reset (reset) and FD transfer states, the VSL line L290 is at high impedance (Hi-Z), so a state that does not change significantly is maintained. In the subsequent AD conversion state, the value of the second frame is read. In the second frame, there is no sensitivity to the second color pattern, and the exposure component of reflected ambient light is primarily converted to stored charge. Therefore, the voltage corresponding to the accumulated charge amount becomes smaller than the value of the first frame. In this way, it swings slightly downward with respect to the AZ reference voltage. As shown in FIG. 29, the large value in the first frame results in an upward swing. After the VSL (after passing through the AZ) L290 is stabilized, the second AD conversion is performed. The ADC counter increments from the value of the first AD conversion. At the intersection of the voltage on the VSL line and the DAC, the comparator flips and the counter stops counting. Since the first AD conversion counted in the negative direction, the value of this counter is the result of the difference between the two frames. is the value of

FIG. 30 is a diagram showing a case where specific pixels have sensitivity to the second color pattern. The AD conversion method is the same as the procedure shown in FIG. Since the 1st frame is dark and the 2nd frame is bright, the value of VSL (after passing through AZ) L300 in the AD conversion (2) state becomes small, swings slightly downward from the AZ reference voltage, and swings greatly downward in AD conversion (2). , the result is positive because the increment value is greater. 17 and 18, and subsequent processing can be performed in the same manner as in FIGS. In this embodiment, the ADC counter is incremented and decremented. Difference processing may be performed by inverting before . Further, AD conversion of the first frame may be performed in increments during AD conversion (2), and the value may be sent to the signal processing unit. After that, the ADC is cleared to 0 before the AD conversion (3), the AD conversion of the second frame is performed, the value is sent to the signal processing section, and the signal processing section executes the difference between the two in a digital circuit. may

As described above, according to the present embodiment, the projection unit 100 sequentially projects the first color pattern and the second color pattern onto the measurement object T100, and the information processing unit 300 projects the first color pattern, Based on the second color pattern, the distance to the measurement object T100 is measured. In this case, the light projected onto the same projection area of the first color pattern and the light of the second color pattern are light of different wavelength bands. Thus, a color pattern can be configured such that even if light in one wavelength band is absorbed by the measurement target T100, light in the other wavelength band is reflected.

Further, the light projected onto the same projection area of the first color pattern and the second color pattern is composed of a combination of light of three wavelength bands, and the first color pattern has three wavelength bands for the same projection area. In the second color pattern, light in at least one wavelength band is projected, and light in the remaining wavelength bands among the light in the three wavelength bands is projected in the second color pattern. As a result, even if any of the light in the three wavelength bands is absorbed by the measurement object T100, the light in the other wavelength bands is reflected, making it possible to measure the distance to the measurement object T100. In addition, it becomes possible to generate the reflection intensity of each of the light in the three wavelength bands as a pixel signal, and to generate the color information of the measurement object T100.

This technology can be configured as follows.

(1) a projection unit that sequentially projects a predetermined first color pattern and a predetermined second color pattern onto an object to be measured;
an information processing unit that measures the distance to the measurement object based on the first color pattern and the second color pattern;
with
The distance measuring device, wherein light projected onto the same projection area of the first color pattern and the light of the second color pattern are light of different wavelength bands.

(2) The light projected onto the same projection area of the first color pattern and the second color pattern is composed of a combination of light of three wavelength bands, and the first color pattern and the second color pattern are projected onto the same projection area. (1), wherein at least one of the three wavelength bands of light is projected in the pattern, and the remaining wavelength bands of the three wavelength bands of light are projected in the second color pattern; Range finder as described.

(3) The distance measuring device according to (2), wherein each of the light in the three wavelength bands is one of light in the red band, green band, and blue band.

(4) The distance measuring device according to (3), further comprising an imaging unit that sequentially images the object to be measured on which the first color pattern and the second color pattern are projected.

(5) The imaging unit
The distance measurement according to (4), further comprising a timing generation unit that generates a control signal for controlling projection timing of the first color pattern and the second color pattern in the projection unit and imaging timing of the imaging unit. Device.

(6) The imaging unit
A red pixel having sensitivity to light in the red band and having less sensitivity to green and blue bands than light in the red band, and a pixel having sensitivity to light in the green band and having sensitivity to red and blue bands Green pixels whose sensitivity is suppressed from light in the green band, and blue pixels which have sensitivity to light in the blue band and whose sensitivity to red and green bands is suppressed from light in the blue band. The distance measuring device according to (5), which has a pixel array in which imaging pixels are arranged two-dimensionally.

(7) Each of the red pixel, the green pixel, and the blue pixel in the imaging pixel generates more pixel signals for either the first color pattern or the second color pattern , (6).

(8) The imaging unit
A first accumulated charge amount accumulated according to the first color pattern and accumulated according to the second color pattern for each of the red pixel, the green pixel, and the blue pixel in the imaging pixel The distance measuring device according to (6), which converts a voltage corresponding to a difference between the accumulated charge amount and the accumulated charge amount into a digital pixel signal.

(9) Based on the control signal generated by the timing generator,
each of the red pixel, the green pixel, and the blue pixel in the imaging pixel is controlled such that the imaging time for the first color pattern and the imaging time for the second color pattern are equal; (6) A distance measuring device according to the above.

(10) The information processing unit
(6) to an image generation unit that generates a color image of the measurement target based on the output signal of the imaging pixel for the first color pattern and the output signal of the imaging pixel for the second color pattern; The distance measuring device according to any one of (9).

(11) The information processing unit
Based on the digital pixel signal, patterns are sequentially detected from a digital image of the first color pattern and the second color pattern, and the detected pattern, the first color pattern projected by the projection unit, and the second color pattern are detected. The distance measuring device according to (8), further comprising a measuring unit that corresponds to at least one of two color patterns and generates a distance to each part of the measurement object.

(12) The first color pattern and the second color pattern are composed of a plurality of partitioned regions, and the color of light in each partitioned region is the light of the red band, the green band, and the blue band. or any combination of two of the red band, the green band, and the blue band light. Device.

(13) The distance measuring device according to (12), wherein the plurality of partitioned areas are grouped, and the color of light for each partitioned area within the group is different.

(14) The distance measuring device according to (12), wherein in the plurality of partitioned areas, the combinations of two adjacent colors are all different.

(15) The distance measuring device according to (14), wherein the colors of the plurality of partitioned areas are cyclic patterns according to the De Brown sequence.

(16) The distance measuring device according to any one of (12) to (15), wherein the plurality of partitioned areas are tile patterns arranged two-dimensionally in a matrix.

(17) The imaging unit
For each of the red pixel, the green pixel, and the blue pixel in the imaging pixel, a voltage corresponding to a first accumulated charge amount accumulated according to the first color pattern is converted into a first digital signal. and converts a voltage corresponding to the amount of accumulated charge accumulated according to the second color pattern into a second digital signal, and uses the difference between them as a digital pixel signal.

(18) The information processing unit
(1) including a pattern generation unit that generates the first color pattern and the second color pattern, and supplying a signal having information of the first color pattern and the second color pattern to the projection unit; The distance measuring device according to any one of (17) to (17).

(19) The imaging unit
The infrared wavelength band is divided into three wavelength bands, and consists of pixels that are sensitive only to each wavelength band,
The distance measuring device according to (1), wherein the first color pattern and the second color pattern are configured in each of the three wavelength bands of the infrared region.

(20) The imaging unit
The distance measuring device according to (1), wherein the wavelength band is divided into at least two wavelength bands, and is composed of pixels having sensitivity only in each wavelength band.

(21) a projection step of sequentially projecting a predetermined first color pattern and a predetermined second color pattern onto a measurement object;
an information processing step of measuring a distance to the measurement target based on the first color pattern and the second color pattern;
with
The light projected onto the same projection area for the first color pattern and the second color pattern is different from the light projected onto the same projection area for the first color pattern and the second color pattern. A ranging method that uses light in a wavelength band.

Aspects of the present disclosure are not limited to the individual embodiments described above, but include various modifications that can be conceived by those skilled in the art, and the effects of the present disclosure are not limited to the above-described contents. That is, various additions, changes, and partial deletions are possible without departing from the conceptual idea and spirit of the present disclosure derived from the content defined in the claims and equivalents thereof.

1: distance measuring device, 100: projection unit, 200: imaging unit, 206: pixel array, 214: timing control unit, 300: information processing unit, 304: pattern generation unit, 316: three-dimensional measurement calculation unit, 314: image Generation unit, P206: imaging pixels.

Claims

a projection unit that sequentially projects a predetermined first color pattern and a predetermined second color pattern onto an object to be measured;
an information processing unit that measures the distance to the measurement object based on the first color pattern and the second color pattern;
with
The distance measuring device, wherein light projected onto the same projection area of the first color pattern and the light of the second color pattern are light of different wavelength bands.
The light projected onto the same projection area of the first color pattern and the second color pattern is composed of a combination of light of three wavelength bands. The measurement according to claim 1, wherein at least one of three wavelength bands of light is projected, and the remaining wavelength bands of the three wavelength bands of light are projected in the second color pattern. distance device.
The distance measuring device according to claim 1, wherein each of the three wavelength bands of light is one of red band, green band, and blue band light.
an imaging unit that sequentially images the measurement object onto which the first color pattern and the second color pattern are projected;
2. A ranging device according to claim 1, further comprising a ranging device.
The imaging unit is
5. The distance measuring device according to claim 4, further comprising a timing control section for controlling projection timing of said first color pattern and said second color pattern in said projection section and imaging timing of said imaging section.
The imaging unit is
A red pixel having sensitivity to light in the red band and having less sensitivity to green and blue bands than light in the red band, and a pixel having sensitivity to light in the green band and having sensitivity to red and blue bands Green pixels whose sensitivity is suppressed from light in the green band, and blue pixels which have sensitivity to light in the blue band and whose sensitivity to red and green bands is suppressed from light in the blue band. 6. The distance measuring device according to claim 5, having a pixel array in which imaging pixels are two-dimensionally arranged.
3. Each of said red pixel, said green pixel, and said blue pixel in said imaging pixel generates more pixel signals for either said first color pattern or said second color pattern. 7. The distance measuring device according to 6.
The imaging unit is
voltage corresponding to a first accumulated charge amount accumulated according to the first color pattern and the second color pattern for each of the red pixel, the green pixel, and the blue pixel in the imaging pixel; 8. The distance measuring device according to claim 7, wherein the digital pixel signal is converted into a digital pixel signal according to the difference between the voltage corresponding to the second accumulated charge amount accumulated according to .
Based on the control signal generated by the timing control unit,
each of the red pixel, the green pixel, and the blue pixel in the imaging pixel is controlled such that the imaging time for the first color pattern and the imaging time for the second color pattern are equal; The distance measuring device according to claim 6.
The information processing unit
7. The method according to claim 6, further comprising an image generation unit that generates a color image of the measurement object based on the output signal of the imaging pixel for the first color pattern and the output signal of the imaging pixel for the second color pattern. Range finder as described.
The information processing unit
Based on the digital pixel signal, patterns are sequentially detected from a digital image of the first color pattern and the second color pattern, and the detected pattern, the first color pattern projected by the projection unit, and the second color pattern are detected. 9. The distance measuring device according to claim 8, further comprising a measuring unit that associates at least one of two color patterns with the measuring unit and generates a distance to each portion of the measurement target.
The first color pattern and the second color pattern are composed of a plurality of partitioned regions, and the color of light in each partitioned region is any one of the red band, the green band, and the blue band. or a combination of the two of the red band, the green band and the blue band light.
13. The distance measuring device according to claim 12, wherein the plurality of partitioned areas are grouped, and the color of light for each partitioned area within the group is different.
13. The distance measuring device according to claim 12, wherein combinations of two adjacent colors in the plurality of partitioned areas are all different.
15. The distance measuring device according to claim 14, wherein the colors of the plurality of divided areas are cyclic patterns according to the De Brown sequence.
The distance measuring device according to claim 12, wherein the plurality of partitioned areas are tile patterns arranged two-dimensionally in a matrix.
The imaging unit is
For each of the red pixel, the green pixel, and the blue pixel in the imaging pixel, a voltage corresponding to a first accumulated charge amount accumulated according to the first color pattern is converted into a first digital signal. 7. The distance measuring device according to claim 6, wherein a voltage corresponding to a second accumulated charge amount accumulated according to said second color pattern is converted into a second digital signal, and a difference between them is used as a digital pixel signal. .
The information processing unit
2. The projector according to claim 1, further comprising a pattern generation section for generating said first color pattern and said second color pattern, and supplying a signal having information of said first color pattern and said second color pattern to said projection section. The distance measuring device according to .
The imaging unit is
The wavelength band of the infrared region is divided into three wavelength bands, and it is composed of pixels that are sensitive only to each wavelength band,
2. The distance measuring device according to claim 1, wherein said first color pattern and said second color pattern are respectively composed of said three wavelength bands in said infrared region.
The imaging unit is
2. The distance measuring device according to claim 1, wherein at least a wavelength band is divided into two wavelength bands and composed of pixels having sensitivity only in each wavelength band.
a projection step of sequentially projecting a predetermined first color pattern and a predetermined second color pattern onto a measurement target;
an information processing step of measuring a distance to the measurement target based on the first color pattern and the second color pattern;
with
Light projected onto the same projection area of the first color pattern and the second color pattern has different wavelengths from light projected onto the same projection area of the first color pattern and the second color pattern. A ranging method that is band light.