JP5365419B2 - 3D shape sensor - Google Patents

Description

  The present invention relates to a three-dimensional shape sensor based on depth sampling.

  As a method of acquiring three-dimensional shape information for forming a stereoscopic image in a three-dimensional shape sensor, for example, there are a stereo method, a TOF (Time Of Flight) method, a depth sampling method, and the like. The depth sampling method is a method of measuring the position of the subject from the focal length by measuring the sharpness of the image. The depth sampling method uses a lens system with a variable focal length or a microscope in which the lens system itself can easily move between the subject and the like.

  In an image sensor equipped with such an optical system, when the focal length of the lens is a certain value, or when the positional deviation of the lens is known from a certain reference plane in the microscope, the three-dimensional shape sensor is obtained by the image sensor. The obtained image is processed, its sharpness is calculated, and the weight information is mapped onto the two-dimensional coordinates of the original image. The roughness of the mapping varies, but in many cases, the field of view is divided into finite blocks and a weight is added to each block.

  The three-dimensional shape sensor determines that the portion where the mapped weight information is large is an object at a position corresponding to the focal length or lens movement distance at that time. By repeatedly performing this weighting on the same subject and the same field of view at various focal lengths or lens movement distances, the three-dimensional shape of the subject is grasped.

  There is a three-dimensional shape sensor including a liquid lens as a lens having a variable focal length. This liquid lens responds at high speed by changing the focal length linearly with respect to the input voltage.

  FIG. 10 shows a conventional three-dimensional shape sensor equipped with a liquid lens. In FIG. 10, a three-dimensional shape sensor 200 includes a liquid lens 201, a CCD (Charge Coupled Device) 202, a main control unit 207, a lens control unit 206, a CCD control unit 205, a signal processing unit 203, and a distance extraction unit. 204.

  Next, the operation of the three-dimensional shape sensor 200 shown in FIG. 10 will be described. First, when an instruction is given to the three-dimensional shape sensor 200 to measure a three-dimensional shape, the main control unit 207 changes the focal length of the liquid lens 201 to the lens control unit 206 by a predetermined method. To give instructions.

  Upon receiving an instruction from the main control unit 207, the lens control unit 206 generates a lens input voltage having a ramp waveform having a predetermined inclination with respect to time as shown in FIG. Output to the lens 201.

  On the other hand, the main control unit 207 instructs the CCD control unit 205 to take a predetermined number of times while the focal length of the liquid lens 201 is changing. For example, when n times of photographing are performed, the CCD control unit 205 outputs a drive signal for performing photographing to the CCD 202 as shown in FIG. At this time, photographing is performed at the rising timing of the drive signal, and the focal length is set to F1, F2, and F3 as shown in FIG. 11C at the photographing timing. The signal processing unit 203 reads an image from the CCD 202.

  The three-dimensional shape sensor 200 measures the shape of the measurement object 100 by a contrast method. The signal processing unit 203 divides the obtained image into a finite number of blocks. Next, a contrast value is determined in each block. The calculated result is output to the distance extraction unit 204.

  The distance extraction unit 204 performs the following processing. FIG. 12 is a diagram illustrating processing performed on the t-th block by the distance extraction unit.

  First, the distance extraction unit 204 receives focal length information for each of n captured images from the main control unit 207. Further, the contrast value for each block is sent from the signal processing unit 203 to the distance extraction unit 204 for each captured image.

  12, for example, in the t-th block, the distance extraction unit 204 includes a first focal length F1 and contrast value C1 pair 230, a second focal length F2 and contrast value C2 pair 231, A pair 232 of kth focal length Fk and contrast value Ck, and a pair 233 of nth focal length Fn and contrast value Cn are sequentially created.

  The distance extraction unit 204 selects a pair having the maximum contrast value from these pairs 230 to 233 for each block, and adopts the focal distance as the distance from the photographing point to the measurement target.

  As shown in FIG. 12, for example, when the k-th contrast Ck becomes the largest in the t-th block, the focal length Fk at that time is adopted as distance information regarding the block.

  The distance between the measurement object 100 and the CCD 202 is obtained based on the focal length at which the contrast of each block obtained in this way is maximized, and the shape of the measurement object 100 is also determined based on the obtained distance.

  By the way, the faster the reading time of the image sensor, the shorter the time required to complete the distance measurement. Further, the larger the number of pixels of the mounted image sensor, the greater the number of blocks, and the higher the resolution.

  Conventionally, a CCD has been used as an image sensor. However, since a CCD transfers and reads charges, the time required for reading increases in proportion to the increase in the number of pixels. Unlike a CCD charge sequential transfer, a general CMOS image sensor can output pixels by directly addressing them, so that it can be read out faster than a CCD. Accordingly, in order to measure the three-dimensional shape more precisely and at higher speed, it is preferable to use a CMOS image sensor.

  As a conventional technique, Patent Document 1 describes a depth sampling type stereoscopic image forming apparatus in which the sampling interval in depth sampling is variable according to the stereoscopic characteristics and the number of samplings is reduced. Patent Document 2 describes a lens having a variable focal length.

Japanese Patent Laid-Open No. 10-20245 US Pat. No. 6,369,954

  However, when a CMOS image sensor is used for the three-dimensional shape sensor, the following problems occur. That is, a general CMOS image sensor uses a shutter system different from that of a CCD. Specifically, the CCD is a shutter system called a global shutter, in which exposure timing is uniform for all pixels.

  On the other hand, a general CMOS image sensor is a shutter system called a rolling shutter, which is a shutter system in which the exposure timing is shifted for each pixel line. For this reason, the following problems occur.

  As shown in FIG. 11, when a ramp waveform is input to the liquid lens 201, the focal length of the liquid lens 201 is continuously changed, and a pulse drive signal is sent to the image sensor at a predetermined timing to perform imaging, the CCD 202 is used. In the conventional example used, the timing of all the pixels is aligned at the timing of instructing to perform shooting. Therefore, an image photographed at a certain photographing timing is an image having a single focal length that is the entire screen.

  However, when a CMOS image sensor is used, images are sequentially read out for each line, so that the exposure timing is shifted for each line. Since the continuous ramp waveform is input to the liquid lens 201 while the image is being read, the focal length of the liquid lens 201 is shifted, and an image with a different focal length is obtained for each line of the screen. It will be. As a result, the conventional three-dimensional shape sensor cannot measure an accurate distance, and an accurate three-dimensional shape to be measured cannot be obtained.

  An object of the present invention is to provide a three-dimensional shape sensor that can obtain high-definition three-dimensional shape information at high speed using a CMOS image sensor.

According to a first aspect of the present invention, a lens having a variable focal length , a CMOS image sensor using a rolling shutter system for photographing a measurement object through the lens, and an image photographed by the CMOS image sensor are read out for each frame. a signal processing unit for determining a contrast value of each block by dividing the read image into a predetermined number of blocks Te, a lens control unit for changing the focal length of the lens linearly, the lens is changed by the lens control unit A storage unit that stores a table indicating the correspondence between the focal length and time of the image , and a correction value for each block based on the table stored in the storage unit in the focal length information for each frame of the CMOS image sensor Based on the corrected focal length information and the contrast value of each block from the signal processing unit Characterized in that it comprises a distance calculation unit for determining the distance al to the measurement target.

According to the first invention, the distance calculation unit includes the corrected focal length information and the signal obtained by adding the correction value for each block based on the table stored in the storage unit to the focal length information for each frame of the CMOS image sensor. Since the distance from the photographing point to the measurement object is obtained based on the contrast value of each block from the processing unit, high-definition three-dimensional shape information can be obtained at high speed using a CMOS image sensor.

It is a figure which shows the three-dimensional shape sensor of Example 1 carrying a liquid lens. It is a figure which shows the lens input voltage with respect to time of the three-dimensional shape sensor of Example 1, and the focal distance of a liquid lens. It is a figure which shows the example of block division of the obtained image. It is a figure which shows the signal processing procedure of the signal processing part of a three-dimensional shape sensor. It is a figure which shows the algorithm which selects the maximum value of a focal distance and a contrast value. It is a figure which shows the three-dimensional shape sensor of Example 2 carrying a liquid lens. It is a table which shows the correspondence of the focal distance of Example 2, and time. It is a figure which shows the lens input voltage with respect to the time of the three-dimensional shape sensor of Example 2, and the focal distance of a liquid lens. It is a figure which shows the process performed regarding the t-th block by the distance extraction part of Example 2. FIG. It is a figure which shows the conventional three-dimensional shape sensor carrying a liquid lens. It is a figure which shows the lens input voltage with respect to time of the conventional three-dimensional shape sensor, and the focal distance of a liquid lens. It is a figure which shows the process performed regarding the t-th block by a distance extraction part.

  Hereinafter, embodiments of the three-dimensional shape sensor of the present invention will be described in detail with reference to the drawings.

  The three-dimensional shape sensor of Example 1 is characterized in that the liquid lens driving method is devised and the focal length is set to a predetermined value over the entire screen even when a rolling shutter type CMOS image sensor is used.

  In FIG. 1, a three-dimensional shape sensor 10 is a signal processing unit 3, 1 that reads out images of a liquid lens 1, a CMOS image sensor 2, and a CMOS image sensor 2 whose focal length is variable and performs predetermined processing for each frame. The focal length of the liquid lens 1 is changed stepwise for each frame and the focal point of the CMOS image sensor 2 and the image processed by the lens control unit 6 and the signal processing unit 6 that hold the focal length at a predetermined value during one frame period. A distance extraction unit (sometimes referred to as a distance calculation unit) 4 for obtaining a distance from the shooting point to the measurement object based on the distance information, a main control unit 7, and a CMOS image sensor control unit 5 are provided.

  That is, the three-dimensional shape sensor 10 according to the first embodiment replaces the CCD 202, the CCD control unit 205, and the lens control unit 206 in the conventional three-dimensional shape sensor 200 shown in FIG. The difference is that the unit 5 and the lens control unit 6 are provided.

  The liquid lens 1, the main control unit 7, the signal processing unit 3, and the distance extraction unit 4 of the first embodiment are the same as the conventional liquid lens 201, main control unit 207, signal processing unit 203, and distance extraction unit 204 shown in FIG. It is a configuration.

The function of the lens control unit 6 according to the first embodiment will be described with reference to FIG. Here, it is assumed that the distance of the t-th block is measured, and the t-th block is in the Lth row of the CMOS image sensor 2. In FIG. 2, T is a readout time for one frame (one screen), time t0 is a drive signal rising timing, and times t1, t2, t3, and t4 are readout timings of the Lth row of one frame image. is there. Times t1 to t2, times t2 to t3, and times t3 to t4 are read times T for one frame.
Since the conventional lens control unit 206 generates a ramp waveform that changes with time, the focal length of the liquid lens 201 changes continuously with time, and the readout timing of the L row differs from that of the first row. It was the focal length.

  In contrast, in the first embodiment, when an instruction is given to the three-dimensional shape sensor 10 to measure a three-dimensional shape, the main control unit 7 instructs the lens control unit 6 to use the liquid lens 1. An instruction is given to change the focal length of the lens by a predetermined method.

  On the other hand, the main control unit 7 instructs the CMOS image sensor control unit 5 to perform photographing a predetermined number of times while the focal length of the liquid lens 1 is changing. For example, when imaging is performed n times, the CMOS image sensor control unit 5 receives an imaging instruction from the main control unit 7 and drives the CMOS image sensor 2 to perform the imaging shown in FIG. give.

  In response to the instruction from the main control unit 7, the lens control unit 6 increases in voltage stepwise (stepped) every reading time T for one frame in synchronization with the drive signal shown in FIG. A lens input voltage is generated, and a stepped voltage is output to the liquid lens 1.

  As shown in FIG. 2 (c), the liquid lens 1 has stepwise focal lengths F1, F2, and F3 at every readout time T for one frame by the stepped lens input voltage from the lens control unit 6. To rise. The signal processing unit 3 reads an image from the CMOS image sensor 2 for each frame.

  The focal length is held at a predetermined value while reading of all pixels for one frame of the CMOS image sensor 2 is completed, that is, during the period of one frame. For this reason, since the focal length does not change while reading of all the pixels including the Lth row is completed, it is possible to obtain an image equivalent to a case where the image is taken with a conventional CCD.

  The three-dimensional shape sensor 10 measures the shape of the measuring object 100 by a contrast method. As shown in the signal processing procedure of FIG. 4, the signal processing unit 3 divides the obtained image into a finite number of blocks (S101). For example, FIG. 3 shows a block division example 210 of the obtained image. In FIG. 3, the obtained image is divided into blocks by dividing the obtained image into units of 1 × 8 pixels.

Next, a contrast value is determined in each block (S102). The calculation method of the contrast value C is, for example,
C = (lmax−lmin) / (lmax + lmin)
And Here, lmax is the maximum luminance value in the block, and lmin is the minimum luminance value in the block. The calculated result is output to the distance extraction unit 4 (S103).

  Focal length information for each of n captured images is sent from the main control unit 7 to the distance extraction unit 4. Further, the contrast value for each block is sent from the signal processing unit 3 to the distance extraction unit 4 for each captured image.

  As shown in FIG. 12, for example, in the t-th block, the distance extraction unit 4 includes a first focal length F1 and contrast value C1 pair 230, a second focal length F2 and contrast value C2 pair 231, A pair 232 of kth focal length Fk and contrast value Ck, and a pair 233 of nth focal length Fn and contrast value Cn are sequentially created.

  The distance extraction unit 4 selects the pair having the maximum contrast value from these pairs 230 to 233 for each block, and adopts the focal length as the distance from the shooting point to the measurement target.

  As shown in FIG. 12, for example, when the k-th contrast Ck becomes the largest in the t-th block, the focal length Fk at that time is adopted as distance information regarding the block.

  FIG. 5 shows an algorithm for selecting the maximum contrast value. In FIG. 5, Cp represents a contrast value, and Fp represents a focal length. First, Cp = 0 and Fp = 0 are set (S201). Note that p is an integer. Next, it is determined whether Cp is larger than Cp + 1 (S202). If Cp is larger than Cp + 1, Cp = Cp + 1 and Fp = Fp + 1 are set (S203). That is, the larger contrast value of the two compared contrast values and the focal length corresponding thereto are left. Next, it is determined whether or not p + 1 is n (S204). If p + 1 is not n, p is incremented by 1 (S205), the process returns to S202, and the same process is repeated. . If p + 1 becomes n, the process is terminated (S206).

  The distance between the measurement object 100 and the CMOS image sensor 2 is obtained based on the focal length at which the contrast of each block obtained in this way is maximized, and the shape of the measurement object 100 is also determined based on the obtained distance.

  As described above, according to the three-dimensional shape sensor of the first embodiment, the focal length is held at a predetermined value while reading of all pixels for one frame of the CMOS image sensor 2 is completed, that is, during the period of one frame. . For this reason, since the focal length does not change while reading of all the pixels including the Lth row is completed, it is possible to obtain an image equivalent to a case where the image is taken with a conventional CCD. Therefore, high-definition three-dimensional shape information can be obtained at high speed using the rolling shutter type CMOS image sensor 2 having no simultaneity between pixels.

  In the second embodiment, focal length information or information (for example, exposure timing information) that can estimate the focal length is added to each line of the image signal, so that the focal length can be set even if the exposure timing is not uniform in the image. to correct. That is, in the second embodiment, signal processing is performed by adding a correction value for each line to the focal length of the image.

  FIG. 6 is a diagram showing a three-dimensional shape sensor of Example 2 equipped with a liquid lens. The three-dimensional shape sensor 10a of the second embodiment shown in FIG. 6 differs from the three-dimensional shape sensor 10 of the first embodiment shown in FIG. 1 in the configuration of the lens control unit 6a, the main control unit 7a, and the distance extraction unit 4a. .

  The lens control unit 6a linearly changes the focal length of the liquid lens 1. Specifically, the lens control unit 6a receives an instruction from the main control unit 7, and generates a lens input voltage having a ramp waveform having a predetermined inclination with respect to time as shown in FIG. 8B. The lens input voltage is output to the liquid lens 1.

  The main control unit 7a gives an instruction to the CMOS image sensor control unit 5 to perform photographing a predetermined number of times while the focal length of the liquid lens 1 is changing. For example, when n times of photographing are performed, the CMOS image sensor control unit 5 outputs a drive signal for performing photographing to the CMOS image sensor 2 as shown in FIG.

  Further, the main control unit 7a has a memory 71 in which a table showing the correspondence between the focal length and time shown in FIG. 7 is stored. The times Δt1, Δt2,... Δtn shown in FIG. 7 are the times from the read start time t0 of one frame shown in FIG. 8 to the Lth row corresponding to the tth block, and the focal points corresponding to the times Δt1, Δt2,. The distances are ΔF1, ΔF2,... ΔFn.

  Using the table stored in the memory 71, the main control unit 7a outputs focal length information to the distance extraction unit 4a for each line (for each block). That is, the main control unit 7a refers to the focal length information for each frame of the CMOS image sensor 2 (F1, F2, F3,... Shown in FIG. 8C) with reference to the table stored in the memory 71. Correction focal length information obtained by adding correction values ΔF1, ΔF2,... ΔFn (ΔF shown in FIG. 8C) for each is output to the distance extraction unit 4a.

  For example, when the t-th block is present in the Lth row, the focal length of the liquid lens 1 is F1 in the first row (time t0) as shown in FIG. In the row (time t1), the lens input voltage composed of the ramp waveform is larger than that in the first row, and is shifted from the F1 by ΔF to (F1 + ΔF).

  The distance extraction unit 4a uses the corrected focal length information from the main control unit 7a as the distance information of the t-th block. Specifically, when the distance extraction unit 4a creates a pair of the corrected focal length information from the main control unit 7a and the contrast value from the signal processing unit 3, as shown in FIG. In the block, the first corrected focal length F1 + ΔF and contrast value C1 pair 30, the second corrected focal length F2 + ΔF and contrast value C2 pair 31, the kth corrected focal length Fk + ΔF and contrast value Ck pair 32, the nth. A pair 33 of the corrected focal length Fn + ΔF and the contrast value Cn is sequentially created.

  The distance extraction unit 4a selects the pair having the maximum contrast value from these pairs 30 to 33 for each block, and adopts the focal length as the distance from the shooting point to the measurement target.

  As described above, according to the three-dimensional shape sensor of the second embodiment, high-definition three-dimensional shape information can be obtained at high speed by using the corrected focal length information obtained by adding the correction value for each block to the focal length information. it can.

  In addition, this invention is not limited to the three-dimensional shape sensor of Example 1, 2 mentioned above. In the second embodiment, the correction value for each block is stored in the table as the focal length information. However, the correction value may be obtained by calculation for each block without using the table.

  The present invention can be applied to a stereoscopic image forming apparatus using a three-dimensional shape sensor.

DESCRIPTION OF SYMBOLS 1,201 Liquid lens 2 CMOS image sensor 3,203 Signal processing part 4,4a, 204 Distance extraction part 5 CMOS image sensor control part 6,6a, 206 Lens control part 7,7a, 207 Main control part 10,10a, 200 Three-dimensional shape sensor 71 Memory 100 Measuring object 202 CCD
205 CCD controller

Claims (2)

A lens with variable focal length;
A CMOS image sensor using a rolling shutter system for photographing a measurement object through the lens;
A signal processing unit that reads an image captured by the CMOS image sensor for each frame , divides the read image into a predetermined number of blocks, and determines a contrast value of each block ;
A lens controller that linearly changes the focal length of the lens;
A storage unit for storing a table indicating a correspondence relationship between the focal length of the lens and time changed by the lens control unit;
Corrected focal length information obtained by adding correction values for each block based on the table stored in the storage unit to the focal length information for each frame of the CMOS image sensor, and the contrast value of each block from the signal processing unit A distance calculation unit for obtaining a distance from the shooting point to the measurement object based on
A three-dimensional shape sensor comprising: The three-dimensional shape sensor according to claim 1, wherein the lens is a liquid lens .

Images