JPWO2019058750A1 - Imaging system, imaging method and image sensor - Google Patents

Abstract

The image pickup system includes an illumination capable of switching and projecting a plurality of different pattern lights onto an object, an image sensor capable of imaging an object on which the pattern light is projected, and a control unit that controls the illumination and the image sensor. The image sensor can be exposed at different times for each pixel, and the control unit switches between the pattern light and the exposed pixel in conjunction with each other.

Description

The present invention relates to an image pickup system, an image pickup method, and an image pickup device that acquire a three-dimensional shape of an object.

A phase shift method or the like is known as a method for performing three-dimensional measurement without contact with an object. In this phase shift method, the object is kept fixed, that is, stationary, and the striped pattern is projected while phase-shifting over the entire object, and the area camera captures a grid image of the entire object.

However, the phase shift method has not been considered to be an appropriate method for measuring a moving object because it is necessary to take a plurality of images (phase shift images) of an object in different phases. Specifically, in order to measure a moving object by the phase shift method, it is necessary to acquire a plurality of phase shift images at high speed.

Conventionally, a method of acquiring a plurality of phase-shifted images at high speed and substantially simultaneously has been proposed. For example, attempts have been made to increase the speed by linearly polarized the incident light and rotating the polarization direction (see, for example, Patent Document 1). Further, for the first, second, and third wavelength components having different wavelength components, a filter lattice striped plate in which regions that transmit one or two of each wavelength component are arranged in order in a striped pattern is used, and three-dimensionally by a phase shift method. An apparatus for measuring is disclosed (see, for example, Patent Document 2). Further, a shape measuring device using a moire optical system, which synchronizes the relative velocity between the optical system and the test object, the operation cycle of the light receiving element, and the lighting time of a plurality of arranged light sources, is disclosed. (See, for example, Patent Document 3).

Japanese Patent No. 3747471 (International Publication WO2003 / 079467) Japanese Patent No. 4880872 (Japanese Unexamined Patent Publication No. 2006-227652) Japanese Patent No. 3921432 (Japanese Unexamined Patent Publication No. 2004-108829)

However, each of the above methods has problems such as mechanical durability, wavelength characteristics of an object, and reduction of light intensity.

Therefore, an object of the present invention is to provide an imaging system capable of switching a plurality of different pattern lights to perform high-speed imaging.

The imaging system according to the present invention includes illumination capable of switching and projecting a plurality of different pattern lights onto an object.
An image sensor capable of imaging the object on which the pattern light is projected,
A control unit that controls the illumination and the image sensor,
With
The image sensor can be exposed at different times for each pixel.
The control unit switches between the pattern light and the pixel to be exposed in conjunction with each other.

The imaging method according to the present invention includes a step of switching and projecting a plurality of different pattern lights onto an object.
A step of exposing and imaging each pixel group in synchronization with the switching of the pattern light by an image sensor having a pixel group consisting of a plurality of pixels of the object on which the pattern light is projected as many as the pattern light.
including.

According to the image pickup system, the image pickup method, and the image pickup device according to the present invention, a plurality of different pattern lights can be switched and imaged at high speed. Further, if the present invention is applied to the phase shift method, it is possible to acquire a plurality of phase shift images at a sufficiently high speed (substantially simultaneously) as compared with a conventional image sensor, and it is possible to measure a moving object.

It is the schematic which shows the structure of the imaging system which concerns on Embodiment 1. FIG. (A) to (d) are examples of four projection patterns. It is a top view which shows an example of the arrangement of a plurality of pixels constituting an image sensor. It is the schematic which shows the circuit of one pixel of a CMOS image sensor which is an example of an image sensor. It is a schematic plan view which shows an example of the circuit of the image sensor which includes four pixel groups. It is a timing chart of the control signal to each circuit of four pixel groups. It is a block diagram which shows the functional structure of the control part of the image pickup system of FIG. It is a block diagram which shows the physical structure of the control part of FIG. 6A. It is a timing chart which shows the timing of switching of four projection patterns and the start of exposure in synchronization with each pixel group. It is a flowchart of the imaging method which concerns on Embodiment 2. It is a flowchart of the imaging method which concerns on Embodiment 3.

The imaging system according to the first aspect includes illumination capable of switching and projecting a plurality of different pattern lights onto an object.
An image sensor capable of imaging the object on which the pattern light is projected,
A control unit that controls the illumination and the image sensor,
With
The image sensor can be exposed at different times for each pixel.
The control unit switches between the pattern light and the pixel to be exposed in conjunction with each other.

In the image pickup system according to the second aspect, in the first aspect, the image pickup device is provided with a pixel group composed of a plurality of pixels to be exposed at the same time as many as the number of pattern lights.
The control unit may switch between the pattern light and the pixel group to be exposed in conjunction with each other.

The image pickup system according to the third aspect is a common pixel transfer control signal line that controls at least one timing of exposure start or exposure end for each same pixel group in the first or second aspect. May have.

In the image pickup system according to the fourth aspect, in the second or third aspect, the image pickup element has a common pixel reset control signal line that resets the signal charge accumulated in each pixel for each same pixel group. You may.

In the imaging system according to the fifth aspect, in any one of the second to fourth aspects, the illumination may project a grid pattern onto the object while shifting the phase. In addition to the two-dimensional grid pattern, the grid pattern also includes a unidirectional stripe pattern.

In the imaging system according to the sixth aspect, in any one of the second to fifth aspects, the control unit calculates the three-dimensional shape of the object based on a plurality of images acquired for each pixel group. A shape calculation unit may be further provided.

In the imaging system according to the seventh aspect, in any one of the second to sixth aspects, the control unit has a timing of switching the pattern light and a timing of at least one of the exposure start and the exposure end of the pixel group. A synchronization control unit may be further provided to control the synchronization of the above.

In the image pickup system according to the eighth aspect, in any one of the second to seventh aspects, the image pickup device may have pixels belonging to different pixel groups arranged in order along one direction.

The imaging method according to the ninth aspect includes a step of switching and projecting a plurality of different pattern lights onto an object.
A step of exposing and imaging each pixel group in synchronization with the switching of the pattern light by an image sensor having a pixel group consisting of a plurality of pixels of the object on which the pattern light is projected as many as the pattern light.
including.

In the imaging method according to the tenth aspect, in the ninth aspect, the plurality of different pattern lights may be a plurality of lattice patterns having different phases, which are formed while shifting the phases.

The imaging method according to the eleventh aspect may further include the step of calculating the three-dimensional shape of the object based on the plurality of images acquired for each pixel group in the ninth or tenth aspect.

The image pickup device according to the twelfth aspect has a plurality of pixel groups composed of a plurality of pixels.
For each pixel group
A common pixel transfer control signal line that controls at least one timing of exposure start or exposure end of each pixel,
A common pixel reset control signal line that resets the signal charge accumulated in each pixel,
With
The exposure time can be controlled for each pixel group.

Hereinafter, the imaging system and the imaging method according to the embodiment will be described with reference to the accompanying drawings. In the drawings, substantially the same members are designated by the same reference numerals.

(Embodiment 1)
<Imaging system>
FIG. 1 is a schematic view showing a configuration of an imaging system according to the first embodiment.
The image pickup system 30 includes an illumination 10, an image pickup element 22, and a control unit 26. A plurality of different patterns of light can be switched and projected onto the object 2 by the illumination 10. It is possible to image an object on which pattern light is projected by the image sensor. The control unit controls the illumination and the image sensor. The image sensor can be exposed at different times for each pixel. Further, the control unit 26 switches the pattern light and the pixel to be exposed in conjunction with each other to take an image.
As a result, a plurality of different pattern lights can be switched and imaged at high speed.

The components constituting the imaging system 30 will be described below.

<Lighting>
FIG. 2 shows an example of four projection patterns (a) to (d). A plurality of different pattern lights can be switched and projected onto the object 2 by the illumination 10. The illumination 10 includes, for example, a light source 12 and a grid 14. In the example of FIG. 2, four grid patterns (FIGS. 2A to 2D) whose phases are shifted can be projected by moving the grid 14. The method of projecting a phase-shifted pattern is not limited to this. For example, by using a projector as lighting, it is possible to project while switching the grid pattern. Preferably, the grid 14 is a fixed type, and the light source 12 is a switching light source composed of a plurality of LEDs. As a result, the phase of the grid pattern can be shifted while switching the lighting position of the LED, so that faster pattern light switching control becomes possible.

<Camera>
The camera 20 includes a lens 21, an image sensor 22, and an image control circuit 24.

<Lens>
The lens 21 may be any lens that can form an image on the image sensor 22, and may be a lens that can be normally used.

<Image sensor>
FIG. 3 is a plan view showing an example of an arrangement of a plurality of pixels A, B, C, and D constituting the image sensor 22. FIG. 4 is a schematic view showing a circuit of one pixel 40 of a CMOS image sensor which is an example of an image sensor.
The image pickup device 22 has, for example, three or more pixel groups 38 composed of a plurality of pixels 40. In the example shown in FIG. 3, the image sensor 22 has four pixel groups 38a, 38b, 38c, and 38d including pixels A, B, C, and D, respectively. Further, the pixels A, B, C, and D included in each of the four pixel groups 38a, 38b, 38c, and 38d are arranged at the corners of the square to form one processing unit 36. The processing unit 36 is not limited to the case of forming a square, and may be arranged in one line.
The image pickup device may be provided with as many pixel groups as the number of pattern lights, which are composed of a plurality of pixels to be exposed at the same time.
The number of pixel groups may be three or more, and is not limited to the above four pixel groups, and may be five or more pixel groups.

The image sensor may have a common pixel transfer control signal line that controls at least one timing of exposure start or exposure end for the same pixel group. Further, the same pixel group may have a common pixel reset control signal line that resets the signal charge accumulated in each pixel.

The image sensor 22 is, for example, a CMOS image sensor in which a plurality of pixels shown in FIG. 3 are arranged. In the circuit of the pixel 40 shown in FIG. 4, the photodiode (PD) 41 of the light receiving portion, the floating diffusion layer (FD) 42, and the four MOS-FETs (M1, M2, M3, M4) 43a, 43b, 43c, It is composed of 43d and. Further, the pixels 40 include a pixel transfer control signal line (TRG) 44, a pixel read-out selection control signal line (SEL) 46, a vertical signal line (VSL) 47, and a pixel reset control signal line (RST) 48. It is connected.

When the pixel 40 is irradiated with light, it is converted into electrons in the photodiode 41, and a charge corresponding to the light intensity is accumulated in the photodiode 41. The MOS-FET (M1) 43a controls charge transfer between the photodiode (PD) 41 and the floating diffusion layer (FD) 42. When the signal of the pixel transfer control signal line (TRG) 44 is applied to the gate electrode of the MOS-FET (M1) 43a, the electric charge accumulated in the photodiode (PD) 41 is transferred to the floating diffusion layer (FD) 42. Will be done. The floating diffusion layer (FD) 42 is connected to the gate electrode of the MOS-FET (M3) 43c. When the control signal of the pixel readout selection control signal line (SEL) 46 is applied to the gate electrode of the MOS-FET (M4) 43d, it corresponds to the charge accumulated in the floating diffusion layer 42 from the vertical signal line (VSL) 47. The voltage can be read out as a signal. When the reset signal of the pixel reset control signal line (RST) 48 is applied to the gate electrode of the MOS-FET (M2) 43b, the electric charge accumulated in the floating diffusion layer (FD) 42 passes through the MOS-FET (M2) 43b. Since it flows, the charge accumulation state is reset.

First, the start of exposure will be described.
When the pixel reset control signal line (RST) 48 is ON, the electric charge accumulated in the floating diffusion layer (FD) 42 flows through the MOS-FET (M2) 43b, but by itself, the photodiode (PD) 41 of the pixel 40 The charge accumulated in is not reset. When the pixel reset control signal line (RST) 48 is turned ON and the pixel transfer control signal line (TRG) 44 is turned ON, the electric charge accumulated in the photodiode (PD) 41 is transferred to the floating diffusion layer (FD) 42. Is transferred to, and further flows through the MOS-FET (M2) 43b. As a result, the charge accumulation state of the photodiode (PD) 41 and the floating diffusion layer (FD) 42 is reset, and then the exposure to the pixel 40 is started from the timing when the RST signal and the TRG signal are turned off.

Next, the end of exposure will be described.
The signal of the pixel transfer control signal line (TRG) 44 is applied. As a result, the electric charge accumulated in the photodiode (PD) 41 is transferred to the floating diffusion layer (FD) 42. That is, the exposure to the pixel 40 is completed.

Further, reading of the electric charge accumulated in each pixel will be described.
When the control signal of the pixel read selection control signal line (SEL) 46 is applied, the voltage corresponding to the electric charge accumulated in the floating diffusion layer 42 can be read out as a signal from the vertical signal line (VSL) 47 to the pixel 40. it can.

FIG. 5A is a schematic plan view showing an example of a circuit of the image sensor 50 including four pixel groups. In this image pickup device 50, the pixels 40 shown in FIG. 4 are arranged in a vertical and horizontal grid pattern as shown in FIG. 3, and four pixels belonging to the pixel groups A, B, C, and D are regarded as one pixel unit. The arrangement of the pixel units is repeated two-dimensionally. 3 and 5A show an example in which the pixels constituting the pixel unit are arranged in a 2 × 2 square, but the present invention is not limited to this, and for example, the pixels may be arranged in a straight line vertically or horizontally. Further, each of the four pixel groups has a common pixel transfer control signal line (TRG (A), TRG (B), TRG (C), TRG (D)). Further, each of the four pixel groups has a common pixel reset control signal line (RST (A), RST (B), RST (C), RST (D)). As a result, at least one timing of exposure start or exposure end can be controlled for each pixel group, so that exposure can be performed at different timings. In addition, each line has a pixel read selection control signal line (SEL1, SEL2, SEL3, SEL4). In addition, each row has a vertical signal line (VSL).

Further, the image pickup device 50 includes a vertical scanning control circuit 52, a noise removal circuit 54, and a horizontal transfer control circuit 56. Pixel transfer control signal lines (TRG (A), TRG (B), TRG (C), TRG (D)) and pixel reset control signal lines (RST (A), RST (B), RST (C), RST) (D)) is connected to the vertical scanning control circuit 52. The vertical signal line (VSL) is connected to the horizontal transfer control circuit 56 via the noise reduction circuit 54.

FIG. 5B is a timing chart for switching the control signal and the pattern light to each circuit of the four pixel groups. It will be described that the image is taken by controlling the timing of the exposure start and / or the exposure end for each pixel group with reference to FIG. 5B. In FIG. 5B, as shown in FIG. 5A, pixels A, B, C, and D are arranged in a square, pixels A and B are repeatedly arranged in the first row, and pixels C and D are repeatedly arranged in the second row. It is a timing chart about the arranged image pickup elements. That is, for the first line, the pixel transfer control signal line (TRG (A)) of the pixel group A, the pixel transfer control signal line (TRG (B)) of the pixel group B, and the pixel reset control signal line (RST (RST)) of the pixel group A. A)), a pixel reset control signal line (RST (B)) of the pixel group B, and a pixel read-out selection control signal line (SEL1). Further, regarding the second line, the pixel transfer control signal line (TRG (C)) of the pixel group C, the pixel transfer control signal line (TRG (D)) of the pixel group D, and the pixel reset control signal line (RST (RST)) of the pixel group C. C)), a pixel reset control signal line (RST (D)) of the pixel group D, and a pixel read selection control signal line (SEL2).

(A) First, the pixel reset control signal line (RST (A)) of the pixel group A is turned ON at time t0, and at the same time, the pixel transfer control signal line (TRG (A)) of the pixel group A is turned ON. As a result, the accumulated charge of the photodiode 41 of the pixel 40 of the pixel group A is reset.
(B) Next, at time t1, RST (A) and TRG (A) are turned off, and charge accumulation in the photodiode 41 is started. The PTN (a) is turned on between the time t0 and t1, and the projection of the pattern (a) is started. That is, the exposure to the pixels of the pixel group A is started from the time t1.

(C) The pixel transfer control signal line (TRG (A)) of the pixel group A is turned on at time t2. As a result, in the pixels of the pixel group A, the charges accumulated in the photodiode (PD) 41 are transferred to the floating diffusion layer (FD) 42. That is, the exposure of the pixel group A to the pixel 40 is completed. The time from time t1 to time t2 is the exposure time of the pixels of the pixel group A.
At the same time, the pixel reset control signal line (RST (B)) of the pixel group B is turned ON, and at the same time, the pixel transfer control signal line (TRG (B)) of the pixel group B is turned ON. As a result, the accumulated charge of the photodiode 41 of the pixel 40 of the pixel group B is reset.
(D) Next, at time t3, RST (B) and TRG (B) are turned off, and charge accumulation in the photodiode 41 is started. The PTN (a) is turned off between the times t2 and t3, and the projection of the pattern (a) is completed. On the other hand, the PTN (b) is turned on between the times t2 and t3, and the projection of the pattern (b) is started. That is, the exposure to the pixels of the pixel group B is started from the time t3.

(E) The pixel transfer control signal line (TRG (B)) of the pixel group B is turned on at time t4. As a result, in the pixels of the pixel group B, the charges accumulated in the photodiode (PD) 41 are transferred to the floating diffusion layer (FD) 42. That is, the exposure of the pixel group B to the pixel 40 is completed. The time from time t3 to time t4 is the exposure time of the pixels of the pixel group B.
At the same time, the pixel reset control signal line (RST (C)) of the pixel group C is turned ON, and at the same time, the pixel transfer control signal line (TRG (C)) of the pixel group C is turned ON. As a result, the accumulated charge of the photodiode 41 of the pixel 40 of the pixel group C is reset.
(F) Next, at time t5, RST (C) and TRG (C) are turned off, and charge accumulation in the photodiode 41 is started. During the time t4 to t5, the PTN (b) is turned off and the projection of the pattern (b) is completed. On the other hand, the PTN (c) is turned on during the time t4 to t5, and the projection of the pattern (c) is started. That is, the exposure to the pixels of the pixel group C is started from the time t5.

(G) At time t6, the pixel transfer control signal line (TRG (C)) of the pixel group C is turned on. As a result, in the pixels of the pixel group C, the charges accumulated in the photodiode (PD) 41 are transferred to the floating diffusion layer (FD) 42. That is, the exposure of the pixel group C to the pixel 40 is completed. The time from the time t5 to the time t6 is the exposure time of the pixels of the pixel group C.
At the same time, the pixel reset control signal line (RST (D)) of the pixel group D is turned ON, and at the same time, the pixel transfer control signal line (TRG (D)) of the pixel group D is turned ON. As a result, the accumulated charge of the photodiode 41 of the pixel 40 of the pixel group D is reset.
(H) Next, at time t7, RST (D) and TRG (D) are turned off, and charge accumulation in the photodiode 41 is started. During the time t6 to t7, the PTN (c) is turned off and the projection of the pattern (c) is completed. On the other hand, the PTN (d) is turned on between the times t6 and t7, and the projection of the pattern (d) is started. That is, the exposure to the pixels of the pixel group D is started from the time t7.

(I) At time t8, the pixel transfer control signal line (TRG (D)) of the pixel group D is turned on. As a result, in the pixels of the pixel group D, the charges accumulated in the photodiode (PD) 41 are transferred to the floating diffusion layer (FD) 42. That is, the exposure of the pixel group D to the pixel 40 is completed. The time from time t7 to time t8 is the exposure time of the pixels of the pixel group D. The PTN (d) is turned off during the time t8 to t9, and the projection of the pattern (d) is completed.

(J) At time t9, the pixel read selection control signal line (SEL1) of the first line is turned ON. As a result, the voltage corresponding to the electric charge transferred to the floating diffusion layer (FD) 42 of the pixels of each pixel group (pixel groups A and B) in the first row can be read out as a signal.

(K) At time t10, the pixel read selection control signal line (SEL2) of the second line is turned ON. As a result, the voltage corresponding to the charge transferred to the floating diffusion layer (FD) 42 of the pixels of each pixel group (pixel groups C and D) in the second row can be read out as a signal.
As described above, it is possible to control the timing of the start and end of exposure for each pixel group and take an image.

The PTNs (a) to (d) in FIG. 5B represent the projection timings of the four projection patterns (a) to (d), respectively. In this example, the PTN (a) is turned on between t0 and t1 and turned off between t2-t3, so that the exposure time of the pixel group A and the projection of the pattern (a) are switched in tandem. Similarly, the projections of the projection patterns (b) to (d) are also switched in conjunction with the exposure time of the pixel groups B to D.

<Control unit>
FIG. 6A is a block diagram showing a functional configuration of the control unit 26 of the imaging system 30 of FIG. FIG. 6B is a block diagram showing a physical configuration of the control unit 26 of FIG. 6A. The control unit 26 controls the illumination 10 and the image sensor 22. The control unit 26 has, for example, a synchronous control unit 27 and / or a shape calculation unit 28. Further, the control unit 26 may be realized as a physical configuration by an electric circuit, or may be realized by computer software running on a computer. When configured by a computer, it has, for example, the minimum necessary functions of the CPU 31, the memory 32, the storage medium 33, the input / output unit 34, and the like.

By this control unit, the pattern light and the pixel to be exposed are switched in conjunction with each other. Further, when the image sensor is provided with as many pixel groups as the number of pattern lights to be exposed at the same time, the control unit switches between the pattern lights and the pixel groups to be exposed in conjunction with each other.

<Synchronous control unit>
The synchronization control unit 27 controls the synchronization between the pattern light switching timing and at least one of the exposure start or exposure end timings of the pixel group. The synchronization control unit 27 may control the synchronization by an electric mechanism, or may control the synchronization by a computer program. In the latter case, the synchronization control unit 27 may be configured by a computer program.

<Shape calculation unit>
The three-dimensional shape of the object is calculated based on a plurality of images acquired for each pixel group by the shape calculation unit 28. The shape calculation unit 28 can be realized by a computer program. For example, based on a plurality of lattice images acquired for each pixel group, the phase of each pixel is calculated by a phase shift method, a Fourier transform lattice method, etc., and triangulation is performed based on this phase and the positional relationship between the image sensor and the light source. The three-dimensional shape of the object can be calculated by the principle of. For example, according to the phase shift method described in Japanese Patent Application Laid-Open No. 2012-150018, the projection pattern can be switched at high speed by controlling the lighting of the light source, so that the present invention can be used more preferably.
The shape calculation unit 28 considers that the pixels belonging to the same pixel unit are pixels having substantially the same coordinates (those looking at the same location of the object), and the pixels A, B, C, D in one pixel unit. The phase corresponding to the pixel unit may be calculated from the change in brightness of. It is also possible to collect pixels belonging to the same pixel group, create and save an image for each pixel group, and refer to and process pixels having the same coordinates from a plurality of saved images.

(Embodiment 2)
FIG. 7 is a timing chart showing the timing of exposure start and exposure end for each pixel group in synchronization with the switching of the four projection patterns. FIG. 7A shows a synchronization signal, FIG. 7B shows a projection pattern, and FIGS. 7C to 7F show exposure periods of pixel groups A to D. FIG. 8 is a flowchart of the imaging method according to the second embodiment.
(1) A plurality of different pattern lights are switched and projected onto the object (S01).
(2) An object on which pattern light is projected is exposed and imaged for each pixel group in synchronization with switching of pattern light by an image sensor having a pixel group composed of a plurality of pixels as many as the number of pattern lights (S02). The synchronization between the switching of the pattern light and the exposure for each pixel group will be described.
(A) First, a synchronization signal rises with respect to the start of projection of the projection pattern A (FIG. 7 (a)), and the projection pattern A is projected (FIG. 7 (b)). Along with this, imaging of the pixel group A is started (FIG. 7 (c)).
(B) Next, a synchronization signal rises with respect to the start of projection of the projection pattern B (FIG. 7 (a)), and the projection pattern B is projected instead of the projection pattern A (FIG. 7 (b)). Along with this, imaging of the pixel group B is started (FIG. 7 (d)).
(C) Next, a synchronization signal rises with respect to the start of projection of the projection pattern C (FIG. 7 (a)), and the projection pattern C is projected instead of the projection pattern B (FIG. 7 (b)). Along with this, imaging of the pixel group C is started (FIG. 7 (e)).
(D) Next, a synchronization signal rises with respect to the start of projection of the projection pattern D (FIG. 7 (a)), and the projection pattern D is projected instead of the projection pattern C (FIG. 7 (b)). Along with this, imaging of the pixel group D is started (FIG. 7 (f)).
As described above, a plurality of different pattern lights can be switched and projected onto the object, and the pixel group can be exposed and imaged in synchronization with the switching of the pattern lights.

(Embodiment 3)
FIG. 9 is a flowchart of the imaging method according to the third embodiment.
(1) A grid pattern 4 having different phases is switched and projected onto the object 2 while shifting a plurality of phases (S11).
(2) The object 2 on which the grid pattern 4 is projected is exposed and imaged for each pixel group in synchronization with the switching of the grid pattern 4 by the image sensor 22 having a pixel group consisting of a plurality of pixels for the number of phase shifts. (S12). Since the switching of the pattern light and the synchronization of the exposure for each pixel group are the same as the imaging method of the second embodiment, the description thereof will be omitted.
(3) The three-dimensional shape of the object is calculated based on the plurality of images acquired for each pixel group (S13).
From the above, the three-dimensional shape of the object can be calculated.

It should be noted that the present disclosure includes appropriately combining any of the various embodiments and / or examples described above, and the respective embodiments and / or embodiments. The effects of the examples can be achieved.

Further, the present invention can be applied to various three-dimensional measurement methods for switching and projecting a plurality of pattern lights in addition to the phase shift method and the Fourier transform lattice method. For example, it can be applied to a space-coded pattern method in which a plurality of coded light patterns are projected onto a space and the distance to an object is obtained by the principle of triangulation.

According to the image pickup system, the image pickup method, and the image pickup device according to the present invention, a plurality of different pattern lights can be switched and imaged at high speed.

1 Pattern light projection 2 Object 4 Lattice pattern 6 Imaging 10 Lighting 12 Switching light source 14 Grid 20 Camera 21 Lens 22 Image sensor (image sensor)
24 Imaging control circuit 26 Control unit 27 Synchronous control unit 28 Shape calculation unit 30 Imaging system 31 CPU
32 Memory 33 Storage medium 34 Input / output unit 35 Display unit 36 One processing unit 38 Pixel group 40 Pixel 41 Photodiode 42 Floating diffusion layer 43a, 43b, 43c, 43d MOS-FET (M1, M2, M3, M4)
44, 45 pixel transfer control signal line (TRG)
46 pixel read selection control signal line (SEL)
47 Vertical signal line (VSL)
48-pixel reset control signal line (RST)
50 Image sensor 52 Vertical scanning control circuit 54 Noise removal circuit 56 Horizontal transfer control circuit

Claims (12)

Lighting that can switch and project multiple different patterns of light on an object,
An image sensor capable of imaging the object on which the pattern light is projected,
A control unit that controls the illumination and the image sensor,
With
The image sensor can be exposed at different times for each pixel.
The control unit is an imaging system that switches between the pattern light and the pixels to be exposed in conjunction with each other. The image sensor is provided with as many pixel groups as the number of pattern lights, which are composed of a plurality of pixels to be exposed at the same time.
The imaging system according to claim 1, wherein the control unit interlocks and switches between the pattern light and the pixel group to be exposed. The image pickup system according to claim 2, wherein the image pickup device has a common pixel transfer control signal line that controls at least one timing of exposure start or exposure end for each same pixel group. The image pickup system according to claim 2 or 3, wherein the image pickup device has a common pixel reset control signal line that resets the signal charge accumulated in each pixel for each same pixel group. The imaging system according to any one of claims 2 to 4, wherein the illumination projects a grid pattern onto the object while shifting the phase. The control unit further includes a shape calculation unit that calculates a three-dimensional shape of the object based on a plurality of images acquired for each pixel group.
The imaging system according to any one of claims 2 to 5. Any one of claims 2 to 6, wherein the control unit further includes a synchronization control unit that controls synchronization between the switching timing of the pattern light and at least one timing of the exposure start or the exposure end of the pixel group. The imaging system described in. The image pickup system according to any one of claims 2 to 7, wherein the image pickup element is sequentially arranged with pixels belonging to different pixel groups along one direction. The step of switching and projecting multiple different patterns of light on an object,
A step of exposing and imaging each pixel group in synchronization with the switching of the pattern light by an image sensor having a pixel group consisting of a plurality of pixels of the object on which the pattern light is projected as many as the pattern light.
Imaging methods, including. The imaging method according to claim 9, wherein the plurality of different pattern lights are a plurality of lattice patterns having different phases, which are formed while shifting the phases. The imaging method according to claim 9 or 10, further comprising a step of calculating a three-dimensional shape of an object based on a plurality of images acquired for each pixel group. It has multiple pixel groups consisting of multiple pixels,
For each pixel group
A common pixel transfer control signal line that controls at least one timing of exposure start or exposure end of each pixel,
A common pixel reset control signal line that resets the signal charge accumulated in each pixel,
With
An image sensor capable of controlling the exposure time for each pixel group.

Images