KR102499586B1 - imaging device - Google Patents

Abstract

The distance measurement accuracy is improved by combining a plurality of distance measurement methods while avoiding interference of projection light. The light source projection unit projects the intensity-modulated spatial pattern light. The optical time-of-flight distance measuring camera measures the distance to the subject based on the optical time-of-flight of the modulation component included in the light reflected from the subject of the spatial pattern light. A spatial information distance measuring camera measures a distance to a subject based on spatial information included in reflected light. The depth synthesis unit synthesizes the distance measurement results from the time-of-flight ranging camera and the spatial information ranging camera, and obtains a depth value at each pixel position of an image captured by the time-of-flight ranging camera or the spatial information ranging camera. Decide.

Description

imaging device

The present technology relates to an imaging device. In detail, it relates to an imaging device that measures a distance to a subject, a processing method in these devices, and a program for causing a computer to execute the method.

In a recent imaging device, a technique is used that not only acquires an image signal of a subject but also acquires a depth map expressing distance information in units of pixels. As a technology for acquiring distance information, for example, a ToF method that measures distance based on the time of flight (ToF) of modulation components included in reflected light from a subject, or two [0003] A stereo method for measuring a distance from a deviation amount of an image of a dog is known. There are pros and cons to the technology for acquiring these distance information. For example, in the stereo system, there is an advantage that the distance to be measured can be adjusted by adjusting the baseline length, which is the distance between the two cameras. . In addition, by adding an active light source, it is possible to supplement distance measurement in a dark place or non-edge portion. On the other hand, in the ToF method, since the distance is converted from the phase shift, the calculation is simple, but there is a problem that the distance measurement performance deteriorates because a plurality of measurement results such as multi-pass are mixed at the edge of the object. Therefore, a method combining both the ToF method and the stereo method has been proposed (see Patent Document 1 and Patent Document 2, for example).

Patent Document 1: International Publication No. 2012/137434 Patent Document 2: Japanese Unexamined Publication No. 2005-077130

In the prior art described above, by integrating the distance information of both the ToF method and the stereo method, spatially synthesizing those that can be measured with high accuracy by each method is used to obtain high-quality distance information. I think. However, if both are simply combined, the same scene is projected by two light sources, and mutual projection lights interfere with each other, resulting in a noise component, and as a result, there is a risk of deteriorating distance measurement accuracy.

The present technology was created in view of such a situation, and aims to improve distance measurement accuracy by combining a plurality of distance measurement methods while avoiding interference of projection light.

The present technology has been made to solve the above-mentioned problems, and its first aspect is a light source projection unit that projects intensity-modulated spatial pattern light, and the light of the modulation component included in the reflected light from the subject of the spatial pattern light. An optical time-of-flight distance measuring camera that measures the distance to the subject based on the time-of-flight, a spatial information distance measuring camera that measures the distance to the subject based on the spatial information included in the reflected light, and the optical time-of-flight distance measuring camera Depth synthesis for determining a depth value of each pixel position of an image captured by the time-of-flight distance measurement camera or the spatial information distance measurement camera by synthesizing the distance measurement results of the distance measurement camera and the spatial information distance measurement camera It is an imaging device having a part. This has the effect of improving the distance measurement accuracy by synthesizing the distance measurement results from the optical time-of-flight ranging camera and the spatial information ranging camera.

In addition, in this first aspect, the light source projection unit includes a light source generation unit that generates a light source whose intensity is modulated according to a predetermined modulation signal and a vertical synchronization signal, and transforms the light source in response to a spatial position to produce the spatial pattern light. An optical element to generate may be provided. This brings about an effect of projecting the intensity-modulated spatial pattern light.

Further, in the first aspect, each of the optical time-of-flight ranging camera and the spatial information ranging camera generates a depth value and reliability of each pixel position as the measurement result, and the depth synthesis unit generates the The depth value of each pixel position may be determined based on the magnitude of the reliability in the measurement result. This brings about an effect of determining a depth value based on the degree of reliability of the depth value of the optical time-of-flight ranging camera and the spatial information ranging camera. In this case, the depth synthesis unit may select a depth value having the highest reliability among the measurement results for each pixel to determine a depth value for each pixel position.

Further, in this first aspect, the spatial information ranging camera is provided with two left and right imaging elements, and an angle obtained from left and right images obtained from the two imaging elements regarding the spatial information included in the reflected light. A stereo camera that measures the distance to the subject based on the amount of parallax at the pixel position and the baseline length of the two imaging elements may be used. This brings about an effect of improving the distance measurement accuracy by using the distance measurement result in the stereo camera.

Further, in the first aspect, the spatial information ranging camera may be a structure light camera that measures the distance to the subject based on triangulation calculation with respect to the spatial information included in the reflected light. This brings about an effect of improving the distance measurement accuracy by using the distance measurement result in the structure light camera.

Further, in this first aspect, based on the light flight time of the modulation component included in the reflected light from the subject of the spatial pattern light, a second optical time-of-flight distance measuring camera for measuring the distance to the subject is further provided, , The optical time-of-flight ranging camera and the second optical time-of-flight ranging camera may operate as the spatial information ranging camera. This brings about an effect of improving the distance measurement accuracy by using the result of measuring the distance in the two time-of-flight ranging cameras. In this case, each of the optical time-of-flight ranging camera and the second time-of-flight ranging camera generates a depth value and reliability of each pixel position as the measurement result, and the optical time-of-flight ranging camera and the second time-of-flight ranging camera generates a depth value and reliability of each pixel position, which is the result of the measurement as the spatial information ranging camera, and the depth synthesis unit includes the time-of-flight ranging camera and Determining a depth value at each pixel position by selecting a depth value with the highest reliability among the measurement results of each of the second time-of-flight distance measuring camera and the measurement result of the spatial information distance measuring camera for each pixel also good

Further, in the first aspect, the optical time-of-flight ranging camera and the spatial information ranging camera measure the distance to the subject based on the optical time-of-flight of the modulation component included in the reflected light, and An integrated camera may be used that measures the distance to the subject based on triangulation calculation for the spatial information included in the reflected light. This brings about an effect of improving the distance measurement accuracy by using a plurality of measurement results in the integrated camera.

According to the present technology, it is possible to achieve an excellent effect of improving distance measurement accuracy by combining a plurality of distance measurement methods while avoiding interference of projection light. In addition, the effects described here are not necessarily limited, and any one of the effects described in the present disclosure may be used.

1 is a diagram showing a configuration example of an imaging device 100 in an embodiment of the present technology.
Fig. 2 is a diagram showing a configuration example of a shooting control unit 180 in an embodiment of the present technology.
3 is a diagram showing a configuration example of the light source projection unit 130 in the embodiment of the present technology.
Fig. 4 is a diagram showing an example of a state of spatial pattern light in an embodiment of the present technology;
5 is a diagram showing a configuration example of a ToF camera 110 in the first embodiment of the present technology.
6 is a diagram showing a pulse method as an example of a distance measurement method in an embodiment of the present technology.
7 is a diagram showing a configuration example of a stereo camera 120 in the first embodiment of the present technology.
Fig. 8 is a diagram showing a configuration example of the depth synthesis unit 160 in the first embodiment of the present technology.
Fig. 9 is a flowchart showing an example of an operation sequence of the shooting control unit 180 in the embodiment of the present technology.
Fig. 10 is a flowchart showing an example of an operation sequence of the light source projection unit 130 in the embodiment of the present technology.
Fig. 11 is a flowchart showing an example of an operation procedure of the ToF camera 110 in the first embodiment of the present technology.
Fig. 12 is a flowchart showing an example of an operation procedure of the stereo camera 120 in the first embodiment of the present technology.
Fig. 13 is a diagram showing configuration examples of ToF cameras 110 and cameras 126 in the second embodiment of the present technology.
Fig. 14 is a diagram showing a configuration example of a structure light camera 140 in the third embodiment of the present technology.
Fig. 15 is a diagram showing configuration examples of ToF cameras 110 and ToF cameras 116 in the fourth embodiment of the present technology.
16 is a diagram showing a configuration example of a camera 150 in a fifth embodiment of the present technology.
Fig. 17 is a block diagram showing an example of a schematic configuration of a vehicle control system;
Fig. 18 is an explanatory diagram showing an example of installation positions of an out-of-vehicle information detection unit and an imaging unit;
19 is a diagram showing an example of a schematic configuration of an endoscopic surgical system.
Fig. 20 is a block diagram showing an example of a functional configuration of a camera head and a CCU.

Hereinafter, a mode for implementing the present technology (hereinafter referred to as an embodiment) will be described. Description is made in the following order.

1. First Embodiment (Example using ToF camera and stereo camera)

2. Second Embodiment (Example using ToF camera and monocular camera)

3. Third embodiment (example using ToF camera and structure light camera)

4. Fourth embodiment (example using two ToF cameras)

5. Fifth embodiment (example of integrating ToF camera and structure light camera)

6. Application examples to mobile bodies

7. Applications to endoscopic surgery systems

<1. First Embodiment>

[Configuration of imaging device]

1 is a diagram showing a configuration example of an imaging device 100 in an embodiment of the present technology. This imaging device 100 captures an image of the subject 20 and acquires an image signal and distance information of the subject. This imaging device 100 includes an optical time-of-flight distance measurement camera 11, a spatial information distance measurement camera 12, a light source projection unit 130, a depth synthesis unit 160, and an input A reception unit 170 and a shooting control unit 180 are provided.

The optical time-of-flight distance measuring camera 11 is a camera that measures the distance to the subject based on the optical time-of-flight of the modulation component included in the reflected light from the subject. The spatial information distance measurement camera 12 is a camera that measures a distance to a subject based on spatial information included in reflected light from the subject. These time-of-flight ranging cameras 11 and spatial information ranging cameras 12 generate a captured image as well as a depth map expressing distance information in units of pixels, similarly to a normal camera. In the following, captured images are omitted, and the handling of distance information is paid attention to and described.

The light source projection unit 130 projects light required for distance measurement in the time-of-flight distance measurement camera 11 and the spatial information distance measurement camera 12 . Here, since the optical time-of-flight distance measurement camera 11 measures the distance using modulated light, it is necessary that the light source contains a modulation component. On the other hand, since the spatial information distance measurement camera 12 measures the distance using spatial information, it is necessary that the spatial pattern is included in the light source. As described above, in the case where two light sources are simply projected, there is a possibility that the distance measurement accuracy deteriorates due to mutual interference of projected light. Therefore, this light source projection unit 130 projects spatial pattern light intensity-modulated by one light source. In addition, the subject 20 may be irradiated with ambient light.

The depth synthesis unit 160 synthesizes the distance measurement results from the time-of-flight distance measurement camera 11 and the spatial information distance measurement camera 12, and determines the depth value of each pixel position of the captured image. . This depth synthesis unit 160 receives a depth value and its reliability from the time-of-flight distance measurement camera 11 through signal lines 119 and 118 . Further, this depth synthesis unit 160 receives a depth value and its reliability from the spatial information ranging camera 12 through signal lines 129 and 128 .

The input accepting unit 170 accepts input from the outside. As this input accepting unit 170, a shooting button or the like that accepts an input of a shooting start command or a shooting end command is assumed, for example. The input accepting unit 170 supplies the received input to the shooting control unit 180 via the signal line 179 .

The photographing control unit 180 controls the projection operation of the light source projection unit 130 and also controls the photographing operation of the time-of-flight distance measuring camera 11 and the spatial information distance measuring camera 12. . Control signals from the photographing control unit 180 to the light source projection unit 130, the time-of-flight ranging camera 11, and the spatial information ranging camera 12 are supplied through signal lines 188 and 189.

2 is a diagram showing a configuration example of the shooting control unit 180 in the embodiment of the present technology. The shooting control unit 180 includes a vertical synchronization generating unit 181 and a modulation signal generating unit 182 . When an input of, for example, a shooting start command or a shooting end command is received by the input accepting section 170, these vertical synchronization generators 181 and modulated signal generators 182 detect this through the signal line 179. to perform a predetermined operation.

The vertical sync generator 181 generates a vertical sync (V sync) signal necessary for shooting. The vertical synchronizing signal is a signal necessary for photographing moving picture frames at regular intervals, and is usually a signal with a period of about 30 Hz to 120 Hz. The vertical synchronization generating unit 181 starts outputting the vertical synchronization signal when a shooting start command is input, and ends outputting the vertical synchronization signal when a shooting end command is input. This vertical synchronizing signal is supplied to the light source projection unit 130, the time-of-flight ranging camera 11, and the spatial information ranging camera 12 through the signal line 189.

The modulation signal generating unit 182 generates a modulation signal necessary for imaging. The modulated signal is a signal required for distance measurement in the optical time-of-flight ranging camera 11, and is a signal with a period of about 20 MHz to 100 MHz. The modulated signal generator 182 starts outputting the modulated signal when an imaging start command is input, and ends output of the modulated signal when an imaging end command is input. This modulated signal is supplied to the light source projection unit 130 and the optical time-of-flight ranging camera 11 through the signal line 188 .

3 is a diagram showing a configuration example of the light source projection unit 130 in the embodiment of the present technology. The light source projection unit 130 includes a light source generating unit 131, an optical element 133, and a light source control unit 135.

The light source generator 131 generates a light source for generating projection light. As this light source generating unit 131, for example, an LED (Light Emitting Diode), a laser diode, or the like can be used. The light source generator 131 generates an intensity-modulated light source according to the modulation signal supplied from the light source control unit 135. While no modulating signal is supplied, unmodulated normal light is generated. As for the start and end of light source generation, the projection start and end instructions from the light source control unit 135 are followed.

The optical element 133 deforms the light source from the light source generator 131 according to the spatial position to generate spatial pattern light. As the spatial pattern light, for example, a grid pattern, a random point set, or a random light and shade projection, projection such that a dot shape changes depending on the distance, etc. This is assumed and can be designed arbitrarily.

The light source controller 135 controls the light source generator 131 according to the modulation signal and the vertical synchronization signal supplied from the photographing controller 180 through the signal lines 188 and 189 . Immediately after receiving the vertical synchronizing signal, the light source control unit 135 controls the light source generating unit 131 to generate a light source whose intensity is modulated according to the modulation signal. Then, when the exposure time of the time-of-flight distance measurement camera 11 elapses from the reception timing of the vertical synchronization signal, the supply of the modulation signal is stopped, and the light source generator 131 is controlled to generate unmodulated steady light. . Thereafter, when the exposure time of the spatial information distance measuring camera 12 elapses, the light source generating unit 131 is controlled to terminate generating the light source.

In this way, the intensity-modulated projection light continues for the exposure time of the optical time-of-flight ranging camera 11. Regarding the spatial information distance measurement camera 12, distance measurement can be performed by projection using either modulated projection light or unmodulated normal light. Therefore, when exposure of the light time-of-flight ranging camera 11 is completed first, the remaining exposure time of the spatial information ranging camera 12 projects normal light. Since the modulated light is pulsed light, the projection time for the entire time is halved, and the amount of light in the spatial information ranging camera 12 is halved, so projection with normal light can take pictures with higher sensitivity. am.

Fig. 4 is a diagram showing an example of the state of spatial pattern light in the embodiment of the present technology. Any shape can be used for the spatial pattern light in the embodiment of the present technology. Here, a dot pattern is shown as an example.

The optical element 133 uses, for example, a diffraction grating that projects such a dot pattern. In such a diffraction grating, photons are induced by making a micro structure on the surface of the diffraction grating according to its characteristics so as to transform the beam shape of the laser light source into a free shape. It is manufactured by etching quartz glass or glass materials, or by embossing polymer materials. Further, by designing the diffraction grating, a design in which the shape of a point changes according to the distance is also possible. For example, in a near view, it is close to plane light, and light can be condensed to a point as the distance decreases. It is also possible to increase the light projection distance by condensing light into points.

Also, the spatial pattern light may be slit light. Alternatively, a plurality of frequency slit lights may be switched in a time division manner.

5 is a diagram showing a configuration example of the ToF camera 110 in the first embodiment of the present technology. In this first embodiment, a ToF (Time of Flight) camera 110 is assumed as the optical time-of-flight ranging camera 11 . This ToF camera 110 includes a ToF pixel 111, a depth calculation unit 113, and an exposure control unit 115.

The exposure control unit 115 controls exposure of the ToF pixels 111 according to the modulation signal and the vertical synchronization signal supplied from the shooting control unit 180 through the signal lines 188 and 189 . Upon receiving the vertical synchronizing signal, the exposure control unit 115 controls the ToF pixels 111 to start exposure. Then, when the exposure time of the ToF camera 110 elapses from the timing of receiving the vertical sync signal, the ToF pixel 111 is controlled to end the exposure.

The ToF pixel 111 receives the modulated reflected light that hits the subject 20 and bounces back, and photoelectrically converts it into an image signal. When this ToF pixel 111 photoelectrically converts the reflected light, charge is integrated in two windows of normal phase and reverse phase by one pixel, as will be described later.

The depth calculation unit 113 obtains the amount of phase delay (amount of phase shift) from the correlation between the image signal generated by the ToF pixel 111 and the modulation signal, and converts this amount of phase delay into a depth value representing depth. is to convert In addition, this depth calculation unit 113 generates the reliability of the depth value. The depth value generated by this depth calculation unit 113 and its reliability are supplied to the depth synthesis unit 160 via signal lines 119 and 118 .

6 is a diagram showing a pulse method as an example of a distance measurement method in an embodiment of the present technology. The light source projection unit 130 projects the pulse wave according to the modulation frequency of the modulation signal for a certain period (Δt). The projection light reflected from the subject 20 is delayed by the amount of phase delay φ and observed by the ToF camera 110 . At this time, the ToF pixel 111 accumulates and measures the reflected light in two windows of normal phase (0°) and reverse phase (180°) synchronized with the pulse wave projected from the light source projection unit 130 (Q1, Q2). Using the measured electric charges Q1 and Q2, the distance d to the subject 20 can be calculated by the following equation. However, c is the speed of light.

d = (1/2) c c Δt (Q2/(Q1+Q2))

At this time, the total value of the charges Q1 and Q2 is the intensity of reflected light from the subject 20, and the stronger response is obtained, the better the SN ratio (Signal-Noise ratio) of the signal, and this value is used as the reliability use as That is, the reliability (r) is obtained by the following equation.

r=Q1+Q2

Incidentally, although the pulse method has been described here as an example of a distance measurement method, other methods such as a continuous wave method may be used.

Fig. 7 is a diagram showing a configuration example of the stereo camera 120 in the first embodiment of the present technology. In this first embodiment, a stereo camera 120 is assumed as the spatial information ranging camera 12 . This stereo camera 120 includes a left imaging element 121, a right imaging element 122, a depth calculation unit 123, and an exposure control unit 125.

The exposure control unit 125 controls exposure of the left imaging element 121 and the right imaging element 122 according to a vertical synchronization signal supplied from the shooting control unit 180 through the signal line 189 . Upon receiving the vertical synchronizing signal, the exposure control unit 125 controls the left imaging element 121 and the right imaging element 122 to start exposure. Then, when the exposure time of the stereo camera 120 elapses from the timing of receiving the vertical synchronizing signal, the left imaging element 121 and the right imaging element 122 are controlled to end the exposure.

The left image sensor 121 and the right image sensor 122 receive reflected light that hits the subject 20 and bounces back, and photoelectrically converts it into image signals for left and right images.

The depth calculator 123 calculates the amount of parallax at each pixel position from the left and right images, and calculates and outputs the distance based on the baseline length between the left image sensor 121 and the right image sensor 122. In order to calculate the amount of parallax, a patch image of N×N pixels including a pixel of interest in one side of the image appears as a pseudo patch image at a position in the other side of the image, and a patch image with a minimum error while shifting the position. A block matching method for searching for can be used.

If the object is a flat part without the pattern of a patch image, there is a possibility that the distance cannot be obtained skillfully with the stereo camera 120 . However, if the light projected by the pattern light is observed with sufficient contrast, it is possible to calculate the amount of parallax using the shape of the pattern light as a clue.

In addition, since the stereo camera 120 can improve the accuracy of the depth value by calculating the shift amount of the patch image with finer precision than the pixel, the resolution of the left image sensor 121 and the right image sensor 122 is raised. As a result, the precision of the depth value can be raised. In addition, by adjusting the baseline length between the left imaging element 121 and the right imaging element 122, the range of the measurable distance can be easily adjusted.

In this way, the stereo camera 120 receives the vertical synchronization signal from the shooting control unit 180 and controls shooting of the left image pickup device 121 and the right image pickup device 122 . Then, the depth calculation unit 123 calculates the amount of parallax from the captured left and right images, determines the depth value of each pixel position, and outputs it as a depth map. In addition, the depth calculation unit 123 outputs the reliability obtained in the calculation process.

The amount of parallax can be calculated, for example, with the following absolute difference equation. The patch absolute difference (R _SAD ) is the absolute difference between the pixel values of the pixel (I(i, j)) of the patch of the image (L) and the pixel (T(i, j)) of the patch of the image (R) within the patch. It is the value accumulated by the number of pixels. The more similar the patch images are, the smaller the value of the patch absolute difference (R _SAD ). While shifting the reference position of the patch of the image L by the shift amount, the minimum patch absolute value difference is obtained. The shift amount at the time of this minimum difference is a parallax, and becomes a depth value at that position.

[Equation 1]

Further, the reliability can be calculated as a ratio with the second minimum value, for example. Let R _SAD1 be the minimum absolute value difference, and R _SAD2 be the second minimum absolute value difference at a different shift position, and the reliability (C _SAD ), which is a value obtained by subtracting their ratio from 1, is obtained. This value increases as the difference between the two reliability levels increases, and means that the shift amount can be distinguished from another shift amount as the difference increases.

C _SAD =1.0-R _SAD1 /R _SAD2

8 is a diagram showing a configuration example of the depth synthesis unit 160 in the first embodiment of the present technology. This depth synthesis unit 160 includes coordinate conversion units 161 and 162 and a depth synthesis processing unit 163 .

The coordinate conversion unit 161 applies a coordinate conversion operation to the depth value and reliability supplied from the time-of-flight distance measurement camera 11 through the signal lines 119 and 118, and arranges the coordinate position of the depth map. is to do

The coordinate conversion unit 162 applies a coordinate conversion operation to the depth value and reliability supplied from the spatial information distance measurement camera 12 through the signal lines 161 and 162, and arranges the coordinate position of the depth map. is to do

The depth synthesizing processing unit 163 integrates two depth values whose coordinate positions are ordered by the coordinate conversion units 161 and 162 based on the degree of reliability and outputs them. For example, depth synthesis can be performed by selecting the most reliable depth value for each pixel.

Since the parameters required for coordinate conversion are determined by the respective camera positions of the time-of-flight distance measurement camera 11 and the spatial information distance measurement camera 12, they are calibrated in advance, the parameters are obtained, and the values are used. . These parameters are called, for example, camera external parameters, internal parameters, and rectification parameters, and are generally displayed in the following format.

The internal parameters are expressed by the determinant of the following equation. However, f _x and f _y are focal lengths expressed in units of pixels. c _x and c _y are the principal points of the camera.

[Equation 2]

An external parameter is expressed by the following determinant. However, r is an element of a rotation matrix and t is an element of a translational movement, and it is a simultaneous transformation matrix in which these are arranged.

[Formula 3]

The rectification parameters are nine parameters of a 3x3 matrix, and the camera is parallelized to obtain a search such as block matching on an epipolar straight line.

Regarding the imaging device in this first embodiment, it is necessary to set the distance measurement range of the ToF camera 110 and the stereo camera 120 in consideration. That is, in the ToF camera 110, the modulation frequency determines the distance measurement resolution and the limit distance measurement, and the intensity of the modulated light or the exposure time determines the distance of the distance measurement target. If the intensity of the modulated light is strong or the exposure time is long, the light reaches farther away, but there is a possibility that a nearby object is saturated and the distance cannot be measured. On the other hand, in the stereo camera 120, the baseline length between the left image sensor 121 and the right image sensor 122 determines the distance measurement resolution and the limit distance measurement distance, and the intensity of the projection pattern light or the exposure time determines the distance measurement target determine the distance of

In general, a ToF camera has a low resolution, and a high-resolution stereo camera such as a sensor for photographing or moving pictures is often used. Therefore, it is conceivable to obtain a synthesized depth map by setting the ToF camera 110 for near view and the stereo camera 120 for far view.

Regarding the exposure time, it is conceivable to set the ToF camera 110 to a short exposure time and the stereo camera 120 to a long exposure time. With this setting, a ToF depth map for a short distance and a stereo depth map for a long distance are obtained. This has the effect of reducing the perforation of the plane portion caused by the pattern light and occurring in the stereo camera, and strengthening the projection intensity by condensing the light with a point light source rather than a surface light source. This makes it possible to perform distance measurement with a wide dynamic range in the distance direction.

In this way, by realizing the distance measurement using modulation and the distance measurement using pattern with one light source, projection light can be used without waste, and from the viewpoint of power consumption, it is more advantageous than simply combining the two. Also, interference noise caused by two different light sources can be avoided. In addition, as the number of light sources decreases, the module size decreases, and the size and cost are also advantageous.

[Operation of imaging device]

9 is a flowchart showing an example of an operation sequence of the shooting control unit 180 in the embodiment of the present technology.

Before receiving the shooting start command (step S912: No), the shooting controller 180 is in a standby state (step S911). When the shooting start command is input (step S912: Yes), the vertical synchronization generator 181 starts outputting a vertical synchronization signal (step S913), and the modulation signal generator 182 starts outputting a modulation signal (step S913). Step S914).

After that, when a shooting end command is input (Step S915: Yes), the vertical sync generator 181 ends outputting the vertical sync signal (Step S916), and the modulated signal generator 182 outputs the modulated signal. It ends (step S917).

10 is a flowchart showing an example of an operation sequence of the light source projection unit 130 in the embodiment of the present technology.

Before receiving the vertical synchronizing signal (Step S922: No), the light source projection unit 130 is in a standby state (Step S921). When the vertical synchronizing signal is input (Step S922: Yes), the light source generating unit 131 generates an intensity-modulated light source, and projection by the modulated light starts (Step S923).

After that, when the exposure time of the light time-of-flight distance measuring camera 11 elapses from the reception timing of the vertical synchronizing signal (step S924: Yes), the light source generating unit 131 generates unmodulated normal light, and the normal light Projection by starts (step S925).

After that, when the exposure time of the spatial information ranging camera 12 elapses (step S926: Yes), the light source generator 131 stops generating the light source, and the projection ends (step S927).

Fig. 11 is a flowchart showing an example of an operation sequence of the ToF camera 110 in the first embodiment of the present technology.

Before receiving the vertical synchronizing signal (step S932: No), the ToF camera 110 is in a standby state (step S931). When the vertical synchronizing signal is input (step S932: Yes), the ToF pixel 111 starts exposure (step S933).

Then, when the exposure time of the ToF camera 110 elapses from the timing of receiving the vertical synchronizing signal (step S934: Yes), the exposure of the ToF pixel 111 ends (step S935).

Then, the depth calculation unit 113 obtains the amount of phase delay from the correlation between the image signal and the modulated signal, and converts this amount of phase delay into a depth value indicating the depth (step S936). Further, the depth calculation unit 113 generates the reliability of the depth value, and outputs the depth value and the reliability (step S937).

Fig. 12 is a flowchart showing an example of an operation sequence of the stereo camera 120 in the first embodiment of the present technology.

Before receiving the vertical synchronizing signal (step S942: No), the ToF camera 110 is in a standby state (step S941). When the vertical synchronizing signal is input (Step S942: Yes), the left imaging element 121 and the right imaging element 122 start exposure (Step S943).

Then, when the exposure time of the stereo camera 120 elapses from the timing of receiving the vertical synchronizing signal (step S944: Yes), the left image sensor 121 and the right image sensor 122 end exposure (step S945). .

Then, the depth calculator 123 calculates the amount of parallax at each pixel position from the left and right images, and calculates the distance based on the baseline length between the left image sensor 121 and the right image sensor 122 (step S946). Further, the depth calculation unit 123 generates a reliability of the depth value, and outputs the depth value and the reliability (step S947).

In this way, according to the first embodiment of the present technology, the distance measurement accuracy can be improved by synthesizing the depth value and reliability obtained by each of the ToF camera 110 and the stereo camera 120. The projection light projected from the light source projection unit 130 is intensity-modulated spatial pattern light, and includes both a modulation component required for the ToF camera 110 and a spatial pattern required for the stereo camera 120. Therefore, since only one light source is used, interference of projection light can be avoided.

<2. Second Embodiment>

In the above-described first embodiment, the stereo camera 120 is provided in addition to the ToF camera 110, but by capturing one of the left and right images with the ToF camera 110, the stereo camera 120, not the normal It becomes possible to use a monocular camera of Since the value of Q1+Q2 of the ToF camera 110 is the image itself in which the scene was shot, the depth value and the normal image are simultaneously acquired from the ToF camera 110. In this second embodiment, it is assumed that the captured image by the ToF camera 110 is one of the left and right images, and that the ToF camera 110 and a normal monocular camera are combined to acquire the left and right images.

Incidentally, since the entire configuration as the imaging device is the same as that of the first embodiment, detailed description is omitted.

Fig. 13 is a diagram showing a configuration example of the ToF camera 110 and the camera 126 in the second embodiment of the present technology. In this second embodiment, a combination of a ToF camera 110 and a camera 126 is assumed as the spatial information ranging camera 12 . As for the optical time-of-flight ranging camera 11, a ToF camera 110 is assumed as in the first embodiment.

The camera 126 is a monocular camera and captures a left image of a left and right image. At this time, the right image of the left and right images captured by the ToF pixel 111 of the ToF camera 110 is used. In addition, this sharing of left and right images is an example, and the ToF camera 110 may capture a left image, and the camera 126 may capture a right image.

This camera 126 includes a left imaging element 121, a depth calculation unit 123, and an exposure control unit 125.

The exposure control unit 125 controls exposure of the left imaging element 121 according to the vertical synchronization signal supplied from the shooting control unit 180 through the signal line 189 .

The left imaging element 121 receives the reflected light that hits the subject 20 and bounces back, and photoelectrically converts it into an image signal of the left image.

The depth calculator 123 calculates the amount of parallax at each pixel position from the left and right images obtained from the left image sensor 121 and the ToF pixel 111, and calculates the amount of parallax between the left image sensor 121 and the ToF pixel 111. Based on the baseline length, the distance is calculated and output.

In this way, according to the second embodiment of the present technology, by using the captured image of the ToF camera 110 as one of the left and right images, the number of cameras is reduced and the ToF camera and the stereo camera are combined. Similarly, distance measurement accuracy can be improved.

<3. Third Embodiment>

In the first embodiment described above, the stereo camera 120 is assumed as the spatial information ranging camera 12, but other types of cameras may be used as long as distance measurement can be performed using spatial information. In this third embodiment, a structure light camera is assumed as the spatial information ranging camera 12 .

Incidentally, since the entire configuration as the imaging device is the same as that of the first embodiment, detailed description is omitted.

Fig. 14 is a diagram showing a configuration example of a structure light camera 140 in the third embodiment of the present technology. This structure light camera 140 is a camera that acquires a three-dimensional shape by projecting a pattern with a camera. This structure light camera 140 includes an imaging element 141, a depth calculation unit 143, and an exposure control unit 145.

The exposure control unit 145 controls exposure of the imaging element 141 according to a vertical synchronizing signal supplied from the shooting control unit 180 through the signal line 189 . The imaging element 141 receives reflected light that hits the subject 20 and bounces back, and photoelectrically converts it into an image signal. The depth calculator 143 analyzes how the known pattern is deformed in the projected scene and where the pattern appears in the shooting scene, and calculates the depth distance by triangulation calculation. is to calculate

By using this structure light camera 140, it is possible to reduce the number of cameras compared to using a stereo camera.

In this way, according to the third embodiment of the present technology, the distance measurement accuracy can be improved by synthesizing the depth value and reliability obtained by each of the ToF camera 110 and the structure light camera 140.

<4. Fourth Embodiment>

In the second embodiment described above, the ToF camera 110 and the camera 126 are combined, but the camera 126 may be replaced with another ToF camera. In this fourth embodiment, it is assumed that the captured image by the ToF camera 110 is one of the left and right images, and the left and right images are acquired by combining the ToF camera 110 with another ToF camera.

Incidentally, since the entire configuration as the imaging device is the same as that of the first embodiment, detailed description is omitted.

Fig. 15 is a diagram showing configuration examples of the ToF camera 110 and the ToF camera 116 in the fourth embodiment of the present technology. In this fourth embodiment, a combination of a ToF camera 110 and a ToF camera 116 is assumed as the spatial information ranging camera 12 . As for the optical time-of-flight ranging camera 11, a ToF camera 110 is assumed as in the first embodiment.

The ToF camera 116 is a ToF camera different from the ToF camera 110, and captures a left image of the left and right images. At this time, the right image of the left and right images captured by the ToF pixel 111 of the ToF camera 110 is used. Incidentally, this sharing of left and right images is an example, and the ToF camera 110 may capture a left image and the ToF camera 116 may capture a right image.

The depth values generated by the ToF cameras 110 and 116 and their reliability are combined in the depth synthesis unit 165 . The depth synthesis unit 165 generates the depth value and reliability of the ToF camera 110 alone, the depth value and reliability of the ToF camera 116 alone, the ToF camera 110 as the right image, and the ToF camera 116 as the left image Three sets of depth values and reliability of the left and right images can be used. At this time, depth synthesis can be performed by selecting the most reliable depth value for each pixel.

In this fourth embodiment, the dynamic range of brightness can be widened by changing the exposure time between imaging as a ToF camera and imaging as a stereo camera. In addition, by changing the baseline length between the ToF camera 110 and the ToF camera 116, the dynamic range of distance measurement can be widened. Further, for example, it is conceivable to use depth values by individual ToF cameras 110 and 116 for short distances, and to use active stereo depth values of ToF cameras 110 and 116 for long distances. .

In this way, according to the fourth embodiment of the present technology, the distance measurement accuracy can be further improved by synthesizing the depth value and reliability obtained by each of the ToF cameras 110 and 116.

<5. Fifth Embodiment>

In the third embodiment described above, the ToF camera 110 and the structure light camera 140 are combined, but they may be integrated into one camera. In this fifth embodiment, a camera integrated as the optical time-of-flight ranging camera 11 and the spatial information ranging camera 12 is assumed.

Fig. 16 is a diagram showing a configuration example of a camera 150 in the fifth embodiment of the present technology. This camera 150 integrates a ToF camera and a structure light camera. This camera 150 includes a ToF pixel 151, depth calculation units 153 and 154, and an exposure control unit 155.

The exposure control unit 155 controls exposure of the ToF pixels 151 according to the modulation signal and the vertical synchronization signal supplied from the shooting control unit 180 through the signal lines 188 and 189 . The ToF pixel 151 receives the modulated reflected light that hits the subject 20 and bounces back, and photoelectrically converts it into an image signal.

The depth calculation unit 153 obtains the amount of phase delay from the correlation between the image signal generated by the ToF pixel 151 and the modulation signal, and converts this amount of phase delay into a depth value representing depth. The depth calculation unit 154 analyzes how the known pattern is deformed in the projected scene and where the pattern appears in the shooting scene, and calculates the depth distance by triangulation calculation. That is, distance measurement using a modulated signal is performed by the depth calculator 153, and distance measurement using a pattern is performed by the depth calculator 154.

In this way, according to the fifth embodiment of the present technology, the distance measurement using the modulated signal and the distance measurement using the pattern are performed by the integrated camera, thereby reducing the number of cameras and improving the distance measurement accuracy. can make it

As described so far, the embodiment of the present technology has the following advantages.

First, interference of projection light can be avoided by overlapping spatial information and temporal information in the light source. Since the spatial direction (structured light) and the temporal direction (modulation) are independent, these information can be utilized without waste.

Second, from the viewpoint of the way the reflected light is viewed, since the ToF camera has low resolution, the high-resolution structured light pattern is not visible, and it can be treated as almost equivalent to surface light emission, and distance measurement by high-speed modulation is possible. In active stereo, since the modulation of the light source is sufficiently fast, it is equivalent to projecting at a constant intensity, and it is possible to perform distance measurement using a high-resolution spatial pattern.

Thirdly, the division of the distance measurement range can be changed by using the characteristics of the ToF camera and the stereo camera. A ToF camera can measure distance with a short exposure and is difficult to saturate even at a short distance, so it is suitable for handling the front side of the depth value. On the other hand, since the active stereo camera has long exposure, high resolution, and high sensitivity, it is suitable for handling the deep side of the depth value.

Fourth, by using the characteristics of a stereo camera, it is possible to exhibit performance impossible only with a ToF camera. That is, active stereo can measure the distance on a surface without texture by using a high-resolution pattern, and can increase the reach distance by condensing light.

<6. Applications to mobile bodies>

The technology of the present disclosure (this technology) can be applied to various products. For example, the technology according to the present disclosure is realized as a device mounted on any type of mobile body such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility, airplanes, radio-controlled unmanned aerial vehicles, ships, and robots. It may be.

17 is a diagram to which the technology of the present disclosure may be applied. It is a block diagram showing a schematic configuration example of a vehicle control system that is an example of a moving body control system.

The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001 . In the example shown in FIG. 17 , the vehicle control system 12000 includes a driving system control unit 12010, a body system control unit 12020, an outside information detection unit 12030, an in-vehicle information detection unit 12040, and integrated control. unit 12050. Also, as functional configurations of the integrated control unit 12050, a microcomputer 12051, an audio image output unit 12052, and an in-vehicle network I/F (interface) 12053 are shown.

The drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the driving system control unit 12010 includes a driving force generating device for generating driving force of the vehicle, such as an internal combustion engine or a driving motor, a driving force transmission mechanism for transmitting the driving force to wheels, and steering adjusting the steering angle of the vehicle. It functions as a control device such as a mechanism and a braking device that generates braking force for the vehicle.

The body system control unit 12020 controls the operation of various devices installed in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as head lamps, back lamps, brake lamps, blinkers, or fog lamps. . In this case, the body system control unit 12020 may receive radio waves transmitted from a portable device that has replaced a key or signals from various switches. The body system control unit 12020 receives input of these radio waves or signals, and controls the door lock device, power window device, lamps, and the like of the vehicle.

An out-of-vehicle information detection unit 12030 detects information outside the vehicle in which the vehicle control system 12000 is installed. For example, an imaging unit 12031 is connected to the outside-in-vehicle information detection unit 12030 . The out-of-vehicle information detection unit 12030 causes the imaging unit 12031 to capture an image of the outside of the vehicle and receives the captured image. The out-of-vehicle information detection unit 12030 may perform object detection processing or distance detection processing, such as people, cars, obstacles, signs, or characters on the road, based on the received image.

The imaging unit 12031 is a photosensor that receives light and outputs an electrical signal corresponding to the received amount of the light. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

In-vehicle information detection unit 12040 detects in-vehicle information. Connected to the in-vehicle information detection unit 12040 is a driver state detector 12041 that detects the driver's state, for example. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 detects the degree of driver's fatigue or concentration based on detection information input from the driver state detection unit 12041. The degree may be calculated, or it may be determined whether the driver is not asleep.

The microcomputer 12051 calculates control target values of the driving force generating device, the steering mechanism, or the braking device based on the inside/outside information acquired by the outside information detection unit 12030 or the inside information detection unit 12040, A control command can be output to the drive system control unit 12010 . For example, the microcomputer 12051 provides ADAS (Advanced Driver Assistance System) including collision avoidance or shock mitigation of vehicles, following driving based on inter-vehicle distance, vehicle speed maintaining driving, vehicle collision warning, or vehicle lane departure warning. Assistance System) can perform cooperative control for the purpose of realizing the function.

In addition, the microcomputer 12051 controls the driving force generating device, steering mechanism, braking device, etc. based on the information about the surroundings of the vehicle acquired by the out-of-vehicle information detection unit 12030 or the in-vehicle information detection unit 12040, thereby It is possible to perform cooperative control for the purpose of autonomous driving, etc., in which the vehicle travels autonomously without driver's operation.

In addition, the microcomputer 12051 can output a control command to the body control unit 12020 based on the out-of-vehicle information acquired by the out-of-vehicle information detection unit 12030. For example, the microcomputer 12051 controls the headlamps in response to the position of the preceding vehicle or the oncoming vehicle detected by the out-of-vehicle information detection unit 12030 to switch high beams to low beams, or the like. ) can be performed for the purpose of cooperative control.

The audio image output unit 12052 transmits at least one of audio and image output signals to an output device capable of visually or aurally notifying information to occupants of the vehicle or outside the vehicle. In the example of Fig. 17, an audio speaker 12061, a display unit 12062, and an installation panel 12063 are exemplified as output devices. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

18 is a diagram showing an example of an installation position of the imaging unit 12031 .

In FIG. 18 , a vehicle 12100 includes imaging units 12101 , 12102 , 12103 , 12104 , and 12105 as an imaging unit 12031 .

The imaging units 12101, 12102, 12103, 12104, and 12105 are provided at positions such as the front nose, side mirrors, rear bumper, back door, and top of the windshield of the vehicle 12100, for example. An image capturing unit 12101 provided on the front nose and an image capturing unit 12105 provided on the upper part of the windshield inside the vehicle mainly acquires an image in front of the vehicle 12100. The imaging units 12102 and 12103 provided in the side mirrors mainly acquire images of the side of the vehicle 12100 . An imaging unit 12104 provided on the rear bumper or back door mainly acquires an image of the rear of the vehicle 12100 . Forward images acquired by the imaging units 12101 and 12105 are mainly used for detecting a preceding vehicle or pedestrian, an obstacle, a signal, a traffic sign, a lane, or the like.

18 shows an example of the imaging range of the imaging units 12101 to 12104 . The imaging range 12111 represents the imaging range of the imaging unit 12101 provided on the front nose, and the imaging ranges 12112 and 12113 represent the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively. A range 12114 indicates an imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, by overlapping image data captured by the imaging units 12101 to 12104, a bird's eye view image of the vehicle 12100 viewed from above can be obtained.

At least one of the imaging units 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

For example, the microcomputer 12051 determines the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the temporal change of the distance (vehicle (relative speed with respect to 12100), in particular, a predetermined speed (eg, 0km/Hz A three-dimensional object traveling in the above) can be extracted as a preceding vehicle. In addition, the microcomputer 12051 sets a head-to-head distance to be secured in advance in front of the preceding vehicle, and performs automatic brake control (including follow-stop control), automatic acceleration control (including follow-start control), and the like. there is a number In this way, it is possible to perform cooperative control for the purpose of autonomous driving, etc., in which the vehicle travels autonomously without driver's operation.

For example, the microcomputer 12051 converts three-dimensional object data about three-dimensional objects to other three-dimensional objects such as two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, telephone poles, etc., based on distance information obtained from the imaging units 12101 to 12104. It can be classified and extracted, and used for automatic avoidance of obstacles. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as obstacles that can be seen by the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines the collision risk indicating the degree of risk of collision with each obstacle, and when the collision risk is higher than the set value and there is a possibility of collision, the audio speaker 12061 and the display unit 12062 are used. It is possible to provide driving assistance for collision avoidance by outputting a warning to the driver or performing forced deceleration or avoidance steering using the drive system control unit 12010.

At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not there is a pedestrian in the captured images of the imaging units 12101 to 12104 . Recognition of these pedestrians is, for example, the order of extracting feature points from images captured by the imaging units 12101 to 12104 as infrared cameras, and pattern matching processing is performed on a series of feature points representing the contours of objects to determine whether or not they are pedestrians. It is performed in the order of determining. When the microcomputer 12051 determines that a pedestrian exists in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio image output unit 12052 gives the recognized pedestrian a rectangular outline for emphasis. The display portion 12062 is controlled so as to overlap. Furthermore, the audio image output unit 12052 may control the display unit 12062 to display an icon or the like representing a pedestrian at a desired position.

Above, to which the technology of the present disclosure can be applied. An example of a vehicle control system has been described. The technology of the present disclosure can be applied to the imaging unit 12031 among the configurations described above.

<7. Applications to endoscopic surgical systems>

The technology of the present disclosure (this technology) can be applied to various products. For example, the technology of the present disclosure may be applied to an endoscopic surgical system.

19 shows a diagram to which the technology (this technology) of the present disclosure can be applied. It is a drawing showing an example of a schematic configuration of an endoscopic surgical system.

In FIG. 19 , a state in which a surgeon (doctor) 11131 is performing an operation on a patient 11132 on a patient bed 11133 using the endoscopic surgery system 11000 is shown. As shown in the figure, the endoscopic surgical system 11000 includes an endoscope 11100, an inflating tube 11111 and an energy treatment instrument 11112, and other surgical tools ( 11110), a support arm device 11120 for supporting the endoscope 11100, and a cart 11200 on which various devices for endoscopic surgery are mounted.

The endoscope 11100 is composed of a lens barrel 11101 in which an area of a predetermined length from the tip is inserted into the body cavity of a patient 11132, and a camera head 11102 connected to the base end of the lens barrel 11101. In the illustrated example, the endoscope 11100 configured as a so-called hard mirror having a hard lens barrel 11101 is shown, but the endoscope 11100 may be configured as a so-called soft mirror having a flexible lens barrel.

The front end of the lens barrel 11101 is provided with an opening into which an objective lens is fitted. A light source device 11203 is connected to the endoscope 11100, and the light generated by the light source device 11203 is directed to the tip of the lens barrel 11101 by a light guide extended inside the lens barrel 11101. Light is guided and irradiated toward an observation object in the body cavity of the patient 11132 through an objective lens. Further, the endoscope 11100 may be a direct view mirror, a strabismus or a side view mirror.

An optical system and an imaging device are provided inside the camera head 11102, and reflected light from an object to be observed (observation light) is focused on the imaging device by the optical system. The observation light is photoelectrically converted by the imaging device, and an electrical signal corresponding to the observation light, that is, an image signal corresponding to the observation image is generated. The image signal is transmitted to a camera control unit (CCU: Camera Control Unit) 11201 as RAW data.

The CCU 11201 is constituted by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like, and controls operations of the endoscope 11100 and the display device 11202 in a comprehensive manner. Further, the CCU 11201 receives an image signal from the camera head 11102, and for displaying an image based on the image signal, for example, development processing (demosaic processing), etc. Various image processing is performed.

The display device 11202 displays an image based on an image signal subjected to image processing by the CCU 11201 under control from the CCU 11201 .

The light source device 11203 is constituted by, for example, a light source such as an LED (Light Emitting Diode), and supplies irradiated light to the endoscope 11100 when imaging a description part or the like.

Input device 11204 is an input interface to endoscopic surgical system 11000 . The user can input various types of information or input instructions to the endoscopic surgical system 11000 through the input device 11204 . For example, the user inputs an instruction or the like to change imaging conditions (type of irradiation light, magnification, focal length, etc.) by the endoscope 11100 .

A treatment instrument control device 11205 controls driving of an energy treatment instrument 11112 for tissue cauterization, incision, blood vessel sealing, and the like. The relief device 11206 inflates the body cavity of the patient 11132 for the purpose of securing a visual field by the endoscope 11100 and securing an operator's working space, by injecting gas into the body cavity through the relief tube 11111. send. The recorder 11207 is a device capable of recording various types of information related to surgery. The printer 11208 is a device capable of printing various kinds of information related to surgery in various formats such as text, images, or graphs.

In addition, the light source device 11203 for supplying irradiation light when photographing the description with the endoscope 11100 can be configured with, for example, a white light source composed of an LED, a laser light source, or a combination thereof. When a white light source is constituted by a combination of RGB laser light sources, since the output intensity and output timing of each color (each wavelength) can be controlled with high precision, the white balance of the captured image in the light source device 11203 can be adjusted can do Further, in this case, laser light from each of the RGB laser light sources is time-divided to irradiate the object of observation, and driving of the imaging element of the camera head 11102 is controlled in synchronization with the timing of the irradiation, thereby obtaining an image corresponding to each of RGB. It is also possible to image in time division. According to this method, a color image can be obtained without providing a color filter in the imaging device.

Further, the driving of the light source device 11203 may be controlled such that the intensity of the light to be output is changed every predetermined period of time. By controlling the drive of the imaging element of the camera head 11102 in synchronization with the timing of the change in the intensity of the light, acquiring images in time division, and combining the images, so-called blackout and whitewashing are achieved. An image with a high dynamic range free of distortion can be created.

Further, the light source device 11203 may be configured to be capable of supplying light in a predetermined wavelength band corresponding to special light observation. In special light observation, for example, using the wavelength dependence of light absorption in body tissue, by irradiating a narrowband light compared to the irradiation light (i.e., white light) at the time of normal observation, mucous membranes BACKGROUND OF THE INVENTION [0002] So-called narrow band imaging is performed in which a predetermined tissue such as a blood vessel in the superficial layer is imaged with high contrast. Alternatively, in special light observation, fluorescence observation may be performed in which an image is obtained by fluorescence generated by irradiation with excitation light. In fluorescence observation, a body tissue is irradiated with excitation light and fluorescence from the body tissue is observed (autofluorescence observation), or a reagent such as indocyanine green (ICG) is applied to the body tissue. In addition, it is possible to obtain a fluorescent image by irradiating the body tissue with excitation light corresponding to the fluorescence wavelength of the reagent. The light source device 11203 can be configured to be capable of supplying narrowband light and/or excitation light corresponding to observation of such special light.

Fig. 20 is a block diagram showing an example of the functional configuration of the camera head 11102 and CCU 11201 shown in Fig. 19 .

The camera head 11102 includes a lens unit 11401, an imaging unit 11402, a driving unit 11403, a communication unit 11404, and a camera head control unit 11405. The CCU 11201 includes a communication unit 11411, an image processing unit 11412, and a control unit 11413. The camera head 11102 and the CCU 11201 are connected by a transmission cable 11400 so that communication with each other is possible.

The lens unit 11401 is an optical system provided at a connection portion with the lens barrel 11101 . Observation light received at the tip of the lens barrel 11101 is guided to the camera head 11102 and enters the lens unit 11401. The lens unit 11401 is constituted by combining a plurality of lenses including a zoom lens and a focus lens.

The imaging unit 11402 is composed of an imaging element. The imaging element constituting the imaging unit 11402 may be one (so-called single-plate type) or plural (so-called multi-plate type). In the case where the imaging unit 11402 is configured in a multi-plate type, for example, image signals corresponding to each of RGB are generated by each imaging element, and a color image may be obtained by combining them. Alternatively, the imaging unit 11402 may be configured to have a pair of imaging elements for respectively acquiring image signals for the right eye and for the left eye corresponding to 3D (Dimensional) display. By performing 3D display, it becomes possible for the operator 11131 to grasp the depth of the biological tissue in the surgical part more accurately. In addition, when the image pickup unit 11402 is configured in a multi-plate type, a plurality of lens units 11401 may also be provided corresponding to each image pickup device.

In addition, the imaging unit 11402 does not necessarily need to be provided in the camera head 11102. For example, the imaging unit 11402 may be provided inside the barrel 11101 immediately after the objective lens.

The drive unit 11403 is composed of actuators and moves the zoom lens and focus lens of the lens unit 11401 along the optical axis by a predetermined distance under control from the camera head control unit 11405. Thereby, the magnification and focus of the image captured by the imaging unit 11402 can be appropriately adjusted.

The communication unit 11404 is constituted by a communication device for transmitting and receiving various kinds of information to and from the CCU 11201. The communication unit 11404 transmits image signals obtained from the imaging unit 11402 as RAW data to the CCU 11201 via a transmission cable 11400.

Further, the communication unit 11404 receives a control signal for controlling driving of the camera head 11102 from the CCU 11201 and supplies it to the camera head control unit 11405. The control signal includes, for example, information to the effect of specifying the frame rate of the captured image, information to the effect of designating the exposure value during imaging, and/or information to the effect of designating the magnification and focus of the captured image, etc. Information about conditions is included.

Incidentally, imaging conditions such as the frame rate, exposure value, magnification, and focus may be appropriately designated by the user or may be automatically set by the control unit 11413 of the CCU 11201 based on the obtained image signal. In the latter case, the so-called AE (Auto Exposure) function, AF (Auto Focus) function, and AWB (Auto White Balance) function are installed in the endoscope 11100.

The camera head control unit 11405 controls driving of the camera head 11102 based on a control signal from the CCU 11201 received via the communication unit 11404.

The communication unit 11411 is constituted by a communication device for transmitting and receiving various kinds of information to and from the camera head 11102 . The communication unit 11411 receives an image signal transmitted from the camera head 11102 through the transmission cable 11400 .

Further, the communication unit 11411 transmits a control signal for controlling driving of the camera head 11102 to the camera head 11102 . Image signals and control signals can be transmitted by electrical communication or optical communication.

The image processing unit 11412 performs various image processes on an image signal that is RAW data transmitted from the camera head 11102.

The control unit 11413 performs various controls related to capturing an image of a predicated portion or the like by the endoscope 11100 and displaying a captured image obtained by capturing an image of the predicated portion or the like. For example, the control unit 11413 generates a control signal for controlling driving of the camera head 11102 .

Further, the control unit 11413 causes the display device 11202 to display a captured image with descriptions and the like based on the image signal subjected to image processing by the image processing unit 11412 . At this time, the control unit 11413 may recognize various objects in the captured image using various image recognition techniques. For example, the control unit 11413 detects the shape, color, etc. of the edge of an object included in a captured image, and uses an instrument such as forceps, a specific body part, bleeding, or an energy treatment instrument ( 11112) can be recognized when using. When displaying a captured image on the display device 11202, the control unit 11413 may use the recognition result to display various types of surgery support information superimposed on the image of the predicate. By superimposing the operation support information and presenting it to the operator 11131, it becomes possible to reduce the burden on the operator 11131 and to ensure that the operator 11131 performs surgery.

A transmission cable 11400 connecting the camera head 11102 and the CCU 11201 is an electrical signal cable corresponding to electrical signal communication, an optical fiber corresponding to optical communication, or a composite cable thereof.

Here, in the illustrated example, communication is performed by wire using the transmission cable 11400, but communication between the camera head 11102 and the CCU 11201 may be performed wirelessly.

Above, to which the technology of the present disclosure can be applied. An example of an endoscopic surgical system has been described. The technology of the present disclosure can be applied to the imaging unit 11402 of the camera head 11102 among the above-described configurations.

In addition, although the endoscopic surgery system has been described here as an example, the technology of the present disclosure may be applied to other, for example, microscopic surgery systems.

In addition, the embodiment described above shows an example for realizing the present technology, and the matters in the embodiments and the specific matters of the invention in the claims each have a corresponding relationship. Similarly, the invention specific matter in the claims and the matter in the embodiment of the present technology having the same name as this have a correspondence relationship, respectively. However, the present technology is not limited to the embodiments, and can be implemented by making various modifications to the embodiments without departing from the gist thereof.

In addition, the processing procedure described in the above-described embodiment may be understood as a method having a series of these procedures, or as a program for executing a series of these procedures in a computer or a recording medium storing the program also good As this recording medium, CD (Compact Disc), MD (MiniDisc), DVD (Digital Versatile Disc), memory card, Blu-ray (registered trademark) Disc, or the like can be used, for example.

In addition, the effect described in this specification is only an example and is not limited, and other effects may be present.

In addition, the present technology can also take the following configurations.

(1) a light source projection unit for projecting intensity-modulated spatial pattern light;

an optical time-of-flight distance measuring camera for measuring a distance to the subject based on an optical time-of-flight of a modulation component included in light reflected from the subject of the spatial pattern light;

a spatial information distance measurement camera for measuring a distance to the subject based on spatial information included in the reflected light;

A depth value of each pixel position of an image captured by the optical time-of-flight ranging camera and the spatial information ranging camera is synthesized by synthesizing the distance measurement result from the optical time-of-flight ranging camera or the spatial information ranging camera An imaging device having a depth synthesis unit that determines.

(2) the light source projection unit,

A light source generator for generating intensity-modulated light sources according to a predetermined modulation signal and a vertical synchronization signal;

The imaging device according to (1) above, comprising an optical element for generating the spatial pattern light by deforming the light source in response to a spatial position.

(3) each of the optical time-of-flight distance measurement camera and the spatial information distance measurement camera generates a depth value of each pixel position and its reliability as the measurement result;

The imaging device according to (1) or (2), wherein the depth synthesis unit determines a depth value at each pixel position based on the magnitude of the reliability in the measurement result.

(4) The imaging device according to (3), wherein the depth synthesizing unit determines a depth value at each pixel position by selecting a depth value having the highest reliability among the measurement results for each pixel.

(5) The spatial information ranging camera is provided with two left and right imaging elements, and the amount of parallax at each pixel position obtained from the left and right images obtained from the two imaging elements with respect to the spatial information included in the reflected light and the imaging device according to any one of (1) to (4) above, which is a stereo camera that measures the distance to the subject based on the baseline lengths of the two imaging elements.

(6) The spatial information ranging camera is any one of (1) to (4) above, which is a structure light camera that measures the distance to the subject based on triangulation calculation with respect to the spatial information included in the reflected light. The imaging device described in one.

(7) Further comprising a second time-of-flight distance measuring camera for measuring the distance to the subject based on the time-of-flight of the modulation component included in the reflected light from the subject of the spatial pattern light,

The imaging device according to any one of (1) to (4), wherein the optical time-of-flight ranging camera and the second optical time-of-flight ranging camera operate as the spatial information ranging camera.

(8) Each of the optical time-of-flight distance measuring camera and the second optical time-of-flight distance measuring camera generates a depth value of each pixel position and its reliability as the measurement result,

The optical time-of-flight distance measurement camera and the second optical time-of-flight distance measurement camera generate a depth value of each pixel position and its reliability, which are the measurement results as the spatial information distance measurement camera;

The depth synthesis unit converts the depth value having the highest reliability among the measurement results of the optical time-of-flight ranging camera and the second optical time-of-flight ranging camera and the measurement result of the spatial information ranging camera to a pixel The imaging device according to (7) above, in which the depth value of each pixel position is determined by selection for each pixel position.

(9) The optical time-of-flight distance measuring camera and the spatial information distance measuring camera measure the distance to the subject based on the optical time-of-flight of the modulation component included in the reflected light, and the space included in the reflected light The imaging device according to any one of (1) to (4) above, which is an integrated camera that measures the distance to the subject based on triangulation calculation for information.

11: optical time-of-flight distance measurement camera
12: Spatial information distance measurement camera
20: subject
100: imaging device
110, 116: ToF camera
111: ToF pixel
113: depth calculation unit
115: exposure control unit
120: stereo camera
121: left imaging device
122: right imaging element
123: depth calculator
125: exposure control unit
126: camera
130: light source projection unit
131: light source generator
133: optical element
135: light source control unit
140: structure light camera
141: imaging device
143: depth calculation unit
145: exposure control unit
150: camera (ToF camera and structure light camera)
151: ToF pixel
153, 154: depth calculator
155: exposure control unit
160, 165: depth synthesis unit
161, 162: coordinate conversion unit
163: depth synthesis processing unit
170: input reception unit
180: shooting control unit
181: vertical synchronization generating unit
182: modulation signal generator

Claims (9)

a light source projection unit for projecting intensity-modulated spatial pattern light;
a first optical time-of-flight distance measuring camera for measuring a distance to the subject based on an optical time-of-flight of a modulation component included in reflected light of the spatial pattern light from the subject;
a spatial information distance measurement camera for measuring a distance to the subject based on spatial information included in the reflected light;
Each pixel position of an image captured by the first time-of-flight ranging camera or the spatial information ranging camera by synthesizing the result of distance measurement by the first time-of-flight ranging camera and the spatial information ranging camera; A depth synthesis unit for determining a depth value of
A second time-of-flight distance measuring camera for measuring a distance to the subject based on the time-of-flight of the modulation component included in light reflected from the subject of the spatial pattern light,
The first time-of-flight ranging camera and the second time-of-flight ranging camera operate as the spatial information ranging camera;
Each of the first time-of-flight distance measurement camera and the second time-of-flight distance measurement camera generates a depth value of each pixel position and its reliability as the measurement result;
The first time-of-flight distance measurement camera and the second time-of-flight distance measurement camera generate a depth value of each pixel position and reliability thereof, which are the measurement results as the spatial information distance measurement camera;
The depth synthesis unit converts a depth value having the highest reliability among the measurement results of the first time-of-flight ranging camera and the second time-of-flight ranging camera, respectively, and the measurement result of the spatial information ranging camera, to a pixel An imaging device characterized in that a depth value of each pixel position is determined by selecting each pixel position. According to claim 1,
The light source projection unit,
A light source generator for generating intensity-modulated light sources according to a predetermined modulation signal and a vertical synchronization signal;
and an optical element for generating the spatial pattern light by deforming the light source corresponding to a spatial position. According to claim 1,
The imaging device according to claim 1, wherein the depth synthesis unit determines a depth value of each pixel position based on the degree of reliability in the measurement result. According to claim 3,
The imaging device according to claim 1 , wherein the depth synthesis unit determines a depth value at each pixel position by selecting a depth value having the highest reliability among the measurement results for each pixel. According to claim 1,
The first optical time-of-flight distance measuring camera and the spatial information distance measuring camera measure the distance to the subject based on the optical time-of-flight of the modulation component included in the reflected light, and the space included in the reflected light An imaging device characterized by being an integrated camera that measures a distance to the subject based on triangulation calculation for information. delete delete delete delete

Images