JP7025177B2 - Imaging device - Google Patents

Description

Embodiments of the present invention relate to an image pickup apparatus.

In the past, CCD (Charge Coupled Device) image sensors were the mainstream in image pickup devices, for example, endoscope devices, but in recent years, CMOS (CMOS) has advantages such as cost reduction, single power supply, and low power consumption. Complementary Metal Oxide Semiconductor) Image sensors have become mainstream. In CMOS image sensors, the rolling shutter method is generally adopted in many cases, but the global shutter method may also be adopted.

Japanese Unexamined Patent Publication No. 6-225317 Japanese Unexamined Patent Publication No. 11-331857

An object to be solved by the present invention is to provide an image pickup apparatus capable of ensuring sufficient image quality for observation.

The image pickup apparatus of the embodiment is arranged in a matrix, and includes a plurality of pixels that generate an electric signal by receiving light incident along the optical axis in a light receiving period generated at a fixed cycle, and the plurality of pixels. A rolling shutter method in which exposure is started sequentially at least one row from the first row to the last row, and the operation of outputting a video signal based on the electric signal in order from the row where the exposure is completed is repeated every frame. The image sensor, a relative position changing portion that changes the relative position between the optical axis and the image sensor during a non-light receiving period in which the plurality of pixels do not receive light, and a state in which the relative position is changed. The image generation unit that generates image data based on the video signal obtained and the video signal obtained in a state where the relative position is not changed, and the light receiving period are common to the exposure period of each row of the plurality of pixels. An image sensor driving unit for driving the image sensor is provided so as to be within the global exposure period, and a global exposure period in a state where the relative position is changed and a global exposure period in a state where the relative position is not changed. Are alternately arranged, and the image data is output to the outside in a period twice the period of the one frame .

FIG. 1 is a diagram showing a configuration example of an image pickup system including the image pickup apparatus according to the first embodiment. FIG. 2 is a diagram showing an example of pixel arrangement and relative position change of the image sensor according to the first embodiment. FIG. 3 is a diagram showing another example of pixel arrangement and relative position change of the image sensor according to the first embodiment. FIG. 4 is a diagram showing an example of the relative position changing mechanism according to the first embodiment. FIG. 5 is a diagram showing another example of the relative position changing mechanism according to the first embodiment. FIG. 6 is a diagram for explaining an example of the operation of the image pickup apparatus according to the comparative example. FIG. 7 is a diagram for explaining an example of the operation of the image pickup apparatus according to the first embodiment. FIG. 8 is a diagram showing a configuration example of an image pickup system including the image pickup apparatus according to the second embodiment. FIG. 9 is a diagram showing an example of the color scheme of the bayer filter.

Hereinafter, each embodiment of the image pickup apparatus will be described with reference to the drawings. The embodiment is not limited to the following contents. Further, in principle, the contents described in one embodiment or modification are similarly applied to other embodiments or modifications.

(First Embodiment)
FIG. 1 is a diagram showing a configuration example of an image pickup system 1 including the image pickup apparatus 10 according to the first embodiment. As shown in FIG. 1, the image pickup system 1 according to the first embodiment includes an image pickup device 10, a light source 30, and an optical fiber 31.

The image pickup apparatus 10 is used, for example, as a medical rigid endoscope, and is an apparatus for imaging the inside of the subject 100. The image pickup apparatus 10 includes a scope 11, a camera head 12, a camera cable 13, and a CCU (Camera Control Unit) 14.

The scope 11 is inserted into the body of the subject 100 when imaging is performed. An objective lens 11a is provided at the tip of the scope 11. The scope 11 has a rigidity that does not bend.

The camera head 12 includes a color separation optical system 12a, image sensors 12b, 12c, 12d, an image sensor control circuit 12g, and a relative position changing mechanism 12h. The color separation optical system 12a is, for example, a three-decomposition dichroic prism. The color separation optical system 12a, for example, splits the incident light into red (R) light, green (G) light, and blue (B) light. The image sensors 12b, 12c, 12d are, for example, CMOS image sensors. The image sensor 12b corresponds to red color, for example, and is attached to the emission surface of the red spectrum of the color separation optical system 12a. The image sensor 12c corresponds to green, for example, and is attached to the emission surface of the green spectrum of the color separation optical system 12a. The image sensor 12d corresponds to, for example, blue, and is attached to the emission surface of the blue spectrum of the color separation optical system 12a. The end surface of the color separation optical system 12a to which the image pickup surfaces of the image sensors 12b, 12c, and 12d are attached is formed so as to substantially coincide with the image formation surface of the objective lens 11a on the optical path.

Each of the image sensors 12b, 12c, and 12d includes a plurality of pixels (imaging elements). The plurality of pixels are arranged in a matrix on the imaging surface of each image sensor 12b, 12c, 12d. Each pixel of the image sensor 12b, 12c, 12d generates a video signal (electrical signal) by receiving light by drive control by the image sensor control circuit 12g, and outputs the generated video signal. For example, each pixel of the image sensors 12b, 12c, and 12d outputs a video signal by receiving the return light (reflected light) of the light irradiated to the internal tissue of the subject 100 by the light source 30.

For example, the image sensors 12b, 12c, and 12d start exposure sequentially at least one row from the first row to the last row of a plurality of pixels, and output a video signal in order from the row where the exposure is completed. Is a rolling shutter type image sensor that repeats every frame (image). Here, exposure means, for example, that a pixel can store an electric charge. Light reception means that it actually receives light and accumulates electric charges. The image sensors 12b, 12c, and 12d output video signals to the CCU 14 via, for example, the camera cable 13. An analog signal or a digital signal video signal is output from the image sensors 12b, 12c, and 12d.

The image sensor control circuit 12g is an image based on a control signal output from the control circuit 14a described later and various synchronization signals such as a horizontal synchronization signal and a vertical synchronization signal output from the timing signal generation circuit 14c described later. The sensors 12b, 12c, and 12d are driven and controlled.

The relative position changing mechanism 12h has an optical axis of incident light from the objective lens 11a and image sensors 12b, 12c, based on the control signal output from the control circuit 14a and the synchronization signal output from the timing signal generation circuit 14c. The relative position of 12d is periodically changed (optical axis shift). The relative position changing mechanism 12h includes, for example, a driving mechanism such as a motor or an actuator. The relative position changing mechanism 12h is an example of the relative position changing unit.

FIG. 2 is a diagram showing an example of pixel arrangement and relative position change of the image sensors 12b, 12c, 12d according to the first embodiment. On the left side of FIG. 2, the arrangement of the pixels (positions relative to the image formation by the incident light) of the image sensors 12b, 12c, and 12d before the relative position change is schematically shown. In this example, by the spatial pixel shift method, the pixels of the image sensor 12c corresponding to green (Gch), the pixels of the image sensor 12b corresponding to red (Rch), and the image sensor 12d corresponding to blue (Bch) are used. The pixels are arranged so as to be offset by half a pixel (1/2 pixel) in the vertical direction and the horizontal direction. The relative position change mechanism 12h shifts all pixels by half a pixel in the vertical and horizontal directions from the state shown on the left side of FIG. 2 by changing the relative position (optical axis shift) as shown on the right side of FIG. .. That is, the relative position changing mechanism 12h changes the relative positions of the optical axis and the image sensors 12b, 12c, 12d by the distance corresponding to the half pixel in the diagonal direction of the two-dimensional arrangement of the plurality of pixels. By returning the relative position, the relative position returns from the state on the right side of FIG. 2 to the state on the left side.

FIG. 3 is a diagram showing another example of pixel arrangement and relative position change of the image sensors 12b, 12c, 12d according to the first embodiment. On the left side of FIG. 3, the arrangement of the pixels (positions relative to the image formation by the incident light) of the image sensors 12b, 12c, and 12d before the relative position change is schematically shown. In this example, the pixels of the image sensor 12c corresponding to green (Gch), the pixels of the image sensor 12b corresponding to red (Rch), and the pixels of the image sensor 12d corresponding to blue (Bch) are located at the same position. Have been placed. That is, the spatial pixel shift method is not used. The relative position change mechanism 12h shifts all pixels by half a pixel in the vertical and horizontal directions from the state shown on the left side of FIG. 3 by changing the relative position (optical axis shift) as shown on the right side of FIG. .. By returning the relative position, the relative position returns from the state on the right side of FIG. 3 to the state on the left side.

FIG. 4 is a diagram showing an example of the relative position changing mechanism 12h according to the first embodiment. FIG. 4 shows a case where the optical axis shift is performed by, for example, a parallel plate 12i formed of a light-transmitting material such as glass, which is arranged at a position intersecting the optical axis. In the case of the camera head 12 of FIG. 1, for example, the parallel plate 12i is arranged at a position intersecting the optical axis between the color separation optical system 12a and the scope 11. In the upper part of FIG. 4, the parallel plate 12i is maintained perpendicular to the optical axis of the incident light, and the incident light is incident on the color separation optical system 12a as it is. In the lower part of FIG. 4, the parallel plate 12i is tilted by the drive control of the relative position change mechanism 12h, so that the optical axis of the incident light is translated and the incident light is incident on different positions of the color separation optical system 12a.

FIG. 5 is a diagram showing another example of the relative position changing mechanism 12h according to the first embodiment. FIG. 5 shows, for example, a case where the optical axis shift is performed by translation of the color separation optical system 12a itself to which the image sensors 12b, 12c, and 12d are attached. In the case of the camera head 12 of FIG. 1, for example, a mechanical drive mechanism is provided on a pedestal or the like that supports the color separation optical system 12a. In the upper part of FIG. 5, the incident light is incident at a predetermined position (center portion or the like) of the color separation optical system 12a, for example. In the lower part of FIG. 5, the color separation optical system 12a is moved in parallel by the drive control of the relative position change mechanism 12h, so that the optical axis of the incident light is relatively parallel movement, and the incident light is the color separation optical system 12a. Incident at different positions.

Returning to FIG. 1, the camera cable 13 is a cable accommodating a signal line for transmitting and receiving a video signal, a control signal, and a synchronization signal between the camera head 12 and the CCU 14.

The CCU 14 generates image data based on the video signal output from the camera head 12, and outputs the image data to the display 101 connected to the CCU 14. The CCU 14 includes a control circuit (MPU: Micro-Processing Unit) 14a, an image processing circuit (DSP: Digital Signal Processor) 14b, a timing signal generation circuit (TG: Timing Generator) 14c, and an output circuit (Output I / F). It includes a 14d and a storage circuit 14e. When the image sensors 12b, 12c, and 12d output a video signal of an analog signal, the CCU 14 includes an AD (Analog to Digital) converter (not shown) or the like. Such an AD converter converts, for example, a video signal of an analog signal output from the image sensors 12b, 12c, 12d into a video signal of a digital signal. That is, the AD converter converts an analog-format video signal into a digital-format video signal.

The control circuit 14a controls various components of the image pickup apparatus 10. For example, the control circuit 14a outputs a control signal to the image sensor control circuit 12g, the relative position change mechanism 12h, the image processing circuit 14b, and the timing signal generation circuit 14c, and outputs the control signal to the camera head 12, the image processing circuit 14b, and the timing signal. The generation circuit 14c is controlled. The control circuit 14a, for example, reads a control program of the image pickup device 10 stored in the storage circuit 14e and executes the read control program to execute a control process for controlling various components of the image pickup device 10. Alternatively, the control circuit 14a has a storage circuit (not shown) inside, and executes a control program stored in the storage circuit.

The image processing circuit 14b is a video signal that is a digital signal based on a control signal output from the control circuit 14a and various synchronization signals such as a horizontal synchronization signal and a vertical synchronization signal output from the timing signal generation circuit 14c. Is subjected to various signal processing. For example, when the image processing circuit 14b outputs a digital signal video signal from the image sensors 12b, 12c, 12d, the image processing circuit 14b performs various signal processing on the video signal. Alternatively, when the video signal of the analog signal is output from the image sensors 12b, 12c, 12d, the image processing circuit 14b performs various signal processing on the video signal converted into a digital format by the AD converter. The image processing circuit 14b is an example of an image generation unit.

When the spatial pixel shift method (arrangement as shown on the left side of FIG. 2) is used for the arrangement of the image sensors 12b, 12c, 12d as the signal processing, the image processing circuit 14b considers the pixel shift. Perform pixel interpolation processing. This pixel interpolation processing is performed, for example, for removing moiré, and by using a signal imaged at a spatially displaced position in the processing, the false signal component that causes moiré can be canceled out. It will be possible. Further, the image processing circuit 14b performs pixel interpolation processing based on the video signals before and after the relative position is changed by the relative position changing mechanism 12h.

The image processing circuit 14b outputs an RGB signal obtained as a result of signal processing on the video signal to the output circuit 14d as image data indicating an image displayed on the display 101. In this way, the image processing circuit 14b generates and outputs image data based on the video signal converted into the digital format.

The timing signal generation circuit 14c transmits various synchronization signals such as a horizontal synchronization signal and a vertical synchronization signal based on a clock signal generated by an oscillation circuit (not shown), and other synchronization signals for synchronizing the entire image pickup apparatus 10. Generate. Then, the timing signal generation circuit 14c outputs various generated synchronization signals to the image sensor control circuit 12g, the relative position change mechanism 12h, the control circuit 14a, and the image processing circuit 14b.

Further, the timing signal generation circuit 14c generates a light source control signal based on the clock signal and the control signal output from the control circuit 14a. The light source control signal is a control signal for controlling the light emitted from the light source 30 and synchronizing the entire image pickup apparatus 10. Here, in conjunction with the control timing of the relative position change mechanism 12h, the light source 30 is turned off in the transitional period of the relative position change and return, and is controlled to be turned on in the stable period of the relative position. Further, when a rolling shutter type image sensor is used, distortion of the video signal is prevented by synchronizing the lighting period of the light source 30 with the global exposure period common to the exposure periods of each row of a plurality of pixels. Then, the timing signal generation circuit 14c outputs the generated light source control signal to the light source 30. The timing signal generation circuit 14c is realized by, for example, FPGA (Field Programmable Gate Array). The control circuit 14a, the timing signal generation circuit 14c, and the image sensor control circuit 12g are examples of the image sensor drive unit.

The output circuit 14d outputs the RGB signal (image data) output from the image processing circuit 14b to the display 101. As a result, the display 101 displays an image based on the RGB signal. The output circuit 14d is realized by, for example, an HDMI (High-Definition Multimedia Interface) (registered trademark) driver IC (Integrated Circuit), an SDI (Serial Digital Interface) driver IC, or the like.

The storage circuit 14e is realized by, for example, a RAM (Random Access Memory), a semiconductor memory element such as a flash memory, a hard disk, an optical disk, or the like. The storage circuit 14e stores various programs. For example, the storage circuit 14e stores a control program executed by the control circuit 14a when the control circuit 14a does not have a storage circuit inside. Further, a video signal is stored in the storage circuit 14e by the image processing circuit 14b.

The light source 30 emits white light based on the light source control signal. The light source 30 includes a drive circuit 30a and a white LED (Light Emitting Diode) 30b. The drive circuit 30a performs drive control for driving and lighting the white LED 30b based on the light source control signal output from the timing signal generation circuit 14c. The white LED 30b emits white light by drive control by the drive circuit 30a. The white light is, for example, visible light. The optical fiber 31 guides the white light from the light source 30 to the tip of the scope 11 and emits it from the tip of the scope 11.

Further, in the present embodiment, as the image sensors 12b, 12c, 12d, so-called multi-frame exposure image sensors are used. That is, in the present embodiment, the exposure period of the image sensors 12b, 12c, and 12d is the same as the period during which one frame of video signal is output from the image pickup apparatus 10 to the display 101. Then, the control circuit 14a outputs a control signal for controlling the image sensors 12b, 12c, and 12d to the image sensor control circuit 12g so that the read-out period is shorter than the exposure period.

To give a specific example, the frame rate of the video signal (image) output from the image pickup apparatus 10 to the display 101 is A [fps (frame per second)]. In this case, an image sensor whose reading period can be set to 1 / (MA) [s] is used as the image sensors 12b, 12c, 12d of the image pickup apparatus 10. That is, image sensors capable of outputting the video signal of each line for each 1 / (M · n · A) [s] are used as the image sensors 12b, 12c, 12d. However, "M" is a number larger than 1, and "n" is the number of rows of pixels of the image sensors 12b, 12c, and 12d. Hereinafter, the case of M = 2 will be described as an example, but M may be a number different from 2 and larger than 1.

Then, the control circuit 14a outputs a control signal for outputting the video signal of one frame to the image sensors 12b, 12c, 12d in the readout period 1 / (2A) [s] shorter than the exposure period 1 / A [s]. It is output to the image sensor control circuit 12g.

In the example described later, the case of A = 60 will be described as an example. That is, it is assumed that the exposure period and the period in which one frame of video signal is output from the image pickup apparatus 10 to the display 101 are the same 1/60 [s], and the readout period is 1/120 [s]. ..

The configuration example of the image pickup apparatus 10 of the image pickup system 1 according to the first embodiment has been described above. Here, the image pickup apparatus according to the comparative example will be described. The image pickup apparatus according to the comparative example does not have the relative position changing mechanism 12h in the first embodiment described above, and does not change the relative positions of the optical axis of the incident light and the image sensors 12b, 12c, 12d. Further, the image pickup apparatus according to the comparative example continuously lights up by the light source 30 and does not repeatedly turn on and off. Further, the image pickup apparatus according to the comparative example corresponds to the fact that the relative position is not changed, and performs pixel interpolation processing based on the video signals before and after the relative position change in the signal processing in the image processing circuit 14b. do not have. However, in the comparative example, it is assumed that the spatial pixel shift method in the three-plate system as shown on the left side of FIG. 2 is used, and the pixel interpolation process in consideration of the pixel shift is performed in the signal processing in the image processing circuit 14b. It shall be. Further, in the comparative example, so-called multi-frame exposure is not performed, but one-frame exposure is performed.

FIG. 6 is a diagram for explaining an example of the operation of the image pickup apparatus according to the comparative example. FIG. 6 shows the light source control signal of the image pickup device according to the comparative example, the exposure timing of each line of a plurality of pixels included in the image sensor, the output timing of the video signal output from the image sensor, and the output timing from the image pickup device. An example of the relationship with the output timing of the video signal is shown. As shown in FIG. 6, in the image pickup apparatus according to the comparative example, the light source control signal is a value instructing lighting at all times while imaging is performed, and the light source is maintained in the lighting state. Further, as the exposure timing of the first frame, the exposure is sequentially started line by line from the first line to the last line of the plurality of pixels. In the image pickup apparatus according to the comparative example, the exposure period is 1/60 [s].

Then, as shown in FIG. 6, in the image pickup apparatus according to the comparative example, video signals are output in order from the row at which the exposure is completed. That is, the video signals of the first frame are sequentially output line by line from the first line to the last line. Here, in the image pickup apparatus according to the comparative example, the period (reading period) in which the video signal of one frame is output from the image sensor is 1/60 [s], which is the same as the exposure period. Then, as shown in FIG. 6, the same processing is performed for the second and subsequent frames.

Here, in the image pickup apparatus according to the comparative example, the spatial pixel shift method in the three-plate system is used, and the pixel interpolation process in consideration of the pixel shift is performed. Therefore, moire is effectively removed for images that are close to achromatic colors and have the same ratio of red, green, and blue. However, the farther away from the achromatic color, the smaller the effect of removing moire, and the deterioration of the image quality may occur.

Further, in the image pickup apparatus according to the comparative example, the image sensor has a period of receiving light (light receiving period) in all rows of a plurality of pixels, which is the same period as the exposure period. Therefore, from the first row to the last row, the light receiving period of each row is sequentially shifted in the time axis direction. As described above, in the image pickup apparatus according to the comparative example, since the light receiving period is different for each row, the image may be distorted for a fast-moving subject.

As another comparative example, a three-panel dual green system or a four-panel imaging system may be applied to the image sensor. The three-plate dual green method uses two image sensors that are half a pixel offset from each other with respect to the green spectrum, and one image sensor is also used for red and blue by a filter. It can be abbreviated as "G1 / G2 / RB". The four-panel imaging method uses two image sensors that are half a pixel offset from each other with respect to the green spectrum, and one image sensor for each of red and blue. It can be abbreviated as "G2 / R / B".

Here, in the three-plate dual green method, moire can be removed for green and achromatic images, but the effect is small for other colors such as red or blue monochromatic systems, and red and blue are produced. Since one image sensor is also used, there is a problem that the number of red and blue pixels is small and the resolution is lowered. In the four-panel imaging method, moire can be removed for green and achromatic images, but there is a problem that the effect is small for other colors such as red or blue monochromatic systems. Further, in any of the comparative examples, the distortion of the image caused by the rolling shutter method cannot be eliminated. As described above, the comparative example has a problem that the image quality is not sufficient for observing a user such as a doctor who observes an image.

Therefore, the image pickup apparatus 10 according to the first embodiment performs the following operations so as to ensure sufficient image quality for the user to observe under the above-described configuration.

FIG. 7 is a diagram for explaining an example of the operation of the image pickup apparatus 10 according to the first embodiment. FIG. 7 shows a light source control signal given to the light source 30 from the timing signal generation circuit 14c of the image pickup apparatus 10 according to the first embodiment, and a state of relative position change (optical axis shift) by the relative position change mechanism 12h. The exposure timing of each row of a plurality of pixels included in the image sensors 12b, 12c, 12d, the output timing of the video signal output from the image sensors 12b, 12c, 12d, and the output timing of the video signal output from the output circuit 14d. An example of the relationship is shown.

The light source control signal given to the light source 30 from the timing signal generation circuit 14c is, for example, a rectangular signal having a period of 1/60 [s], and repeats low-level and high-level signal output every 1/120 [s]. .. For example, from time t0 to time t1 shown in FIG. 7, it becomes a low-level signal instructing to turn off, and from time t1 to time t2, it becomes a high-level signal instructing to turn on, and this is a high-level signal instructing to turn on after that. Repeated. The drive circuit 30a of the light source 30 emits white light to the white LED 30b only during the period when the light source control signal is at a high level, based on the light source control signal.

The relative position change (optical axis shift) by the relative position change mechanism 12h is performed, for example, in a cycle of 1/60 [s], and shifts to the shift state (0.5 pixel shift) and returns to the normal state (No shift). And are repeated. For example, the transition to the shift state is started at the time t0 shown in FIG. 7, and the transition is completed by the time t1. Further, the return to the normal state is started after the lapse of time t2, and the return is completed by time t3.

The exposure timing of each row of a plurality of pixels included in the image sensors 12b, 12c, and 12d is, for example, 1/60 [s] of two 1/120 [s] frames as one frame by double-speed two-frame exposure. It is used as. For example, the time t0 to the time t2 shown in FIG. 7 is the exposure period of the first row of a certain frame, the exposure is started sequentially for each row thereafter, and the time t1 to the time t3 is the last of the frame. The exposure period of the row. Due to the light source control signal, only the light receiving period (Shift0) is from time t1 to time t2 in this exposure period, and this period is a period common to the exposure period of each row of the plurality of pixels of the image sensors 12b, 12c, and 12d ( Global exposure period). Therefore, even when the subject moves quickly, distortion of the image is prevented. Further, the exposure period of one frame is followed by the exposure period of the next frame, and these are repeated. That is, the exposure of the first row of the next frame starts from the time t2 when the exposure of the first row of a certain frame ends. The light receiving period in this frame is represented by Normal1. The light receiving period of the next frame is represented by Shift1, and the light receiving period of the next frame is represented by Normal2.

The output timing of the video signal output from the image sensors 12b, 12c, 12d is, for example, from the end time of the exposure in the first row of the frame to the end time of the exposure in the last row. For example, for the first frame shown in FIG. 7, a video signal (Shift0) is output from the image sensors 12b, 12c, and 12d during the period from time t2 to time t3. Since the video signals output from the image sensors 12b, 12c, and 12d are used when processing the video signals of the following frames, they are stored by the storage circuit 14e at least until the processing for the next frame is performed. The video signal in the next frame is represented by Normal1. The video signal of the next frame is represented by Shift1, and the video signal of the next frame is represented by Normal2.

The video signals output from the output circuit 14d as the output of the image pickup apparatus 10 are the video signals acquired from the image sensors 12b, 12c, 12d in a state where the relative position change (optical axis shift) is not performed, and the relative position change (relative position change (). It is a video signal obtained by pixel interpolation processing from the video signal acquired from the image sensors 12b, 12c, 12d in the state where the optical axis shift) is performed. The output timing of the video signal output from the output circuit 14d is one frame from the time when the video signal is output from the image sensors 12b, 12c, 12d to the time when the video signal of the next frame is output. In the example of FIG. 7, the video signal Video0 ((Normal0 + Shift0) / 2) is output from time t2 to time t4. (Normal0 + Shift0) / 2 indicates that the video signal is obtained by pixel interpolation processing from the video signal Normal0 and the video signal Shift0. Since it is the first frame, the video signal Normal0 in the normal state does not exist, and the video signal Video0 is based only on the video signals Shift0 acquired from the image sensors 12b, 12c, and 12d between the time t2 and the time t3. It becomes a thing. Further, from time t4 to time t6, the video signals Shift0 acquired from the image sensors 12b, 12c, 12d between time t2 and time t3, and acquired from the image sensors 12b, 12c, 12d between time t4 and time t5. The video signal Video1 ((Normal1 + Shift0) / 2) based on the obtained video signal Normal1 is output. The same applies to the following frames, and the video signal Video2 ((Normal1 + Shift1) / 2) and the video signal Video3 ((Normal2 + Shift1) / 2) are output in order.

Moire is effectively removed by the pixel interpolation processing performed by the image processing circuit 14b, and high image quality is achieved. That is, by using the signal at the position shifted by half a pixel obtained by changing the relative position, it is possible to remove moire for all colors and improve the image quality. In other words, in the case of the three-plate type of red (R), green (G), and blue (B), the six-plate type (G1 / G2 / R1 / R2 / B1 / B2) is pseudo by changing the relative position. It will be realized, and it will be possible to remove moire for video signals of all colors, not limited to achromatic colors.

(Variation example of the first embodiment)
In the first embodiment described above, an example applied to a rigid endoscope has been described, but the scope may be applied to a flexible endoscope having flexibility. Moreover, the application target is not limited to medical use.

Further, in the first embodiment described above, an example using a rolling shutter type image sensor has been described, but a global shutter type image sensor having a common global exposure period for each row of a plurality of pixels may be used. .. The image sensor of the global shutter system starts the exposure of a plurality of pixels at the same time, and repeats the process of collectively outputting the electric signals generated by the plurality of pixels for each frame.

Further, in the above-mentioned first embodiment, it is assumed that the image is taken in a space such as the inside of the body where light is not received in the absence of external lighting, but it can also be applied to the image in a space with external light. .. In this case, for example, in the camera head 12, a mechanical or electric shutter is provided in the middle of the path of the optical axis from the objective lens 11a of the scope 11 to the color separation optical system 12a. The mechanical shutter is composed of, for example, an operating member that opens and closes (transmits / shields) an optical path. The electric shutter is composed of, for example, a liquid crystal filter capable of switching between transmission and light shielding. When used only in a space with outside light, the light source 30 and the optical fiber 31 are unnecessary.

Then, the shutter shields light during the transitional period in which the relative position change is performed by the relative position change mechanism 12h based on the signal having the same timing as the light source control signal generated by the timing signal generation circuit 14c, and the relative position change is completed. Transmission is performed only during the global exposure period in which light is received.

(Second embodiment)
Next, the image pickup apparatus according to the second embodiment will be described. In the second embodiment, the same components as those in the first embodiment may be designated by the same reference numerals and description thereof may be omitted. The image pickup apparatus according to the second embodiment is different from the first embodiment in that a single plate type image sensor is used for the camera head.

FIG. 8 is a diagram showing a configuration example of an image pickup system 1 including the image pickup apparatus 10 according to the second embodiment. In FIG. 8, the camera head 12 includes a bayer filter 12e, an image sensor 12f, an image sensor control circuit 12g, and a relative position changing mechanism 12h. The bayer filter 12e is arranged on the exposed surface of the image sensor 12f, and limits the color incident on each pixel of the image sensor 12f to any of red, green, blue, and the like. FIG. 9 is a diagram showing an example of the color scheme of the bayer filter 12e, in which two of the four adjacent pixels are assigned to green and the remaining two are assigned to red and blue.

Returning to FIG. 8, the image sensor 12f is, for example, a CMOS image sensor. The image pickup surface of the image sensor 12f is arranged so as to substantially coincide with the image formation surface of the objective lens 11a of the scope 11. The image sensor 12f includes a plurality of pixels (imaging elements). The plurality of pixels are arranged in a matrix on the image pickup surface of the image sensor 12f. Each pixel of the image sensor 12f generates a video signal (electrical signal) by receiving light by drive control by the image sensor control circuit 12g, and outputs the generated video signal. For example, each pixel of the image sensor 12f outputs a video signal by receiving the return light (reflected light) of the light irradiated to the internal tissue of the subject 100 by the light source 30.

The image sensor 12f may be either the rolling shutter method or the global shutter method described above. Further, a video signal of an analog signal or a digital signal is output from the image sensor 12f.

The image sensor control circuit 12g is the same as that of the first embodiment described above, but since the control target is the single plate type image sensor 12f, there is a difference in control according to the difference in the target.

The relative position change mechanism 12h is the same as that of the first embodiment described above, and the specific method of the relative position change (optical axis shift) is also the same as that shown in FIG. 4 or FIG. The color separation optical system 12a and the image sensors 12b, 12c, 12d are replaced with the bayer filter 12e and the image sensor 12f. However, in the first embodiment described above, the bayer has a color arrangement as shown in FIG. 9, although the distance corresponding to half a pixel (1/2 pixel) is shifted in the diagonal direction (vertical direction and horizontal direction). For the image sensor 12f accompanied by the filter 12e, it is effective to shift the distance corresponding to one pixel in the diagonal direction, the horizontal direction, or the vertical direction in addition to the case of shifting the distance corresponding to half a pixel in the diagonal direction. That is, in the color scheme as shown in FIG. 9, since the same color is used every other pixel, information on an intermediate position for the same color can be obtained by shifting one pixel.

In FIG. 8, the configuration and operation of the CCU 14 are the same as those in the first embodiment described above, but since the control target is the single plate type image sensor 12f, there is a difference in control according to the difference in the target. .. For example, when acquiring an RGB video signal having the same resolution as that of the image sensor 12f, the image processing circuit 14b directly uses the image processing circuit 14b for each pixel based on the video signal output from the pixels around the pixel. An estimation process for estimating each video signal of the remaining two colors that cannot be obtained is executed. Further, when the amount of pixel shift of the relative position change changes, the parameter in the pixel interpolation processing changes.

According to the second embodiment, by using the single plate type image sensor 12f, the camera head 12 can be configured in a small size, and high image quality can be achieved in the small housing.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and variations thereof are included in the scope of the invention described in the claims and the equivalent scope thereof, as are included in the scope and gist of the invention.

1 Image pickup system 10 Image pickup device 12b-12d, 12f Image sensor 12h Relative position change mechanism 14b Image processing circuit

Claims (8)

It is arranged in a matrix and contains a plurality of pixels that generate an electric signal by receiving light incident along the optical axis in a light receiving period generated at a fixed cycle, and includes a plurality of pixels from the first row to the last of the plurality of pixels. A rolling shutter type image sensor that starts exposure sequentially at least one row at a time toward a row and outputs a video signal based on the electric signal in order from the row where the exposure is completed, every frame .
A relative position changing unit that changes the relative position between the optical axis and the image sensor during a non-light receiving period in which the plurality of pixels do not receive light.
An image generation unit that generates image data based on a video signal obtained in a state where the relative position is changed and a video signal obtained in a state where the relative position is not changed .
An image sensor driving unit for driving the image sensor is provided so that the light receiving period is within the global exposure period common to the exposure periods of each row of the plurality of pixels .
The global exposure period in the state where the relative position is changed and the global exposure period in the state where the relative position is not changed are arranged alternately, and the image data is externally arranged in a period twice the period of the one frame. An image pickup device characterized by being output . A claim that either the video signal obtained in the state where the relative position is changed or the video signal obtained in the state where the relative position is not changed is duplicated and used in each image data to be continuously output. Item 1. The image pickup apparatus according to Item 1. The image pickup device further includes a shutter arranged in the middle of the path of the optical axis.
The image pickup apparatus according to claim 1 , wherein the shutter shields light during a transitional period in which the relative position is changed, and transmits light during the global exposure period . A light source that emits light during the light receiving period is provided.
The image pickup apparatus according to any one of claims 1 to 3, wherein the plurality of pixels receive the reflected light of the light emitted from the light source. The image sensor is a three-plate type sensor that receives light dispersed in each of the green, red, and blue colors.
One of claims 1 to 4, wherein the relative position changing unit changes the relative position of the optical axis and the image sensor by a distance corresponding to a half pixel in a diagonal direction of a two-dimensional arrangement of the plurality of pixels. The imaging device according to one. The image sensor is a single-panel sensor in which a filter of any color of green, red, or blue is arranged on the plurality of pixels.
The relative position changing unit changes the relative position of the optical axis and the image sensor by a distance corresponding to one pixel in the vertical, horizontal or diagonal direction of the two-dimensional arrangement of the plurality of pixels, according to claims 1 to 1. The image pickup apparatus according to any one of 4. The relative position changing portion changes the relative position of the optical axis and the image sensor by driving a light transmissive flat plate arranged at a position intersecting the optical axis, according to claims 1 to 6. The imaging device according to any one. The imaging device according to any one of claims 1 to 6, wherein the relative position changing unit changes the relative position between the optical axis and the image sensor by driving the image sensor.

Images