JP2018007840A - Image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of acquiring a parallax image which can perform a detailed observation in a near point observation.SOLUTION: An image processing device generates a parallax image on the basis of first and second images in which the same subject is imaged from different directions, and it includes: a distance information generation unit for generating distance information which shows a distance, which is a distance acquired from the first and second images, from an imaging surface to the subject; a shift amount setting unit for setting a shift amount in the parallax direction of the first image and the second image on the basis of the distance information; and a parallax image generation unit for generating a parallax image on the basis of the first and second images and the shift amount set by the shift amount setting unit.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing apparatus.

  2. Description of the Related Art In recent years, there has been known an image creation method for capturing a subject and acquiring a stereoscopic image from two image data for left eye and right eye having parallax with each other (see, for example, Patent Document 1). In Patent Document 1, by calculating the depth of each point based on the shift amount of the corresponding point between the left-eye image and the right-eye image, the depth is obtained for all corresponding pixel positions, and based on the calculated depth. Thus, a parallax image for stereoscopic display is created by moving each pixel position.

  By the way, even in an endoscope system in which an endoscope used in the medical field or the like and a processing device (processor) are detachable, it is desired to observe the observation target as a stereoscopic image for facilitating diagnosis and examination. There is a request. As a technique for meeting this requirement, an endoscope including two optical systems for the left eye and right eye and an image sensor that receives light from each optical system is known. The processor to which the endoscope is attached receives left-eye and right-eye image data from the attached endoscope, and generates a parallax image from the two image data.

Japanese Patent No. 4061305

  In near-point observation with a small distance to the observation target, there is a demand for observation of unevenness of a living tissue, for example. However, when the depth calculation method disclosed in Patent Document 1 is applied to a conventional endoscope system, the depth becomes small when the distance to the observation target is small, so that a parallax image with a small parallax between the two images is generated. Is done. As a result, in the parallax image in which the observation target is shown at the near point, the unevenness may not be fully expressed in the displayed stereoscopic image, and it may not be possible to observe in detail. For this reason, there has been a demand for a technique for generating a parallax image that allows detailed observation even in near-point observation.

  The present invention has been made in view of the above, and an object thereof is to provide an image processing apparatus capable of acquiring a parallax image capable of performing detailed observation in near-point observation.

  In order to solve the above-described problems and achieve the object, an image processing apparatus according to the present invention generates an image of parallax based on first and second images obtained by imaging the same subject from different directions. A distance information generating unit that generates a distance information that is a distance obtained from the first and second images and that indicates a distance from the imaging surface to the subject, and based on the distance information, the first image The parallax image based on the shift amount setting unit for setting the shift amount in the parallax direction between the first image and the second image, the first and second images, and the shift amount set by the shift amount setting unit. And a parallax image generation unit to be generated.

  In the image processing apparatus according to the present invention as set forth in the invention described above, the shift amount setting unit determines whether or not the distance is smaller than a preset distance based on the distance information. Based on the result, a shift amount in the parallax direction between the first image and the second image is set.

  Further, in the image processing apparatus according to the present invention, in the above invention, the distance information generation unit may compare the distance information when the distance is smaller than a preset distance with a case where the distance is equal to or greater than the preset distance. The shift amount is set to a large value.

  In the image processing apparatus according to the present invention, in the above invention, the distance information generation unit sequentially generates the distance information for the input first and second images, and the shift amount setting unit includes: It is determined whether or not the distance is smaller than the preset distance based on the distance information generated by the distance information generation unit, and the number of consecutive frames determined to be smaller than the preset distance is The shift amount is set based on the distance information when the distance is equal to or greater than the threshold, and the number of consecutive frames determined to be smaller than the preset distance is preset when the number of consecutive frames is smaller than the threshold. It is characterized in that it is set to the shift amount.

  In the image processing apparatus according to the present invention, in the above invention, the near point shift amount set when the distance is smaller than the preset distance and the distance are more than the preset distance. The far point shift amount set when the distance is set is set in advance, the near point shift amount is larger than the far point shift amount, and the shift amount setting unit sets the distance in advance. And when the set shift amount that is set is larger than the far point shift amount, the near point shift amount and the far point shift amount are calculated from the set shift amount. A value obtained by subtracting a subtraction amount smaller than the difference is set as the shift amount.

  According to the present invention, there is an effect that a parallax image capable of performing detailed observation in near-point observation can be acquired.

FIG. 1 is a diagram showing a schematic configuration of an endoscope system according to an embodiment of the present invention. FIG. 2 is a block diagram showing a schematic configuration of the endoscope system according to the embodiment of the present invention. FIG. 3 is a diagram illustrating a parallax image generated by the endoscope system according to the embodiment of the present invention. FIG. 4 is a flowchart showing image processing performed by the processing apparatus according to the embodiment of the present invention. FIG. 5 is a flowchart illustrating image processing performed by the processing device according to the first modification of the embodiment of the present invention. FIG. 6 is a flowchart illustrating image processing performed by the processing device according to the second modification of the embodiment of the present invention. FIG. 7 is a flowchart showing image processing performed by the processing device according to the third modification of the embodiment of the present invention.

  Hereinafter, modes for carrying out the present invention (hereinafter referred to as “embodiments”) will be described. In the embodiment, a medical endoscope system that captures and displays an image in a subject such as a patient will be described as an example of a system including an image processing apparatus according to the present invention. Moreover, this invention is not limited by this embodiment. Further, in the description of the drawings, the same portions will be described with the same reference numerals.

(Embodiment)
FIG. 1 is a diagram showing a schematic configuration of an endoscope system according to an embodiment of the present invention. FIG. 2 is a block diagram showing a schematic configuration of the endoscope system according to the present embodiment.

  An endoscope system 1 shown in FIGS. 1 and 2 includes an endoscope 2 that captures an image in a subject (hereinafter also referred to as an endoscopic image) by inserting a distal end portion into the subject, It has a light source 3a that generates illumination light emitted from the distal end of the endoscope 2, performs predetermined signal processing on the image signal captured by the endoscope 2, and comprehensively controls the operation of the endoscope system 1 as a whole. The processing apparatus 3 to control and the display apparatus 4 which displays the endoscopic image produced | generated by the signal processing of the processing apparatus 3 are provided.

  The endoscope 2 includes an insertion portion 21 having an elongated shape having flexibility, an operation portion 22 that is connected to a proximal end side of the insertion portion 21 and receives input of various operation signals, and an insertion portion from the operation portion 22. And a universal cord 23 that includes various cables that extend in a direction different from the direction in which 21 extends and are connected to the processing device 3 (including the light source unit 3a).

  The insertion unit 21 receives a light and performs photoelectric conversion to generate a signal to generate a signal. The insertion unit 21 includes an image pickup element 244 in which pixels are arranged in a two-dimensional shape, and a bendable portion formed by a plurality of bending pieces. And a long flexible tube portion 26 connected to the proximal end side of the bending portion 25 and having flexibility. The insertion part 21 is inserted into the body cavity of the subject, and the imaging element 244 images a subject such as a living tissue at a position where external light does not reach.

  The distal end portion 24 is configured by using glass fiber or the like, and forms a light guide path for light emitted from the light source portion 3a, an illumination lens 242 provided at the distal end of the light guide 241, and a condensing first lens. The first optical system 243a and the second optical system 243b are provided at the imaging positions of the optical system 243, receive the light collected by the first optical system 243a and the second optical system 243b, and photoelectrically convert them into electrical signals. And an imaging element 244 (imaging unit) that performs predetermined signal processing.

  The first optical system 243a is configured using one or a plurality of lenses, and is provided in front of the image sensor 244, and forms an image of incident light from a subject. The first optical system 243a may have an optical zoom function that changes the angle of view and a focus function that changes the focus.

  The second optical system 243b is configured using one or a plurality of lenses, and is provided in front of the image sensor 244 to form incident light from a subject with parallax with the first optical system 243a. The second optical system 243b may have an optical zoom function that changes the angle of view and a focus function that changes the focus.

  The image sensor 244 photoelectrically converts light from the first optical system 243a and the second optical system 243b to generate an electrical signal (image signal). Specifically, in the imaging element 244, a plurality of pixels each having a photodiode that accumulates charges according to the amount of light, a capacitor that converts charges transferred from the photodiodes to voltage levels, and the like are arranged in a matrix, Each pixel photoelectrically converts light from the first optical system 243a and the second optical system 243b to generate an electrical signal, and a pixel arbitrarily set as a readout target among a plurality of pixels of the light receiving unit 244a And a reading unit 244b that sequentially reads out the electrical signals generated by and outputs them as image signals. The light receiving unit 244a is provided with a color filter, and each pixel receives light in one of the wavelength bands of the color components of red (R), green (G), and blue (B). The image sensor 244 controls various operations of the distal end portion 24 in accordance with the drive signal received from the processing device 3. The image sensor 244 is realized by using, for example, a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. Further, the image sensor 244 may be a single plate image sensor, or may use a plurality of image sensors such as a three plate type.

  The light receiving unit 244a includes a first region that receives the light imaged by the first optical system 243a and a second region that receives the light imaged by the second optical system 243b, and different optical systems in each region. The imaged light is photoelectrically converted to generate an electrical signal for one frame constituting one image. In the present embodiment, description will be made assuming that the light imaged by the first optical system 243a and the second optical system 243b is received by the same imaging device, but the first optical system 243a and the second optical system are described. Two imaging elements may be provided according to 243b.

  The operation unit 22 includes a bending knob 221 that bends the bending unit 25 in the vertical direction and the left-right direction, and a treatment tool insertion unit 222 that inserts a treatment tool such as a biopsy forceps, an electric knife, and an inspection probe into the body cavity of the subject. In addition to the processing device 3, it has a plurality of switches 223 which are operation input units for inputting operation instruction signals of peripheral devices such as air supply means, water supply means, and screen display control. The treatment tool inserted from the treatment tool insertion portion 222 is exposed from the opening (not shown) via the treatment tool channel (not shown) of the distal end portion 24.

  The universal cord 23 includes at least a light guide 241 and a collective cable 245 in which one or a plurality of signal lines are collected. The collective cable 245 is a signal line for transmitting an image signal, a signal line for transmitting a drive signal for driving the image sensor 244, information including unique information regarding the endoscope 2 (image sensor 244), and the like. Including a signal line for transmitting and receiving. In the present embodiment, the description will be made assuming that an electrical signal is transmitted using a signal line. However, an optical signal may be transmitted, or the endoscope 2 and the processing device 3 may be connected by wireless communication. A signal may be transmitted between them.

  Further, the endoscope 2 includes an imaging information memory 27 that stores imaging information such as imaging conditions of the imaging element 244 of the endoscope 2. The imaging information memory 27 is a memory for recording imaging information of the endoscope 2, and processes the identification information of the endoscope 2 by communication processing with the processing device 3 when the endoscope 2 is attached to the processing device 3. Output to device 3. Alternatively, the connector is provided with a connection pin in accordance with a rule corresponding to the identification information of the endoscope 2, and the processing device 3 includes a connection pin on the processing device 3 side and a connection pin on the endoscope 2 side when the endoscope 2 is mounted. The connection of the endoscope 2 may be recognized based on the connection state. The imaging information includes information such as the number of pixels of the imaging device 244, pixel size, focal length, and addresses of the first and second areas.

  Next, the configuration of the processing device 3 will be described. The processing device 3 includes an image processing unit 31, a parallax image generation unit 32, a distance information generation unit 33, a shift amount setting unit 34, an input unit 35, a storage unit 36, and a control unit 37. The image processing apparatus according to the present invention includes at least an image processing unit 31, a parallax image generation unit 32, a distance information generation unit 33, a shift amount setting unit 34, a storage unit 36, and a control unit 37. Constructed using.

  The image processing unit 31 receives from the endoscope 2 an image signal that is image data representing an endoscopic image captured by the image sensor 244. When receiving an analog image signal from the endoscope 2, the image processing unit 31 performs A / D conversion to generate a digital image signal. Further, when receiving an image signal as an optical signal from the endoscope 2, the image processing unit 31 performs photoelectric conversion to generate a digital image signal.

  The image processing unit 31 performs signal processing such as pixel defect correction, optical correction, color correction, optical black subtraction, noise reduction, white balance adjustment, and interpolation processing on the image signal input from the endoscope 2. In the pixel defect correction, the pixel value of the defective pixel is given based on the pixel values of the pixels around the defective pixel. Optical correction corrects optical distortion of the lens. In the color correction, color temperature correction and color deviation correction are performed. The image processing unit 31 generates a processing signal including a corrected image generated by the signal processing described above.

  In addition, the image processing unit 31 generates images corresponding to light received in the first and second regions of the light receiving unit 244a, respectively. Specifically, the image processing unit 31 includes an image for the left eye that is an image corresponding to an electrical signal obtained by photoelectric conversion of the light imaged by the first optical system 243a by the light receiving unit 244a, and an image formed by the second optical system 243b. A processing signal including the right-eye image that is an image corresponding to the electrical signal obtained by photoelectrically converting the received light by the light receiving unit 244a is generated. The image processing unit 31 inputs the generated processing signal to the parallax image generation unit 32.

  When the processing signal is input from the image processing unit 31, the parallax image generation unit 32 generates a parallax image in which the left-eye image and the right-eye image are shifted from each other to generate parallax, and the signal including the parallax image is generated. On the other hand, signal processing is performed so that the signal can be displayed on the display device 4 to generate an image signal for display. Specifically, the parallax image generation unit 32 performs a zoom process, an enhancement process, a compression process, or the like on the image signal including the left-eye image and the right-eye image, and then sets the shift amount setting unit 34. The right-eye image and the left-eye image are relatively shifted based on the shift amount thus generated to generate a display image signal. The parallax image generation unit 32 generates a composite image in which image information including an endoscopic image corresponding to the processing signal input from the image processing unit 31 and character information related to the image information are superimposed as a display image. The parallax image generation unit 32 transmits the generated display image signal to the display device 4.

  The image processing unit 31 and the parallax image generation unit 32 are a general-purpose processor such as a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array) which is a programmable logic device capable of rewriting processing contents. It is configured using a dedicated processor such as various arithmetic circuits that execute specific functions such as.

Here, the parallax image generated by the parallax image generation unit 32 will be described with reference to FIG. FIG. 3 is a diagram illustrating a parallax image generated by the endoscope system according to the embodiment of the present invention. Parallax image generating unit 32, as shown in FIG. 3, according to the shift amount of the horizontal line D _R, is set a horizontal line D _L, the right-eye image IM _R has included in the image IM _L for the left eye generating a parallax image IM _D by placing alternately staggered Te. Specifically, the shift parallax image generating unit 32, the horizontal line D _L of the odd lines with the image IM _L for the left eye, the even line with the right-eye image IM _R and a horizontal line D _R, is set They are arranged alternately by shifting according to the amount. Such parallax images IM _D is also referred to as line-by-line image. In the present embodiment, the parallax image is a line-by-line image as an example. However, the present invention is not limited to this, and any parallax image may be used.

  When the distance information generation unit 33 receives an image signal, which is image data representing an endoscopic image captured by the imaging element 244, from the endoscope 2, the distance information generation unit 33 measures the distance to the subject using the image included in the image signal. The measurement result is generated as distance information indicating the depth. Specifically, the distance information generation unit 33 measures the distance by triangulation using, for example, the above-described right eye image and left eye image. At this time, the distance information generation unit 33 may measure the depth using the pixel values of the entire image, or may measure the distance using the pixel values of a partial area of the image. In this case, the distance information generation unit 33 measures the distance using the pixel value of the corresponding area after detecting the corresponding area from the other image with respect to the measurement area set in one image. The detection method of the corresponding region at this time can be performed using a known method, for example, template matching such as SSD (Sum of Squared Difference) or SAD (Sum of Absolute Difference). When setting a measurement area for an image, the user may set it via the input unit 35. When a plurality of areas are set, the average information, the mode value, the maximum value, and the minimum value are selected to generate distance information having one distance. By measuring the distance using only a part of the image, the load for generating the distance information by the distance information generating unit 33 can be reduced. At this time, since the generated distance information may fluctuate due to the influence of disturbance, the fluctuation may be suppressed by applying a lag reed filter in order to stabilize the distance information.

The shift amount setting unit 34 sets the shift amounts of the left-eye image and the right-eye image based on the distance information generated by the distance information generation unit 33. The shift amount referred to here is, for example, a shift amount of the right-eye image with respect to the left-eye image, which is a shift amount in the parallax direction. The parallax direction is the left-right direction of the image for the left eye and the image for the right eye, and corresponds to, for example, a direction along the epipolar line. Specifically, the shift amount setting unit 34 first determines whether or not the subject is located at the near point based on the distance to the subject and the distance threshold value stored in the storage unit 36. At this time, the shift amount setting unit 34 sets the shift amount according to the distance if the subject is located at the near point, and is set in advance if the subject is not located at the near point, that is, if the subject is located at the far point. Set the shift amount. The far point shift amount may be zero, for example, or may be a value set according to the distance between the first optical system 243a and the second optical system 243b, for example, the distance between the centers, from the acquired imaging information. Good. For example, if the shift amount is zero, the horizontal line D _L having the image IM _L for the left eye, the parallax image with uniform and horizontal lines D _R having the right-eye image IM _R is generated. In this case, since no parallax occurs, the parallax image observed by the user does not feel a sense of depth. On the other hand, the shift amount is, if set to greater than zero, and horizontal line D _L having the image IM _L for the left eye, and the horizontal line D _R having the right-eye image IM _R, shifted away from each other A parallax image is generated. In this case, parallax occurs according to the shift amount, and the user can observe a parallax image having a sense of depth. In the present embodiment, the shift amount is described as the shift amount of the right-eye image with respect to the left-eye image. However, the shift amount may be the shift amount of the left-eye image with respect to the right-eye image. The relative shift amount between the left-eye image and the right-eye image (each shift amount from the reference position of the left-eye image and the right-eye image) may be used. For example, when the shift amount is the shift amount of the right eye image with respect to the left eye image, the shift amount of the right eye image is set according to the shift amount, and the shift amount of the left eye image is set to zero. Yes.

  The input unit 35 is realized by using a keyboard, a mouse, a switch, and a touch panel, and receives input of various signals such as an operation instruction signal for instructing an operation of the endoscope system 1. The input unit 35 may include a portable terminal such as a switch provided in the operation unit 22 or an external tablet computer.

  The storage unit 36 includes various programs for operating the endoscope system 1, data including various parameters necessary for the operation of the endoscope system 1, image information subjected to predetermined image processing, and the image. Information relating to a synthesis process for generating a synthesized image in which character information relating to information is superimposed, such as so-called on-screen display (OSD) process, is stored. The character information is information indicating patient information, device information, examination information, and the like. Further, the storage unit 36 stores identification information of the processing device 3. Here, the identification information includes unique information (ID) of the processing device 3, model year, specification information, and the like.

  The storage unit 36 also stores various programs including an image acquisition processing program for executing the image acquisition processing method of the processing device 3. Various programs can be recorded on a computer-readable recording medium such as a hard disk, a flash memory, a CD-ROM, a DVD-ROM, or a flexible disk and widely distributed. The various programs described above can also be obtained by downloading via a communication network. The communication network here is realized by, for example, an existing public line network, a LAN (Local Area Network), a WAN (Wide Area Network) or the like, regardless of wired or wireless.

  In addition, the storage unit 36 stores information such as a shift amount of the left-eye image and the right-eye image set in accordance with the distance information, and a threshold value for determining whether or not the subject is located at the near point. An image generation information storage unit 361 is provided. The shift amount is stored in association with the distance acquired as distance information. Further, the parallax image generation information storage unit 361 may store a function representing the relationship between the shift amount and the distance.

  The storage unit 36 having the above configuration is realized by using a ROM (Read Only Memory) in which various programs are installed in advance, a RAM (Random Access Memory) storing a calculation parameter and data of each process, a hard disk, and the like. Is done.

  The control unit 37 is configured by using a general-purpose processor such as a CPU or a dedicated processor such as various arithmetic circuits that execute specific functions such as an ASIC, and drive control of each component including the image sensor 244 and the light source unit 3a, and Input / output control of information for each component is performed. The control unit 37 refers to control information data (for example, readout timing) for imaging control stored in the storage unit 36, and uses the imaging element as a drive signal via a predetermined signal line included in the collective cable 245. To 244. In addition, the control unit 37 performs control to display an image corresponding to the display image signal generated by the parallax image generation unit 32 on the display device 4.

  Next, the configuration of the light source unit 3a will be described. The light source unit 3 a includes an illumination unit 301 and an illumination control unit 302. Under the control of the illumination control unit 302, the illumination unit 301 sequentially switches and emits illumination light with different exposure amounts to the subject (subject). The illumination unit 301 includes a light source 301a and a light source driver 301b.

  The light source 301a is configured using an LED light source that emits white light, one or a plurality of lenses, and the like, and emits light (illumination light) by driving the LED light source. Illumination light generated by the light source 301 a is emitted from the tip of the tip 24 toward the subject via the light guide 241. The light source 301a is realized using any one of an LED light source, a laser light source, a xenon lamp, a halogen lamp, and the like.

  The light source driver 301b causes the light source 301a to emit illumination light by supplying a current to the light source 301a under the control of the illumination control unit 302.

  The illumination control unit 302 controls the amount of power supplied to the light source 301a and the drive timing of the light source 301a based on a control signal (dimming signal) from the control unit 37.

  The display device 4 displays a display image corresponding to the image signal received from the processing device 3 (parallax image generation unit 32) via the video cable. The display device 4 is configured using a monitor such as liquid crystal or organic EL (Electro Luminescence).

  The user observes the parallax image displayed on the display device 4 through glasses having polarization characteristics. Accordingly, the user can observe an image in which the unevenness of the subject is emphasized by observing the image for the left eye with the left eye and the image for the right eye with the right eye.

  Next, image processing performed by the endoscope system 1 will be described. FIG. 4 is a flowchart showing image processing performed by the processing apparatus according to the embodiment of the present invention. Hereinafter, description will be made assuming that each unit operates under the control of the control unit 37.

  First, when the processing device 3 acquires an image signal from the endoscope 2, the image processing unit 31 generates a right eye image and a left eye image (step S101). Specifically, the image processing unit 31 includes an image for the left eye that is an image corresponding to an electrical signal obtained by photoelectric conversion of the light imaged by the first optical system 243a by the light receiving unit 244a, and an image formed by the second optical system 243b. A processing signal including the right-eye image that is an image corresponding to the electrical signal obtained by photoelectrically converting the received light by the light receiving unit 244a is generated. The image processing unit 31 inputs the generated processing signal to the parallax image generation unit 32.

  In step S <b> 102 following step S <b> 101, when the distance information generation unit 33 receives an image signal that is image data representing an endoscopic image captured by the imaging element 244 from the endoscope 2, an image included in the image signal is displayed. The distance to the subject is measured using this, and the measurement result is generated as distance information.

  In steps S103 to S105 following step S102, the shift amount setting unit 34 sets the shift amounts of the left eye image and the right eye image based on the distance information generated by the distance information generation unit 33. Specifically, the shift amount setting unit 34 first determines whether or not the subject is located at a near point based on the distance to the subject and the distance threshold value stored in the storage unit 36 (step). S103). The threshold is set to 4 mm as the distance to the subject, for example. At this time, when the shift amount setting unit 34 determines that the distance is smaller than the threshold value and the subject is located at the near point (step S103: Yes), the process proceeds to step S104. On the other hand, when the shift amount setting unit 34 determines that the distance is equal to or greater than the threshold and the subject is not located at the near point, that is, the subject is located at the far point (step S103: No), the process proceeds to step S105. .

  In step S104, the shift amount setting unit 34 sets a shift amount according to the distance. The shift amount setting unit 34 refers to the parallax image generation information storage unit 361, acquires the shift amount corresponding to the distance, and sets the acquired shift amount as the shift amount of the image. When the setting of the shift amount by the shift amount setting unit 34 is completed, the control unit 37 proceeds to step S106.

  In step S105, the shift amount setting unit 34 sets a preset shift amount. The shift amount setting unit 34 sets, for example, a shift amount set for the far point, for example, zero or a value set according to the distance between the first optical system 243a and the second optical system 243b. Set to. When the setting of the shift amount by the shift amount setting unit 34 is completed, the control unit 37 proceeds to step S106.

  In step S106, the parallax image generation unit 32 performs zoom processing, enhancement processing, compression processing, or the like on the image signal including the image for the left eye and the image for the right eye, and then sets the shift amount setting unit 34. A display image signal including a parallax image obtained by relatively shifting the image for the right eye and the image for the left eye based on the shift amount thus generated is generated.

  In step S107 following step S106, the control unit 37 displays a display image that is a composite image in which image information including the parallax image generated by the parallax image generation unit 32 and character information related to the image information are superimposed on the display device 4. To display.

  In step S108 following step S107, the control unit 37 determines whether or not a new image signal is input from the endoscope 2. For example, when the control unit 37 determines that a new image signal is input from the endoscope 2 (step S108: Yes), the control unit 37 returns to step S101 and repeats the above-described processing. On the other hand, if it is determined that a new image signal has not been input (step S108: No), the above-described image processing ends.

  According to the embodiment of the present invention described above, a depth sensation in which the unevenness of the near point is emphasized is expressed based on the distance information calculated from the left eye image and the right eye image. Since a possible shift amount is set and a parallax image is generated based on this shift amount, it is possible to give a sense of depth to the image displayed on the display device 4. As described above, according to the present embodiment, it is possible to acquire a parallax image that allows detailed observation in near-point observation.

  In the above-described embodiment, the shift amount is set for each frame. However, the shift amount may be set every preset frame. In the above-described embodiment, the shift amount is described as being set corresponding to the distance. However, only two shift amounts, a near-point shift amount and a far-point shift amount, are set in advance. Then, the shift amount setting unit 34 may set any shift amount according to the determination result of the far point / near point.

(Modification 1 of embodiment)
In the first modification, when the frames determined to be near points continue, the shift amount is switched to the shift amount corresponding to the near point. FIG. 5 is a flowchart illustrating image processing performed by the processing device according to the first modification of the embodiment of the present invention. The endoscope system according to the first modification will be described as having the same configuration as the endoscope system 1 described above. Hereinafter, description will be made assuming that each unit operates under the control of the control unit 37.

  First, when the processing device 3 acquires an image signal from the endoscope 2, the image processing unit 31 generates a right-eye image and a left-eye image as described above (step S201).

  In step S202 subsequent to step S201, when the distance information generation unit 33 receives an image signal, which is image data representing an endoscopic image captured by the image sensor 244, from the endoscope 2, the image included in the image signal is displayed. The distance to the subject is measured using this, and the measurement result is generated as distance information.

  In steps S203 to S206 following step S202, the shift amount setting unit 34, based on the distance information generated by the distance information generation unit 33 and the number of consecutive frames determined to be near points, Sets the shift amount of the eye image. Specifically, the shift amount setting unit 34 first determines whether or not the subject is located at a near point based on the distance to the subject and the distance threshold value stored in the storage unit 36 (step). S203). At this time, when the shift amount setting unit 34 determines that the subject is located at the near point (step S203: Yes), the shift amount setting unit 34 proceeds to step S204. In contrast, if the shift amount setting unit 34 determines that the subject is not located at the near point, that is, the subject is located at the far point (step S203: No), the process proceeds to step S206.

  In step S204, the shift amount setting unit 34 counts the number of consecutive frames determined to be near points, and determines whether the number of consecutive frames is equal to or greater than a threshold value. If the shift amount setting unit 34 determines that the number of consecutive frames is smaller than the threshold (step S204: No), the shift amount setting unit 34 proceeds to step S206. In contrast, when the shift amount setting unit 34 determines that the number of consecutive frames is equal to or greater than the threshold (step S204: Yes), the shift amount setting unit 34 proceeds to step S205. The threshold for the number of continuous frames is the number of frames set for switching to a shift amount that increases the parallax, and is preset and stored in the parallax image generation information storage unit 361.

  In step S205, the shift amount setting unit 34 sets a shift amount according to the distance. The shift amount setting unit 34 refers to the parallax image generation information storage unit 361, acquires the shift amount corresponding to the distance, and sets the acquired shift amount as the shift amount of the image. When the shift amount setting unit 34 completes the setting of the shift amount, the control unit 37 proceeds to step S207.

  In step S206, the shift amount setting unit 34 sets a preset shift amount. The shift amount setting unit 34 sets, for example, a shift amount set for the far point, for example, zero or a value set according to the distance between the first optical system 243a and the second optical system 243b. Set to. When the shift amount setting unit 34 completes the setting of the shift amount, the control unit 37 proceeds to step S207.

  In step S207, the parallax image generation unit 32 performs zoom processing, enhancement processing, compression processing, or the like on the image signal including the image for the left eye and the image for the right eye, and then sets the shift amount setting unit 34. A display image signal including a parallax image obtained by relatively shifting the image for the right eye and the image for the left eye based on the shift amount thus generated is generated.

  In step S208 following step S207, the control unit 37 displays a display image that is a composite image in which the image information including the parallax image generated by the parallax image generation unit 32 and the character information related to the image information are superimposed on the display device 4. To display.

  In step S209 following step S208, the control unit 37 determines whether or not a new image signal is input from the endoscope 2. For example, when the control unit 37 determines that a new image signal is input from the endoscope 2 (step S209: Yes), the control unit 37 returns to step S201 and repeats the above-described processing. On the other hand, if it is determined that no new image signal has been input (step S209: No), the above-described image processing ends.

  According to the first modification described above, similar to the above-described embodiment, when the near point is based on the distance information calculated from the left-eye image and the right-eye image, the unevenness of the near point is reduced. Since the shift amount capable of expressing the enhanced sense of depth is set, and the parallax image is generated based on this shift amount, the sense of depth can be given to the image displayed on the display device 4. In the embodiment described above, a parallax image with a large parallax is generated every time it is determined as a near point. However, when the far point and the near point are determined for each frame, the parallax of images displayed continuously May change from frame to frame, making observation difficult. In the first modification, a parallax image with a large parallax is generated when the number of consecutive frames at the near point is equal to or greater than a threshold value, so that it is possible to suppress a large change in parallax continuously for each frame. be able to.

(Modification 2 of embodiment)
In the second modification, when the shift amount setting unit 34 determines that the point is a far point, the shift amount is changed in a stepwise manner to change the shift amount when the determination result shifts from the near point to the far point. Make it smaller. FIG. 6 is a flowchart illustrating image processing performed by the processing device according to the second modification of the embodiment of the present invention. The endoscope system according to Modification 2 will be described as having the same configuration as the endoscope system 1 described above. Also, the parallax image generation information storage unit 361 performs subtraction when setting the far point shift amount for the far point, the near point shift amount for the near point (> far point shift amount), and the far point shift amount. Description will be made assuming that the quantity is stored. The subtraction amount is smaller than the difference between the near point shift amount and the far point shift amount. Hereinafter, description will be made assuming that each unit operates under the control of the control unit 37.

  First, when the processing device 3 acquires an image signal from the endoscope 2, the image processing unit 31 generates a right-eye image and a left-eye image as described above (step S301).

  In step S302 following step S301, when the distance information generation unit 33 receives an image signal, which is image data representing an endoscopic image captured by the imaging element 244, from the endoscope 2, the image included in the image signal is displayed. The distance to the subject is measured using this, and the measurement result is generated as distance information.

  In steps S303 to S306 following step S302, the shift amount setting unit 34 determines the left eye based on the distance information generated by the distance information generation unit 33 and the previously set shift amount (hereinafter also referred to as a set shift amount). The shift amount of the image for use and the image for the right eye is set. Specifically, the shift amount setting unit 34 first determines whether or not the subject is located at a near point based on the distance to the subject and the distance threshold value stored in the storage unit 36 (step). S303). At this time, if the shift amount setting unit 34 determines that the subject is not located at the near point, that is, the subject is located at the far point (step S303: No), the process proceeds to step S305. In contrast, when the shift amount setting unit 34 determines that the subject is located at the near point (step S303: Yes), the shift amount setting unit 34 proceeds to step S304.

  In step S304, the shift amount setting unit 34 sets the shift amount to the near point shift amount.

  In step S305, the shift amount setting unit 34 determines whether or not the currently set shift amount, that is, the set shift amount is equal to the far point shift amount. If the shift amount setting unit 34 determines that the set shift amount is equal to the far point shift amount (step S305: Yes), the shift amount is left as it was previously set, and the process proceeds to step S307. In contrast, when the shift amount setting unit 34 determines that the set shift amount is not equal to the far point shift amount, that is, the set shift amount is larger than the far point shift amount (step S305: No), step S306 is performed. Migrate to

  In step S306, the shift amount setting unit 34 subtracts the subtraction amount stored in the parallax image generation information storage unit 361 from the set shift amount, and sets the shift amount after the subtraction to the shift amount of the frame. If the shift amount after subtraction is smaller than the set far point shift amount, the shift amount setting unit 34 replaces the far point shift amount with the far point shift amount and sets this as the shift amount. When the setting of the shift amount by the shift amount setting unit 34 is completed, the control unit 37 proceeds to step S307.

  In step S307, the parallax image generation unit 32 performs zoom processing, enhancement processing, compression processing, or the like on the image signal including the left-eye image and the right-eye image, and then sets the shift amount setting unit 34. A display image signal including a parallax image obtained by relatively shifting the image for the right eye and the image for the left eye based on the shift amount thus generated is generated.

  In step S308 following step S307, the control unit 37 displays a display image that is a composite image in which image information including the parallax image generated by the parallax image generation unit 32 and character information related to the image information are superimposed on the display device 4. To display.

  In step S309 following step S308, the control unit 37 determines whether or not a new image signal is input from the endoscope 2. For example, when the control unit 37 determines that a new image signal is input from the endoscope 2 (step S309: Yes), the control unit 37 returns to step S301 and repeats the above-described processing. On the other hand, if it is determined that a new image signal has not been input (step S309: No), the above-described image processing ends.

  According to the second modification described above, when the shift amount setting unit 34 determines that the point is the far point, the shift amount is subtracted stepwise, so that the determination result shifts from the near point to the far point. The change in the shift amount is reduced, and a sudden change in the depth of the parallax image can be suppressed.

  In the above-described embodiment and Modifications 1 and 2, it has been described that the shift amount is switched by determining whether the subject is located at the near point or the far point from the distance information. The shift amount may be switched using the input of the switch 223 as a trigger. As a result, the parallax image with the shift amount switched can be displayed at the timing desired by the user.

  In the above-described embodiment and Modifications 1 and 2, the shift amount according to the distance is set. However, for example, one near-point shift amount is set and the number of near-point determinations is set. The shift amount may be changed in multiple steps, such as sequentially increasing the shift amount in accordance with the above. In addition, when it is determined that the subject is located at a far point, the shift amount may be decreased stepwise without returning to the preset shift amount. By changing the shift amount as described above, the change in the parallax image can be reduced, and the visual burden felt by the user can be suppressed.

(Modification 3 of embodiment)
In the third modification, the shift amount is set based on the generated distance information without determining the near point. FIG. 7 is a flowchart showing image processing performed by the processing device according to the third modification of the embodiment of the present invention. The endoscope system according to Modification 3 will be described as having the same configuration as the endoscope system 1 described above. Note that the parallax image generation information storage unit 361 according to the third modification stores a function or graph representing the relationship between the distance and the shift amount, instead of the distance threshold. The shift amount is associated with the distance so that the closer the point is, the larger the shift amount is. Hereinafter, description will be made assuming that each unit operates under the control of the control unit 37.

  First, when the processing device 3 acquires an image signal from the endoscope 2, the image processing unit 31 generates a right-eye image and a left-eye image as described above (step S401).

  In step S402 following step S401, when the distance information generation unit 33 receives an image signal, which is image data representing an endoscopic image captured by the imaging element 244, from the endoscope 2, the image included in the image signal is displayed. The distance to the subject is measured using this, and the measurement result is generated as distance information.

  In step S403 following step S402, the shift amount setting unit 34 uses the distance information generated by the distance information generation unit 33 and the function or graph stored in the parallax image generation information storage unit 361 for the left eye. Sets the shift amount of the image and the right-eye image. Specifically, the shift amount setting unit 34 sets the shift amount according to the distance based on the distance to the subject and a function representing the relationship between the distance and the shift amount, for example. When the shift amount setting unit 34 completes the setting of the shift amount, the control unit 37 proceeds to step S404.

  In step S404, the parallax image generation unit 32 performs zoom processing, enhancement processing, compression processing, or the like on the image signal including the image for the left eye and the image for the right eye, and then sets the shift amount setting unit 34. A display image signal including a parallax image obtained by relatively shifting the image for the right eye and the image for the left eye based on the shift amount thus generated is generated.

  In step S405 following step S404, the control unit 37 displays a display image that is a composite image in which image information including the parallax image generated by the parallax image generation unit 32 and character information related to the image information are superimposed on the display device 4. To display.

  In step S406 following step S405, the control unit 37 determines whether a new image signal is input from the endoscope 2 or not. For example, when the control unit 37 determines that a new image signal is input from the endoscope 2 (step S406: Yes), the control unit 37 returns to step S201 and repeats the above-described processing. On the other hand, if it is determined that a new image signal has not been input (step S406: No), the above-described image processing ends.

  According to the third modification described above, based on the distance information calculated from the left-eye image and the right-eye image, a shift amount that can express a sense of depth with embossed unevenness is set, and based on this shift amount Since the parallax image is generated, the image displayed on the display device 4 can have a sense of depth. In the embodiment and the first and second modifications described above, it is determined whether or not the position of the subject is a near point. In the third modification, the shift is performed without determining whether or not the subject is a near point. Since the amount is set, the load of the shift amount setting process can be reduced as compared with the embodiment and the first and second modifications.

In the above-described embodiment and Modifications 1 to 3, the processing device 3 has been described as generating a processing signal including an image to which each color component of RGB is added. However, the luminance (based on the YCbCr color space) It may be one that generates a processing signal having a YCbCr color space including a Y component and a color difference component, and is composed of three components: Hue, Saturation Chroma, and Value Lightness Brightness. A processing signal having components divided into colors and luminance may be generated using an HSV color space, an L ^* a ^* b ^* color space using a three-dimensional space, or the like.

  Further, in the above-described embodiment, it is assumed that white illumination light including each color component of RGB is emitted from the light source unit 3a and the light receiving unit is a simultaneous illumination / imaging method in which reflected light from the illumination light is received. However, the light source unit 3a may sequentially emit light of each color component individually, and the light receiving unit may use a surface sequential illumination / imaging method in which the light of each color component is received.

  In the above-described embodiment, the light source unit 3a has been described as being configured separately from the endoscope 2. However, for example, a light source device such as a semiconductor light source provided at the distal end of the endoscope 2 is used. The structure provided in the endoscope 2 may be sufficient. Furthermore, the function of the processing device 3 may be given to the endoscope 2.

  Moreover, although the light source part 3a was demonstrated as what was integrated with the processing apparatus 3 in embodiment mentioned above, the light source part 3a and the processing apparatus 3 are separate bodies, for example, it illuminates the exterior of the processing apparatus 3. The part 301 and the illumination control part 302 may be provided. Further, the light source 301 a may be provided at the tip of the tip portion 24.

  In the above-described embodiment, the endoscope system according to the present invention is described as the endoscope system 1 using the flexible endoscope 2 whose observation target is a living tissue or the like in the subject. However, a camera head is attached to the eyepiece of an optical endoscope such as a rigid endoscope, an industrial endoscope that observes the characteristics of materials, a capsule endoscope, a fiberscope, or an optical endoscope. Even an endoscope system using a connected one can be applied.

  As described above, the image processing apparatus according to the present invention is useful for acquiring a parallax image capable of performing detailed observation in near-point observation.

DESCRIPTION OF SYMBOLS 1 Endoscope system 2 Endoscope 3 Processing apparatus 3a Light source part 4 Display apparatus 21 Insertion part 22 Operation part 23 Universal code 24 Tip part 25 Bending part 26 Flexible tube part 31 Image processing part 32 Parallax image generation part 33 Distance information Generation unit 34 Shift amount setting unit 35 Input unit 36 Storage unit 37 Control unit 301 Illumination unit 302 Illumination control unit 361 Parallax image generation information storage unit

Claims (5)

An image processing device that generates a parallax image based on first and second images obtained by imaging the same subject from different directions,
A distance information generating unit that generates distance information that is a distance obtained from the first and second images and that indicates a distance from the imaging surface to the subject;
A shift amount setting unit that sets a shift amount in a parallax direction between the first image and the second image based on the distance information;
A parallax image generating unit that generates the parallax image based on the first and second images and the shift amount set by the shift amount setting unit;
An image processing apparatus comprising: The shift amount setting unit determines whether the distance is smaller than a preset distance based on the distance information, and based on the determination result, the first image and the second image The image processing apparatus according to claim 1, wherein a shift amount in the parallax direction is set. When the distance is smaller than a preset distance, the distance information generation unit sets the shift amount to a larger value than when the distance is equal to or greater than a preset distance. The image processing apparatus according to claim 2. The distance information generation unit sequentially generates the distance information for the input first and second images,
The shift amount setting unit determines whether or not the distance is smaller than the preset distance based on the distance information generated by the distance information generation unit, and if the distance is smaller than the preset distance When the determined number of consecutive frames is equal to or greater than a threshold, the shift amount is set based on the distance information, and the number of consecutive frames determined that the distance is smaller than a preset distance is the threshold. The image processing apparatus according to claim 2, wherein if it is smaller, the shift amount is set to a preset shift amount. A near point shift amount set when the distance is smaller than the preset distance and a far point shift amount set when the distance is more than the preset distance are preset. The near point shift amount is larger than the far point shift amount,
The shift amount setting unit determines that the distance is greater than or equal to a preset distance, and if the set shift amount is larger than the far point shift amount, the shift amount setting unit The image processing apparatus according to claim 2, wherein a value obtained by subtracting a subtraction amount smaller than a difference between the near point shift amount and the far point shift amount is set as the shift amount.