JP7179837B2 - Endoscope device, endoscope image display method, and operation method of endoscope device - Google Patents

Description

The present invention relates to an endoscope apparatus, an endoscope processor apparatus, and an endoscope image display method, and more particularly to a technique for assisting the operation of the tip of an endoscope.

2. Description of the Related Art In recent years, when an image (moving image) captured by an endoscope is displayed on a monitor and a precise endoscopic examination is performed, there are increasing cases where detailed observation by magnifying observation is required.

When performing magnifying observation, it is easy to lose sight of where the local area under magnifying observation is located in a wide area including its surroundings, often lacking a sense of the whole.

In particular, when magnifying observation is necessary to determine the stage of a disease with regional characteristics, it is difficult to determine to what extent the stage determination made by magnifying observation is effective.

Conventionally, an image processing device that creates a panoramic image from a plurality of images has been proposed in order to widen the field of view (Patent Documents 1 and 2). Patent document 1 describes that an image with an enlarged field of view can be obtained by connecting a group of continuous time-series images (X0 to Xn) obtained by an endoscope.

The invention described in Patent Document 1 is characterized by constructing a new image from which a defect (for example, a halation portion, etc.) of the original image is removed. An image from which abnormal data such as halation is removed is generated by image synthesis such as compensating a part (defective part) with a part of another image (image of the same part without defect).

On the other hand, Patent Document 2 discloses a technique of creating a panoramic image corresponding to the entire shooting range image that can be shot by a network camera, and displaying a display frame indicating which area the network camera is currently shooting on the panoramic image. Have been described.

JP-A-5-108819 JP 2016-82556 A

The invention described in Patent Document 1 is not intended to generate a panoramic image with a wide field of view, and while performing image synthesis between a plurality of images, a defective portion of one image is replaced with another image without defects. Compensation is performed using images of the same location, and images with a wide field of view are simultaneously generated as a result of image synthesis between a plurality of images at this time. Therefore, the frame information indicating the imaging area of the currently captured moving image is not displayed on the panorama image, and the panorama image does not support the operation of the tip of the endoscope.

On the other hand, the invention described in Patent Document 2 allows the user to confirm an area wider than the current shooting angle of view of the camera using a panoramic image, and allows the user to confirm the current shooting angle of view displayed in the panoramic image. With the displayed display frame, it is possible to know what kind of image will be captured by moving the camera in which direction, so that it is possible to assist the user's camera operation.

However, the control target of the invention described in Patent Document 2 is the network, not the endoscope (the camera at the tip of the scope).

Since an image is captured by a normal camera including a network, the camera is held substantially horizontally, and therefore the top and bottom directions of the panoramic image generated from the captured image and the currently captured image of the camera are substantially the same.

Therefore, when the camera is moved in the horizontal direction (panning) or moved in the vertical direction (tilting), the display frame indicating the shooting range also moves in the horizontal direction or the vertical direction on the panorama image. Matches the moving direction of the display frame.

On the other hand, the scope tip of an endoscope moves (bends) and rotates (rolls) in all directions, so the vertical direction of the current image captured by the endoscope (the vertical direction of the imaging device) direction) and the vertical direction of a panorama image generated from an image captured by an endoscope often do not match.

Therefore, according to the invention described in Patent Literature 1, the user can confirm a wider area than the image currently captured by the endoscope in the panoramic image, but the panoramic image includes an arbitrary position. There is a problem that it is not known how to move the tip of the endoscope when imaging an area.

Further, even if the invention described in Patent Document 1 is combined with the technology described in Patent Document 2 to display a display frame indicating an imaging range on a panorama image, a desired area is captured from the display frame. I don't know how to move the tip of the endoscope scope at this time.

SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances. An object of the present invention is to provide an endoscope device, an endoscope processor device, and an endoscope image display method that can assist operation of an endoscope.

In order to achieve the above object, an endoscope apparatus according to one aspect of the present invention includes a panoramic image generation unit that generates a panoramic image based on a moving image captured by an endoscope, and an image captured by the endoscope. a display control unit for displaying the generated moving image and the generated panoramic image on a display device, and the display control unit captures the current moving image displayed on the display device on the panoramic image displayed on the display device. A moving direction of the tip of the endoscope for displaying first information indicating a position and capturing an image of an imaging region including an arbitrary position on a panoramic image with the endoscope. Display the second information for assisting the operation.

According to one aspect of the present invention, the panorama image generated based on the moving image captured by the endoscope is displayed on the display, so that the user can view a wider area than the currently captured moving image. can be confirmed by the panoramic image. Further, the user can confirm the current shooting position of the moving image in the panorama image from the first information displayed on the panorama image, and the imaging area including any position on the panorama image from the second information. When capturing an image with an endoscope, it is possible to know in which direction the distal end of the endoscope should be moved, thereby assisting the operation of the endoscope.

According to another aspect of the present invention, the display control unit detects, by template matching, which area of the panoramic image each frame of the moving image corresponds to, and sets frame information indicating the area detected by template matching to the first is preferably displayed on the panorama image.

The endoscope apparatus according to still another aspect of the present invention includes a first instruction section for designating an arbitrary position on the panoramic image displayed on the display, and the display control section causes the first instruction section to When an arbitrary position is specified on the panorama image, it is preferable to display second information indicating the moving direction of the tip of the endoscope on the current moving image displayed on the display. The user can capture an image of an area including an arbitrary position by manipulating the direction of movement of the tip of the endoscope while viewing the second information indicating the direction of movement of the tip of the endoscope.

In the endoscope apparatus according to still another aspect of the present invention, when an arbitrary position on the panoramic image is designated by the first instruction section, the display control section displays the arbitrary position on the panoramic image and the current moving image. , and the relative angle between the panorama image and the current video, and based on the calculated relative position and relative angle, an imaging area including an arbitrary position is captured with an endoscope scope. It is preferable to determine the direction in which the distal end of the endoscope should move, and to display an indicator indicating the determined direction on the current moving image displayed on the display as the second information.

The relative position between an arbitrary position on the panoramic image and the center position of the current video, and the relative angle between the panoramic image and the current video, for example, detect which area of the panoramic image each frame of the video corresponds to It can be obtained from the movement position and rotation amount of the template (video frame) with respect to the panorama image at the time of template matching. Based on the calculated relative position and relative angle, the imaging state such as the posture of the tip of the endoscope with respect to the panoramic image can be determined. can determine the direction in which to move. An index (for example, an arrow) indicating the movement direction is displayed on the current moving image, and the user moves the tip of the endoscope in the direction indicated by the index to move the endoscope to an arbitrary position. An imaging region containing the image can be imaged.

In the endoscope apparatus according to still another aspect of the present invention, the display control unit calculates the relative angle between the panoramic image and the current moving image, and displays the calculated relative angle on the panoramic image displayed on the display. Based on this, it is preferable to display, as the second information, information indicating the vertical direction of the current moving image with respect to the panoramic image. If the vertical direction is known from the second information in addition to the current imaging position of the moving image for the panoramic image, the user can determine the direction in which the tip of the endoscope should be moved when capturing an arbitrary position within the panoramic image. can be easily grasped.

An endoscope apparatus according to still another aspect of the present invention includes an area designation generation mode setting section for setting an area designation generation mode for designating an attention area of a moving image, wherein the panoramic image generation section sets the area designation generation mode. Then, it is preferable to generate the panorama image based on a plurality of frames including the attention area designated in the area designation generation mode among the frames constituting the moving image. As a result, a panoramic image can be generated from the frames in the moving image in which the attention area is captured, and a suitable panoramic image can be generated when a local area (attention area of the moving image) is magnified and observed with an overall feeling. can.

An endoscope apparatus according to still another aspect of the present invention includes a time-designated generation mode setting unit for setting a time-designated generation mode for designating a time range of a moving image, wherein the panoramic image generation unit is set to the time-designated generation mode. Then, it is preferable to generate a panorama image based on a plurality of frames within a time range designated in the time designation generation mode among the frames constituting the moving image. As a result, a panoramic image can be generated from a moving image within a specified time range. can be imaged.

In the endoscope apparatus according to still another aspect of the present invention, the display control unit displays the generated panoramic image generated by the panoramic image generating unit and the panoramic image being generated by the panoramic image generating unit on the display. It is preferable to let By displaying the panorama image in the middle of generation by the panorama image generation unit, it is possible to check the area not included in the panorama image in the middle of generation.

In the endoscope apparatus according to still another aspect of the present invention, on the panorama image in the process of being generated displayed on the display, a second instruction unit that designates an uncaptured extension area for extending the panorama image in the process of being generated. When the extension area is specified by the second instruction unit, the display control unit displays third information indicating the moving direction of the distal end of the endoscope scope, the imaging area including the extension area. It is preferable to display the third information indicating the direction in which the tip of the endoscope should move for imaging with the endoscope on the current moving image displayed on the display.

By displaying the panorama image in the middle of generation by the panorama image generator, it is possible to check the area not included in the panorama image in the middle of generation, and the user can check the area not included in the panorama image (uncaptured extended area). ), it is possible to acquire the third information that guides the movement direction of the endoscope for imaging the unimaging extended region, thereby including the desired region in the panorama image. panorama image can be extended to

In the endoscope apparatus according to still another aspect of the present invention, a normal image capturing mode for acquiring a normal image corresponding to white light, or a normal image corresponding to white light and a special light image corresponding to special light alternately and the panorama image generation unit, when switched to the multi-frame imaging mode, overlaps the imaging areas from the plurality of special light images captured in the multi-frame imaging mode. A panoramic image is generated by extracting a plurality of sets of feature points between the special light images and performing adjustment between the plurality of special light images and a plurality of normal images obtained by imaging the same region based on the extracted plurality of sets of feature points. is preferred.

Since feature points can often be extracted more accurately from special light images than from normal images corresponding to white light, multiple pairs of feature points between special light images with overlapping imaging areas are extracted from multiple special light images. do. Based on these multiple sets of feature points, information for panorama synthesis (for example, conversion coefficients for projective transformation) is obtained, and the information for panorama synthesis is applied to a normal image captured in the same area as the special light image to create a panorama. Generate an image. In addition, the display of moving images is performed using time-series normal images, but by alternately acquiring normal images and special light images in the multi-frame shooting mode when creating a panoramic image, it is possible to use normal images even while creating a panoramic image. It allows you to display videos.

In the endoscope apparatus according to still another aspect of the present invention, when the panorama image generation unit is switched to the normal image capturing mode, the normal image having an overlapping imaging area is generated from a plurality of normal images captured in the normal image capturing mode. extracting multiple sets of feature points between them, adjusting between multiple normal images based on the extracted multiple sets of feature points to generate a panorama image, and extracting multiple sets of feature points between the normal images is not possible In this case, it is preferable that the shooting mode setting unit switches to the multi-frame shooting mode. A panoramic image is generated by adjusting multiple normal images based on multiple sets of feature points between the normal images, but if multiple sets of feature points between the normal images cannot be extracted, the multi-frame shooting mode is selected. By switching, it is possible to generate a panorama image using a special light image in the same manner as described above.

An endoscope apparatus according to still another aspect of the present invention includes a panoramic imaging mode setting section that sets a panoramic imaging mode for generating a panoramic image, and the panoramic image generating section sets the panoramic imaging mode when the panoramic imaging mode is set. Based on the point in time when the shooting mode was set, a moving image captured by the endoscope after the panorama shooting mode was set, a moving image captured by the endoscope before the panorama shooting mode was set, Alternatively, it is preferable to generate a panoramic image based on a moving image captured by the endoscope before and after setting the panoramic photography mode. That is, a panorama image is generated based on the moving image acquired with reference to the point in time when the panorama shooting mode was set.

In the endoscope apparatus according to still another aspect of the present invention, the panorama image generation unit performs distortion correction for correcting distortion of each frame of a moving image used for generating the panorama image, and based on the distortion-corrected frames, It is preferable to generate panoramic images. Since an endoscope image is generally a wide-angle image, it has large aberrations such as distortion. Therefore, before generating the panoramic image, the distortion of each frame of the moving image used for generating the panoramic image is corrected so that the panoramic image can be generated satisfactorily.

An endoscopic image display method according to still another aspect of the present invention includes steps of: a panoramic image generation unit generating a panoramic image based on a moving image captured by an endoscope; a step of displaying the moving image captured by the mirror scope and the generated panoramic image on the display; a step of displaying first information indicating the above; and a display control unit controlling an endoscope scope for imaging an imaging region including an arbitrary position with respect to an arbitrary position on the panoramic image with the endoscope scope and displaying second information to assist in manipulating the direction of movement of the tip.

In the endoscopic image display method according to still another aspect of the present invention, the step of displaying the first information includes detecting, by template matching, which region of the panoramic image each frame of the moving image corresponds to, and It is preferable to display the frame information indicating the area detected by matching as the first information on the panorama image.

An endoscopic image display method according to still another aspect of the present invention includes the step of designating an arbitrary position on the panorama image displayed on the display by a first instruction unit, and displaying second information. In the step, when an arbitrary position on the panorama image is designated by the first instruction unit, the second information indicating the moving direction of the tip of the endoscope is displayed on the current moving image displayed on the display. Display is preferred.

In the endoscopic image display method according to still another aspect of the present invention, the step of displaying the second information includes: Calculate the relative position between the arbitrary position and the center position of the current video, and the relative angle between the panoramic image and the current video, and view the imaging area including the arbitrary position based on the calculated relative position and relative angle It is preferable to determine the direction in which the tip of the endoscope should move for imaging with the endoscope, and to display an index indicating the determined direction on the current moving image displayed on the display as the second information. .

In the endoscopic image display method according to still another aspect of the present invention, the step of displaying the first information includes calculating the relative angle between the panoramic image and the current moving image, and calculating the relative angle between the panoramic image and the current moving image, and Secondly, it is preferable to display, as the second information, information indicating the vertical direction of the current moving image with respect to the panoramic image based on the calculated relative angle.

An endoscope processor device according to still another aspect of the present invention includes a panoramic image generation unit that generates a panoramic image based on a moving image captured by an endoscope; a display control unit for displaying the displayed panorama image on a display device, the display control unit displaying the current shooting position of the moving image displayed on the display device on the panorama image displayed on the display device; and support the operation of the movement direction of the tip of the endoscope for imaging the imaging area including any position with the endoscope scope for any position on the panoramic image. 2 information is displayed.

According to the present invention, a panoramic image generated based on a moving image captured by an endoscope enables confirmation of a wider area than the currently captured moving image. The first information allows confirmation of the current moving image shooting position in the panoramic image, and the second information enables endoscope imaging of an imaging area including an arbitrary position on the panoramic image with an endoscope. It is possible to know in which direction the distal end of the endoscope should be moved, thereby improving the operability of the endoscope.

FIG. 1 is an external perspective view of an endoscope apparatus 10 according to the present invention. FIG. 2 is a block diagram showing the electrical configuration of the endoscope device 10. As shown in FIG. FIG. 3 is a diagram used to explain mask processing of an endoscopic image. FIG. 4 is a diagram conceptually showing a panorama image creation method in the region designation generation mode. FIG. 5 is a diagram used to explain a specific algorithm for generating a panorama image. FIG. 6 is a diagram conceptually showing a method of creating a panorama image in the time-designated generation mode. FIG. 7 is a diagram showing a display example of the display 14 that displays the normal image 100, the panorama image 200, and the like. FIG. 8 is a diagram showing another display example of the display 14 that displays the normal image 100, the panoramic image 200, and the like. FIG. 9 is a diagram showing still another display example of the display 14 that displays the normal image 100, the panorama image 200, and the like. FIG. 10 is a flow chart showing the main flow of the endoscope image display method according to the present invention. FIG. 11 is a flow chart showing the main flow of the panorama image generation method when the process proceeds to step S20 shown in FIG. FIG. 12 is a flow chart showing a specific algorithm of a method of generating a panorama image when transitioning to step S40 shown in FIG. FIG. 13 is a flow chart showing a method of displaying a normal image and a panorama image when transitioning to step S30 shown in FIG. FIG. 14 is a flow chart showing an embodiment of a panorama image generation method in the case of transition to step S20 shown in FIG. FIG. 15 is a flow chart showing a method of switching between the normal image shooting mode and the multi-frame shooting mode to generate a panorama image. FIG. 16 is a diagram showing a display example of the display 14 that displays the normal image 100, the panorama image 200, etc., and particularly shows another display example of information for supporting the moving direction of the distal end of the endoscope. . FIG. 17 is a diagram showing a display example of the display 14 that displays the normal image 100, the panorama image 200, and the like. In particular, it is a diagram showing still another display example of information for supporting the moving direction of the distal end of the endoscope. be.

Preferred embodiments of an endoscope apparatus, an endoscope processor apparatus, and an endoscope image display method according to the present invention will be described below with reference to the accompanying drawings.

[Overall Configuration of Endoscope Device]
FIG. 1 is an external perspective view of an endoscope apparatus 10 according to the present invention.

As shown in FIG. 1, the endoscope apparatus 10 is roughly divided into an endoscope (flexible endoscope in this case) 11 for imaging an object to be observed in the subject, a light source device 12, and a processor device (endoscope). A mirror processor device 13 and a display device 14 such as a liquid crystal monitor are provided.

The light source device 12 supplies the endoscope 11 with various kinds of illumination light such as white light for capturing a normal image and light in a specific wavelength band for capturing a special light image.

The processor device 13 has a function of generating image data of a normal image and/or a special light image for display or recording based on an image signal obtained by the endoscope 11, a function of controlling the light source device 12, and other functions. In addition, it has a function of generating a panoramic image from time-series normal images according to the present invention, a function of displaying the normal image and the panoramic image on the display 14, and the like. Here, the panorama image in this example is an image including a wider range of observation targets than a normal image generated from two or more normal images.

The display device 14 displays a normal image or a special light image, as well as a normal image and a panorama image based on display image data input from the processor device 13 .

The endoscope 11 is connected to a flexible insertion portion 16 to be inserted into the subject and a proximal end portion of the insertion portion 16 , and is used for grasping the endoscope 11 and operating the insertion portion 16 . and a universal cord 18 that connects the handheld operation unit 17 to the light source device 12 and the processor device 13 .

An illuminating lens 42, an objective lens 44, an imaging device 45, and the like are built in an insertion portion distal end portion 16a (endoscope distal end portion), which is the distal end portion of the insertion portion 16 (see FIG. 2). A bendable bending portion 16b is connected to the rear end of the distal end portion 16a of the insertion portion. A flexible tube portion 16c is connected to the rear end of the bending portion 16b.

The hand operation unit 17 is provided with an angle knob 21, an operation button 22, a forceps inlet 23, and the like. The angle knob 21 is rotated when adjusting the bending direction and bending amount of the bending portion 16b. The operation button 22 is used for various operations such as air/water supply and suction. The forceps inlet 23 communicates with the forceps channel within the insertion section 16 . The hand operation unit 17 is also provided with an endoscope operation unit 46 (see FIG. 2) and the like for performing various settings.

The universal cord 18 incorporates air/water channels, signal cables, light guides, and the like. A connector portion 25 a connected to the light source device 12 and a connector portion 25 b connected to the processor device 13 are provided at the distal end portion of the universal cord 18 . As a result, illumination light is supplied from the light source device 12 to the endoscope 11 via the connector portion 25a, and an image signal obtained by the endoscope 11 is input to the processor device 13 via the connector portion 25b. .

The light source device 12 is provided with a light source operation unit 12a such as a power button, a lighting button for lighting the light source, and a brightness adjustment button. A processor operation section 13a is provided which includes an input section for receiving input from the device.

[Electrical Configuration of Endoscope Device]
FIG. 2 is a block diagram showing the electrical configuration of the endoscope device 10. As shown in FIG.

As shown in FIG. 2, the endoscope 11 is roughly divided into a light guide 40, an illumination lens 42, an objective lens 44, an imaging element 45, an endoscope operation section 46, and an endoscope control section 47. and a ROM (Read Only Memory) 48 .

A large-diameter optical fiber, a bundle fiber, or the like is used for the light guide 40 . The light guide 40 has its incident end inserted into the light source device 12 via the connector portion 25a, and its outgoing end passes through the insertion portion 16 and faces the illumination lens 42 provided in the insertion portion distal end portion 16a. ing. The illumination light supplied from the light source device 12 to the light guide 40 is irradiated onto the observation target through the illumination lens 42 . Then, the illumination light reflected and/or scattered by the observation target enters the objective lens 44 .

The objective lens 44 forms an image of reflected light or scattered light of incident illumination light (that is, an optical image of an observation target) on the imaging surface of the imaging device 45 .

The imaging device 45 is a CMOS (complementary metal oxide semiconductor) type or CCD (charge coupled device) type imaging device, and is relatively positioned and fixed to the objective lens 44 at a position deeper than the objective lens 44 . The imaging surface of the imaging element 45 has a two-dimensional array of a plurality of pixels composed of a plurality of photoelectric conversion elements (photodiodes) that photoelectrically convert an optical image. In addition, on the incident surface side of the plurality of pixels of the imaging element 45 of this example, red (R), green (G), and blue (B) color filters are arranged for each pixel, thereby , B pixels. Note that the filter arrangement of the RGB color filters is generally a Bayer arrangement, but is not limited to this.

The imaging device 45 converts an optical image formed by the objective lens 44 into an electrical image signal and outputs the electrical image signal to the processor device 13 .

If the imaging element 45 is of CMOS type, an A/D (Analog/Digital) converter is built therein, and a digital image signal is directly output from the imaging element 45 to the processor device 13 . If the imaging element 45 is of CCD type, the image signal output from the imaging element 45 is converted into a digital image signal by an A/D converter or the like (not shown) and then output to the processor device 13 . .

The endoscope operation unit 46 has a still image capturing button (not shown), a normal image capturing mode, a special light image capturing mode, a multi-frame capturing mode, a capturing mode setting unit for setting a panorama capturing mode, or a panorama capturing mode. It has a panorama shooting mode setting section, a region designation generation mode setting section, a time designation generation mode setting section, and the like. Note that the photographing mode setting section, the region-designated generation mode setting section, and the time-designated generation mode setting section may be provided in the processor operation section 13 a of the processor device 13 .

The endoscope control unit 47 sequentially executes various programs and data read from the ROM 48 or the like according to the operation of the endoscope operation unit 46 and mainly controls driving of the imaging device 45 . For example, in the normal image shooting mode, the endoscope control unit 47 controls the image sensor 45 so as to read the signals of the R, G and B pixels of the image sensor 45, and selects the special light image shooting mode or multi-frame image shooting mode. In the shooting mode, when violet light is emitted from the V-LED 32a or blue light is emitted from the B-LED 32b as illumination light for obtaining a special light image, these violet light and blue light are emitted. The imaging device 45 is controlled so as to read out only the signals of the B pixels of the imaging device 45 having spectral sensitivity in the wavelength band of light.

In addition, the endoscope control unit 47 communicates with the processor control unit 61 of the processor device 13 to obtain operation information from the endoscope operation unit 46 and the type of the endoscope 11 stored in the ROM 48 . to the processor device 13.

The light source device 12 has a light source control section 31 and a light source unit 32 . The light source controller 31 controls the light source unit 32 and communicates with the processor controller 61 of the processor device 13 to exchange various information.

The light source unit 32 has, for example, a plurality of semiconductor light sources. In this embodiment, the light source unit 32 includes V-LED (Violet Light Emitting Diode) 32a, B-LED (Blue Light Emitting Diode) 32b, G-LED (Green Light Emitting Diode) 32c, and R-LED (Red Light Emitting Diode) 32d has four color LEDs. The V-LED 32a is a violet light source that emits violet light with a central wavelength of 405 nm and a wavelength band of 380 to 420 nm. The B-LED 32b is a blue semiconductor light source that emits blue light with a center wavelength of 460 nm and a wavelength band of 420 to 500 nm. The G-LED 32c is a green semiconductor light source that emits green light with a wavelength band of 480-600 nm. The R-LED 32d is a red semiconductor light source that emits red light with a central wavelength of 620-630 nm and a wavelength band of 600-650 nm. Note that the central wavelengths of the V-LED 32a and the B-LED 32b have a width of about ±5 nm to ±10 nm.

Lighting and extinguishing of each of the LEDs 32a to 32d, the amount of light emitted when lit, etc. can be individually controlled by the light source controller 31 inputting independent control signals to each. In the normal image shooting mode, the light source control unit 31 turns on all of the V-LED 32a, B-LED 32b, G-LED 32c, and R-LED 32d. Therefore, in the normal image capturing mode, white light including violet light, blue light, green light, and red light is used as illumination light.

On the other hand, in the case of the special observation mode, the light source control unit 31 turns on any one light source of the V-LED 32a, B-LED 32b, G-LED 32c, and R-LED 32d, or a plurality of light sources that are appropriately combined, Alternatively, when a plurality of light sources are turned on, the light emission amount (light amount ratio) of each light source is controlled, thereby enabling imaging of a plurality of layers at different depths of the subject.

Light of each color emitted by each of the LEDs 32a to 32d is incident on a light guide 40 inserted through the endoscope scope 11 via an optical path coupling portion formed by mirrors, lenses, etc., and a diaphragm mechanism (not shown). be.

The illumination light of the light source device 12 may be white light (light in a white wavelength band or light in a plurality of wavelength bands), light in one or more specific wavelength bands (special light), or a combination thereof. Light of various wavelength bands is selected according to the purpose. The specific wavelength band of special light is a band narrower than the wavelength band of white.

A first example of a specific wavelength band is, for example, the blue band or the green band in the visible range. The wavelength band of this first example includes a wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less, and the light of the first example has a peak wavelength within the wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less. .

A second example of a specific wavelength band is, for example, the visible red band. The wavelength band of this second example includes a wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less, and the light of the second example has a peak wavelength within the wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less. .

A third example of the specific wavelength band includes a wavelength band in which oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients, and the light in the third example has a peak wavelength in the wavelength band in which oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients. have The wavelength band of this third example includes a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light of the third example is the above 400±10 nm, 440±10 nm, 470 It has a peak wavelength in the wavelength band of ±10 nm or from 600 nm to 750 nm.

A fourth example of the specific wavelength band is the wavelength band (390 nm to 470 nm) of excitation light used for observation of fluorescence emitted by a fluorescent substance in the living body (fluorescence observation) and for exciting this fluorescent substance.

A fifth example of the specific wavelength band is the wavelength band of infrared light. The wavelength band of the fifth example includes a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm, and the light of the fifth example has a peak wavelength in the wavelength band of 790 nm to 820 nm or 905 nm to 970 nm.

The processor device 13 has a processor operation unit 13a, a processor control unit 61, a ROM 62, a digital signal processor (DSP) 63, an image processing unit 65, a display control unit 66, a storage unit 67, and the like. .

The processor operation unit 13a includes a power button, an input unit for receiving input such as a coordinate position indicated on the display 14 by a mouse and a click (execution instruction).

The processor control unit 61 reads out necessary programs and data from the ROM 62 according to operation information from the processor operation unit 13 a and operation information from the endoscope operation unit 46 received via the endoscope control unit 47 . By performing the sequential processing, each part of the processor device 13 is controlled, and the light source device 12 is controlled. The processor control unit 61 may receive necessary instruction inputs from other external devices such as a keyboard connected via an interface (not shown).

Under the control of the processor control unit 61, the DSP 63 functions as one form of an image acquisition unit that acquires image data of each frame of a moving image output from the endoscope 11 (image sensor 45). Various signal processing such as defect correction processing, offset processing, white balance correction, gamma correction, demosaic processing, etc. are performed on image data for one frame of the moving image input from, and image data for one frame is generated. .

An image processing unit 65 having a function as a panorama image generation unit receives image data from the DSP 63 and performs image processing such as color conversion processing, color enhancement processing, and structure enhancement processing on the input image data as necessary. Processing is performed to generate image data representing an endoscopic image showing the observation target. Color conversion processing is processing that performs color conversion on image data by 3×3 matrix processing, gradation conversion processing, three-dimensional lookup table processing, and the like. The color enhancement process is a process of enhancing the color of the image data that has undergone the color conversion process, for example, in a direction that makes a difference in color between blood vessels and mucous membranes. Structure enhancement processing is processing for enhancing specific tissues and structures included in an observation target such as blood vessels and pit patterns, and is performed on image data after color enhancement processing.

The image data of each frame of the moving image processed by the image processing unit 65 is recorded in the storage unit 67 as the instructed still image or moving image when there is an instruction to shoot a still image or moving image. In this example, the image data of each frame of the moving image used for generating the panorama image is also recorded in the storage unit 67 . A moving image for generating a panoramic image is a moving image captured by the endoscope scope 11 after the panoramic shooting mode is set, and a moving image captured by the endoscope 11 before the panoramic shooting mode is set. A moving image captured by the endoscope 11, or a moving image captured by the endoscope 11 before and after setting the panorama shooting mode. This moving image includes frames necessary for generating a desired panoramic image. It can be a moving image of a certain shooting period, and can be updated sequentially. Also, the video saved for panoramic image generation may be deleted once the panoramic image generation is generated.

The image processing unit 65 has a function as a panoramic image generating unit that generates a panoramic image based on a moving image captured by the endoscope 11. Details of image processing when functioning as a panoramic image generating unit are as follows. will be described later.

The display control unit 66 generates display data for displaying a normal image, a special light image, and a panoramic image on the display device 14 from input image data, outputs the generated display data to the display device 14, and displays the data. The image for display is displayed on the device 14 .

The display control unit 66 performs mask processing on each frame 90 of the moving image based on the mask image 92 as shown in FIG. 3 to generate a normal image 100 for displaying the moving image. The mask image 92 is a rectangular image having the same size as the normal image 100, and has a non-mask portion and a mask portion (black portion) exposing a display area (a circular area in this example). Note that mask processing is not limited to generating a circular display image from a rectangular frame. For example, a barrel-shaped display image masked in an arc shape may be generated.

The display control unit 66 outputs the normal image 100 for display to the display device 14, displays the normal image 100 (moving image) on the display device 14, the panoramic photography mode is set by the panoramic photography mode setting unit, and the image processing unit When the panorama image generated by the panorama image generation unit 65 is inputted, the normal image 100 and the panorama image 200 (see FIG. 7) are displayed on the display 14 .

7, the display control unit 66 displays the first information (in this example, the same area as the circular normal image 100) indicating the shooting position of the current moving image (normal image 100) on the panorama image 200. Circular frame information 202) indicating is displayed.

Furthermore, when an arbitrary position 204 on the panorama image 200 is clicked with the mouse (when an arbitrary position is designated) as shown in FIG. An index (an arrow 102 in this example) indicating the direction in which the tip of the endoscope 11 should move for imaging by the endoscope 11 is displayed on the normal image 100 that is the current moving image.

Details of display control by the display control unit 66 for displaying the frame information 202 and the arrow 102 on the display 14 will be described later.

[Create panorama image]
Next, a method of generating a panoramic image by the image processing section 65 functioning as a panoramic image generating section will be described.

When the panoramic imaging mode is set by the endoscope operating section 46 or the processor operating section 13a, the image processing section 65 functions as a panoramic image generating section. It is assumed that the normal image shooting mode is set by the shooting mode setting unit.

When the panoramic photography mode is set under the normal image photography mode, the panoramic image generator generates a panoramic image from each frame (normal image) of the moving image captured in the normal image photography mode. A panorama image is generated as follows in accordance with the "region designation generation mode" or "time designation generation mode" set by the operation unit 46 or the processor operation unit 13a.

<Area specification generation mode>
The "region designation generation mode" is a mode in which a panoramic image is generated when a user (doctor) finds a region of interest in a normal image and designates an arbitrary position including the region of interest.

That is, after the region designation generation mode is set, the doctor clicks an arbitrary position within the region of interest on the normal image displayed on the display 14 using the mouse functioning as the first instruction unit. When specified, panorama synthesis is started. The panorama image generation unit extracts a plurality of frames including the specified attention area from among the frames constituting the moving image captured before the clicked time and held in the storage unit 67, and extracts them. A panorama image is generated by synthesizing multiple frames.

FIG. 4 is a diagram conceptually showing a panorama image creation method in the area specification generation mode.

In FIG. 4, assuming that the frame _I1 is the frame at the region designation time, the doctor clicks with the mouse any position within the region of interest indicated by the dashed frame. The panorama image generator extracts a plurality of frames I ₂ , I ₃ , I ₄ , and I ₅ including the attention area from the past frame group (moving image held in the storage unit ₆₇ ) including the frame I 1 .

The panorama image generator generates a panorama image 200 based on a plurality of frames I ₁ to I ₅ including frame I ₁ . In the example shown in FIG. 4, five frames I ₁ to I ₅ are used to generate the panorama image 200, but the number of frames used for panorama synthesis is not limited to five. It is preferable to extract all the frames that have been combined for panorama synthesis.

Note that the frames used for panorama synthesis are extracted from the group of frames at a time earlier than the frame _I1 at the region-designated time, but the frames to be used for panorama synthesis are extracted at a time later than the frame _I1 at the region-designated time (imaged after the region-designated time). You may extract the frames used for panorama synthesis from a group of video frames), or extract the frames used for panorama synthesis from a group of video frames captured at times both past and future from the specified region time. You may

In addition, the frames to be extracted from the frame group of the moving image are extracted with a time interval higher than the frame rate of the moving image (for example, 30 frames/second, 60 frames/second), or have a certain amount of movement between frames. It is preferable to extract frames. This is to reduce the number of frames to be panorama-synthesized and expand the area of the panorama image to be panorama-synthesized.

Next, a specific algorithm for generating a panorama image by the panorama image generation unit is as follows.

(1) As shown in FIG. 5, assume that two images Ii (i=1, 2) in which a region of interest is captured from different viewpoints are obtained between certain frames.

(2) Extract feature points from the image Ii. For this, feature point algorithms such as SIFT (Scale-invariant feature transform), SURF (Speed-Upped Robust Feature), ORB (Oriented FAST and Rotated BRIEF) are used. Assume that ni (i=1, 2) feature points Pij=(x_ij, y_ij) (j=1, 2, . . . , n) are detected for each image by this process. where (x,y) represent the coordinates in the image plane.

(3) When the feature points Pi for the image Ii are given, the feature points are matched based on the feature amount Di of each feature point. At this time, the feature amount Di is preferably information obtained based on the information of the feature point surrounding area. For example, SIFT is a 128-dimensional real number vector, and ORB is a 256-bit binary string. A pixel value in a W×W area around the feature point may be used. Whether or not certain feature points of the images _I1 and _I2 match is evaluated based on the similarity of feature amounts corresponding to the feature points. For example, if the feature value is SIFT, it is assumed that the Euclidean distance of each real number vector is less than a certain threshold.

(4) When four or more pairs of matching points are found between the images _I1 and _I2 by matching, image synthesis is performed.

If image _I1 is used as a reference image for synthesis, a matrix H for projecting (projective transformation) image _I2 onto the coordinate plane of image _I1 is obtained. When a feature point _P2 =(x_2, y_2) in the image _I2 matches a feature point _P1 =(x_1, y_1) in the image _I1 , the following relationship holds.

The matrix H is obtained by applying the formula [Formula 1] to the set of matched corresponding points and solving the simultaneous equations for the elements of the matrix H. Here, since the coefficient h ₃₃ in the matrix H is 1, 8 coefficients are obtained. A matrix H having eight coefficients (transformation coefficients) can be obtained by solving eight simultaneous equations from the coordinates of four sets of corresponding points (eight in total).

If four or more sets of corresponding points are found, it is preferable to obtain a more optimal matrix H by a RANSAC (Random Sampling Consensus) method or the like. This is to eliminate the effects of noise such as erroneous correspondence and positional deviation of each pair of corresponding points, and to calculate the matrix H with high precision.

Once the matrix H is obtained, all pixels of the image _I2 are transformed by the matrix H and superimposed on the image _I1 to obtain a composite image (panoramic image 200).

When synthesizing the next image _I3 , four sets of corresponding points in the overlapping region of the reference image _I1 and image _I3 are determined. If the proper 4 pairs of corresponding points are not found, find 4 pairs of corresponding points in the image _I2 after projective transformation or in the overlapping area between the panorama image in the process of being generated and the image _I3 . The panorama image is extended by synthesizing the image _I3 . A desired panoramic image can be generated by performing the above panorama synthesis processing on all the images used for generating the panoramic image.

<Time Specified Generation Mode>
In the "Time Specified Generation Mode", when the doctor finds an area of interest and gives an instruction to start panorama imaging at the time the doctor wants to start panorama imaging, panorama synthesis (image linking) will begin, and at the time the doctor wants to finish. This is a mode in which when an end instruction is given, image connection is ended.

That is, after the time-designated generation mode is set, when the doctor gives an instruction of the time at which he/she wants to start panorama imaging (panorama synthesis), the panorama image generation unit then starts panorama synthesis based on the acquired moving image, When the doctor gives an instruction of the desired ending time, the panorama synthesis is ended. If a panoramic image cannot be generated from moving images in real time, the moving images temporarily held in the storage unit 67 can be used.

FIG. 6 is a diagram conceptually showing a panorama image creation method in the time-designated generation mode.

In FIG. 6, frame _I1 indicates the frame at the start time of panoramic imaging instructed by the doctor, and frame _I5 indicates the frame at the end time of panoramic imaging instructed by the doctor.

The panorama image generator generates a plurality of frames _{I2 , I3} , _I4 , I ₅ is extracted and a panoramic image 200 is generated based on a plurality of frames I ₁ -I ₅ including frame I ₁ . Note that the panorama image 200 can be generated in the same manner as the panorama image is generated in the region designation generation mode.

Further, it is preferable that the panorama image generation unit performs distortion correction for correcting distortion of each frame of the moving image used for generating the panorama image, and generates the panorama image based on each frame subjected to the distortion correction.

2. Description of the Related Art Endoscopes in recent years have realized a wide viewing angle, making it possible to image a wider range. However, widening the viewing angle causes distortion in the captured image. The matrix H transforms parallelograms and trapezoids into arbitrary rectangles, so linearity must be preserved. However, since linearity is not maintained due to image distortion, it is conceivable that the matrix H cannot be obtained with high accuracy.

In order to solve this problem, the panorama image generation unit corrects the distortion of each frame of the moving image used to generate the panorama image as preprocessing before obtaining the matrix H.

Distortion correction transforms all pixel positions of image I by the following formula.

Here, k ₁ , k ₂ , k ₃ , p ₁ , and p ₂ in the formula (2) are referred to as distortion coefficients and are camera-specific values. Since the distortion coefficient changes depending on the enlargement ratio, a distortion coefficient corresponding to each enlargement ratio is used. Distortion correction may be performed before or after matching.

<Generation of panorama image in multi-frame shooting mode>
Next, a method of generating a panorama image when the multi-frame shooting mode is set by the shooting mode setting unit will be described.

When the multi-frame shooting mode is set, the light source device 12 emits white light including violet light, blue light, green light, and red light, and V-LED 32a, B-LED 32b, G-LED 32c, and R-LED 32d. Light of one or a plurality of specific wavelength bands (special light) whose lighting is controlled is alternately emitted, and the processor device 13 outputs an image under white light (normal image) and a light under special light from the endoscope 11 images (special light images) are acquired alternately.

The panorama image generation unit (image processing unit 65) generates a plurality of sets of feature points (4 pairs of corresponding points) are obtained, and the matrix H shown in the formula [Equation 1] is calculated. When generating a panoramic image, a normal image to be combined with the reference normal image is transformed by the matrix H and superimposed on the reference normal image to generate a composite image (panorama image).

The reason why the special light image is used to obtain the parameters for panorama synthesis (the matrix H shown in Equation 1) is that the special light image is an image in which the structure (blood vessels) of the subject is easy to see. This is because more feature points can be extracted than in a normal image, and the matrix H can be calculated with high accuracy.

On the other hand, since a doctor generally performs in-vivo screening observation using normal images, the display control unit 66 causes the display 14 to display normal images in the multi-frame imaging mode. In this case, since the frame rate of the normal image is halved, it is preferable to generate an interpolated image by interpolating between the preceding normal images so as not to reduce the frame rate of the normal image.

In the above example, the panorama image generator always uses the special light image to obtain parameters for panorama synthesis when the multi-frame shooting mode is set. Even in such cases, normal images are used to obtain parameters for panorama synthesis, and when normal images cannot detect multiple sets of highly reliable corresponding points, special light images are used to obtain parameters for panorama synthesis. may

Similarly, when the normal image shooting mode is set and it is not possible to detect a plurality of pairs of highly reliable corresponding points in the normal image acquired in the normal image shooting mode, the shooting mode setting unit selects multiple points from the normal image shooting mode. It is preferable to switch to the frame shooting mode and use the special light image to calculate parameters for panorama synthesis.

[How to display videos and panorama images]
As described above, the display control unit 66 outputs the normal image 100 for display to the display device 14 as shown in FIG. When the panorama image generated by the panorama image generation unit of the image processing unit 65 is input, the normal image 100 and the panorama image 200 are displayed on the display 14 .

Further, the display control unit 66 displays frame information 202 indicating the same area as the current moving image (normal image 100) on the panorama image 200 as shown in FIG.

The display control unit 66 detects, by template matching, which area of the panoramic image 200 the current normal image 100 of the moving image corresponds to. Template matching scans the panorama image 200 with the template image while changing the image size and rotation amount of the current normal image 100 (template image). Alternatively, it is a method of detecting which region of the panoramic image 200 the normal image 100 corresponds to by detecting the region with the highest degree of matching.

The display control unit 66 displays on the panorama image 200 frame information 202 indicating the same area as the area of the current normal image 100 detected by template matching. From the frame information 202 displayed on the panoramic image 200, the doctor can easily understand which area of the panoramic image 200 is currently being imaged.

Furthermore, when an arbitrary position 204 on the panorama image 200 is clicked with the mouse (when an arbitrary position is designated) as shown in FIG. An arrow 102 indicating the direction in which the tip of the endoscope 11 should move for imaging with the endoscope 11 is displayed on the current normal image 100 .

When an arbitrary position 204 on the panorama image 200 is clicked with the mouse, the display control unit 66 displays the arbitrary position 204 on the panorama image 200 and the current moving image (the area of the normal image 100 indicated by the frame information 202). A relative position with respect to the central position and a relative angle between the panorama image 200 and the current normal image 100 are calculated (acquired). The relative position between an arbitrary position 204 on the panoramic image 200 and the center position of the current moving image (normal image area), and the relative angle between the panoramic image 200 and the current normal image area are the current normal image of the moving image. However, when detecting which region of the panoramic image 200 corresponds to by template matching, the movement position of the normal image (template image) and the amount of rotation of the template image at the time of the best matching are defined as the relative position and the relative angle, respectively. can be obtained.

Based on the acquired relative position and relative angle, the display control unit 66 determines the direction in which the tip of the endoscope 11 should move in order to capture an image of the imaging region including the arbitrary position 204 with the endoscope 11. . That is, the direction of an arbitrary position 204 viewed from the center position (relative position) of the normal image area on the panoramic image 200 and the vertical direction of the normal image area on the panoramic image 200 (the normal image relative to the panoramic image 200). The vertical direction of the imaging device 45 corresponding to the angle) is calculated as the direction in which the tip of the endoscope 11 should move. Then, the display control unit 66 causes the arrow 102 indicating the calculated direction to be displayed on the normal image 100 .

In FIG. 8, since the current moving image is not rotated with respect to the reference image when the panoramic image 200 was generated (because the tip of the endoscope 11 has not been rotated since the reference image was captured), the panoramic image 200 is not rotated. The direction of an arbitrary position 204 viewed from the center position of the region of the normal image on the image 200 and the direction of the arrow 102 match.

On the other hand, as shown in FIG. 9, when the current moving image is rotated with respect to the reference image when the panorama image 200 was generated (for example, when rotated 135 degrees clockwise), the same arbitrary position The direction in which the tip of the endoscope 11 should move when the imaging region including 204 is imaged by the endoscope 11 is rotated 135 degrees counterclockwise with respect to the arrow 102 shown in FIG. An arrow 102 indicating the direction will be displayed.

By moving the tip of the endoscope 11 in the direction of the arrow 102 displayed on the normal image 100 when the doctor clicks on an arbitrary position, the doctor can view the imaging region including the designated arbitrary position with the endoscope. An image can be taken with the scope 11 .

The arrow 102 preferably disappears when the endoscope scope 11 captures an image of any specified position. Also, the arrow 102 may be displayed outside the display area of the normal image 100 instead of being superimposed on the normal image 100 .

Further, the display control unit 66 acquires a real-time panorama image in the middle of generation generated by the panorama image generation unit (image processing unit 65), thereby causing the display 14 to display the real-time panorama image in the middle of generation. can be done.

Further, in this case, similarly to the case of designating an arbitrary position 204 on the panorama image in the middle of generation displayed on the display device 14, an uncaptured extension region for extending the panorama image in the middle of generation is moved with the mouse (the 2), the display control unit 66 displays an arrow (third information) indicating the movement direction of the distal end of the endoscope 11 and an imaging area including the extension area. An arrow indicating the direction in which the tip of the endoscope should move for imaging with the scope 11 can be displayed on the normal image 100 .

By displaying the panorama image in the middle of generation by the panorama image generation unit, it is possible to check the area not included in the panorama image in the middle of generation, and to acquire the image of the uncaptured area in which the panorama image is to be expanded. The direction in which the distal end of the endoscope 11 should move can be guided.

[Endoscope image display method]
Next, an embodiment of an endoscopic image display method according to the present invention will be described with reference to flowcharts shown in FIGS. 10 to 15. FIG.

<Main flow of endoscopic image display method>
In FIG. 10, when the normal image capturing mode is set, the endoscope device 10 captures a moving image (normal image) with white light (step S10).

The captured moving image is temporarily stored in the storage unit 67 functioning as a buffer memory (step S12).

Subsequently, the processor control unit 61 determines whether or not the panorama imaging mode is set by the endoscope operation unit 46 or the processor operation unit 13a (step S14). If the panorama shooting mode is not set, the display control unit 66 displays the normal image on the display 14 (monitor) (step S18).

On the other hand, if the panorama shooting mode is set, the process proceeds to step S16, where it is determined whether or not a panorama image has been generated.

If a panoramic image has not been generated (“No”), the panoramic image generation unit (image processing unit 65) generates a panoramic image using the moving image held in step S12, the moving image captured thereafter, and the like. Generate (step S20).

If a panoramic image has been generated (“Yes”), the display control unit 66 displays the current normal image and panoramic image on the monitor (step S30).

<How to generate a panorama image (1)>
FIG. 11 is a flow chart showing the main flow of the panorama image generation method when the process proceeds to step S20 shown in FIG.

In FIG. 11, the processor control unit 61 determines which of the region-designated generation mode and the time-designated generation mode is set by the region-designated generation mode setting unit or the time-designated generation mode setting unit (step S21).

If the region designation generation mode is set, the process transitions to step S22, where it is determined whether a region of interest in the normal image has been designated (for example, an arbitrary position including the region of interest has been designated with the mouse). or not). When it is determined that the attention area has been specified, the panorama image generation unit extracts an image group including the attention area from the moving image held in the memory (storage unit 67) (step S23), and creates a panorama image from the extracted image group. is generated (step S40).

On the other hand, if the time-designated generation mode is set, the process proceeds to step S24, where it is determined whether or not the time range of the moving image (the start time and end time of the moving image used to generate the panorama image) has been designated. discriminate. When the time range of the moving image is designated, the image group within the time range from the designated start time to the end time is extracted from the moving image (step S25), and a panorama image is generated from the extracted image group (step S40). .

In this case, when the start time is specified, the generation of the panorama image is started based on each frame of the video captured after that time, and when the end time is specified, new frames to be used for panorama synthesis are generated. Alternatively, after a group of images within the time range from the start time to the end time are held in the memory, the panorama image is generated based on each frame of the moving image held in the memory. An image may be generated.

<How to generate a panorama image (2)>
FIG. 12 is a flow chart showing a specific algorithm of a method of generating a panorama image when transitioning to step S40 shown in FIG.

In FIG. 12, the panorama image generation unit first uses parameter i is set to 1 (step S41).

Subsequently, the panorama image generator extracts feature points between the two images of the normal image _Ii and the normal image _Ii+1 (step S42), and sets a plurality of sets of feature points corresponding to each other in the overlap region between the two images. (in this example, four sets of corresponding points) are determined, the matrix H of the formula [Equation 1] is obtained from the coordinates of the four sets of corresponding points, and all pixels of the normal image _Ii+1 to be synthesized are transformed (projected) by the matrix H. conversion) to generate a synthesized image (panoramic image P _i+1 ) (step S43).

Next, it is determined whether or not the parameter i is N-1 (a value corresponding to the number of times of image synthesis) (step S44). If panorama synthesis of all normal images has not been completed (if i≠N−1), parameter i is incremented by 1 (step S45), the process proceeds to step S42, and panorama synthesis of all normal images is performed. is finished (when i=N−1), the generation of the panoramic image is finished.

<How to display normal images and panorama images>
FIG. 13 is a flow chart showing a method of displaying a normal image and a panorama image when transitioning to step S30 shown in FIG.

In FIG. 13, the display control unit 66 displays the normal image and the panorama image input from the image processing unit 65 on the monitor (display device 14) (step S31).

Subsequently, the current normal image and the panoramic image are subjected to template matching (step S31), and from the result of template matching, the current normal image area on the panoramic image and the relative angle of the normal image to the panoramic image are detected (obtained) ( step S33).

Subsequently, the display control unit 66 causes the frame information 202 indicating the area of the current normal image on the panorama image to be displayed on the panorama image 200 based on the result of template matching (step S34, see FIG. 7).

Next, the display control unit 66 determines whether or not an arbitrary position on the panorama image has been specified (step S35). For example, when an arbitrary position 204 on the panorama image 200 is clicked with the mouse as shown in FIG. 8, it is determined that the arbitrary position 204 has been specified.

When the display control unit 66 determines in step S35 that an arbitrary position on the panorama image has been designated (in the case of "Yes"), the scope tip for imaging the imaging region including the arbitrary position with the endoscope scope 11 is displayed. is determined (step S36). The direction in which the tip of the scope should move is the coordinates of an arbitrary position 204 on the panoramic image 200, the coordinates of the center of the current normal image area on the panoramic image 200, and the coordinates of the panoramic image 200 obtained in step S33. It can be obtained from the relative angle of the normal image 100 .

The display control unit 66 displays the index (arrow 102) indicating the direction determined in step S36 on the normal image 100 (step S37, arrow 102 in FIGS. 8 and 9).

<How to generate a panorama image in multi-frame shooting mode>
FIG. 14 is a flow chart showing an embodiment of a panorama image generation method in the case of transition to step S20 shown in FIG. 10, and particularly shows a case where the multi-frame shooting mode is set.

In FIG. 14, when the shooting mode setting unit switches to the multi-frame shooting mode or sets the multi-frame shooting mode (step S50), white light and special light are alternately emitted from the light source device 12, and white light is emitted. A lower image (normal image) and an image under special light (special light image) are alternately captured (step S51).

Subsequently, a panorama photographing mode for generating a panorama image is set, and it is determined whether or not the normal image and the special light image photographed in the multi-frame photographing mode have been stored in the memory for a certain period of time (step S52).

When moving images (normal images and special light images) for a certain period of time are stored in the memory (if “Yes”), the panorama image generation unit generates the normal image group and the special light image group necessary for generating the panorama image. Extract (step S53).

The panorama image generator generates a plurality of sets of feature points (four sets of corresponding points) between special light images having overlapping imaging areas based on a plurality of special light images corresponding to a plurality of normal images used for panorama synthesis. Then, the parameters for generating the panorama image (matrix H shown in Equation 1) are calculated. When generating a panoramic image, the normal image to be synthesized with the reference normal image is projectively transformed by the matrix H, and superimposed on the reference normal image to generate a synthesized image (panorama image) (step S55). ).

<Switching between normal image shooting mode and multi-frame shooting mode>
FIG. 15 is a flowchart showing a method of switching between normal image shooting mode and multi-frame shooting mode to generate a panorama image.

In FIG. 15, the panorama image generation unit extracts a group of normal images used to generate a panorama image from the moving image held in the memory (step S60). In addition, in the case of the moving image captured in the multi-frame shooting mode, the normal image group is extracted from the moving image of the normal image out of the moving image of the normal image and the moving image of the special light image. Extraction of the normal image group can be performed based on the area-designated generation mode or the time-designated generation mode.

The panorama image generation unit extracts feature points between two normal images having overlapping regions in the normal image group (step S61). Subsequently, it is determined whether or not a plurality of pairs of feature points (four pairs of corresponding points) have been extracted (detected) between normal images having overlapping imaging regions (step S62).

If four appropriate pairs of corresponding points are detected ("Yes"), a panorama image is generated using the normal image group as shown in FIGS. 11 and 12 (step S63).

On the other hand, if four sets of appropriate corresponding points are not detected ("No"), the mode is switched from the normal image capturing mode to the multi-frame capturing mode (step S64), and then extracted from the moving image captured in the multi-frame capturing mode. A panorama image is generated as shown in FIG. 14 using the normal image group and the special light image group (step S65).

[Other Embodiments of Method for Supporting Operation in Moving Direction of Tip of Endoscope]
16 and 17 show other display examples of the second information for assisting the manipulation of the moving direction of the tip of the endoscope for imaging an imaging area including an arbitrary position with the endoscope. It is a diagram.

In the embodiment shown in FIGS. 8 and 9, when an arbitrary position 204 on the panorama image 200 is clicked by the mouse, the display control unit 66 captures an imaging area including the arbitrary position 204 with the endoscope 11. An arrow 102 indicating the direction in which the tip of the endoscope 11 should move is displayed on the current normal image 100 .

On the other hand, in other embodiments shown in FIGS. 16 and 17, when the user captures an arbitrary position 204 on the panorama image 200, the user does not specify the arbitrary position 204 with a mouse or the like. Information is displayed that enables the user to recognize (supports recognition) the direction in which the tip of the endoscope 11 should move when the imaging region including .

In FIG. 16, the same normal image 100 and panoramic image 200 as the normal image 100 and panoramic image 200 shown in FIG. 8 are displayed.

In FIG. 8, an arrow 102 indicating the direction in which the tip of the endoscope 11 should move for imaging an imaging region including an arbitrary position 204 is displayed on the normal image 100 that is the current moving image. 11, information indicating the relative angle of the region of the normal image 100 on the panoramic image 200 with respect to the panoramic image 200, which indicates the vertical direction of the current moving image with respect to the panoramic image 200 (this In the example, an arrow 210) is displayed.

The vertical direction of the normal image area on the panoramic image 200 (vertical direction of the imaging device 45 corresponding to the relative angle of the normal image with respect to the panoramic image 200) is detected (obtained) by template matching between the current normal image and the panoramic image. can do.

In FIG. 16 corresponding to FIG. 8, the current moving image is not rotated with respect to the reference image when the panorama image 200 is generated (the tip of the endoscope 11 has been rotated since the reference image was captured). Therefore, the normal image 100 displayed on the display 14 and the area corresponding to the normal image 100 in the panorama image 200 (the area within the frame information 202) are similar in shape and face in the same direction. .

Then, the display control unit 66 displays an arrow 210 indicating the vertical direction of the area corresponding to the normal image 100 in the panoramic image 200 on the panoramic image 200 (near the frame information 202). The direction of this arrow 210 can be determined by the relative angle between the panoramic image 200 and the current normal image 100 .

In the display state shown in FIG. 16, when the tip of the endoscope 11 is moved to capture an image of an area including an arbitrary position 204 with the endoscope 11, the direction of movement of the tip of the endoscope 11 is , an arbitrary position 204 relative to the direction of arrow 210 indicates the top right direction of the scope tip. This matches the operation direction guided by the arrow 102 displayed on the normal image 100 shown in FIG.

On the other hand, in FIG. 17 corresponding to FIG. 9, since the current normal image 100 is rotated 135 degrees clockwise with respect to the panoramic image 200, the vertical direction of the area corresponding to the normal image 100 in the panoramic image 200 is indicates the lower right direction in FIG.

Therefore, in the case of the display state shown in FIG. The direction is seen by arbitrary position 204 relative to the direction of arrow 210 to the left of the scope tip. This matches the operation direction guided by the arrow 102 displayed on the normal image 100 shown in FIG.

[others]
In the present embodiment, frame information surrounding the area of the current moving image (area on the panoramic image) is displayed as the first information indicating the shooting position of the current moving image on the panoramic image. However, any information may be used as long as it allows the user to recognize the area of the current moving image on the panorama image. Similarly, the arrow is displayed as the second information for assisting the operation of moving the distal end of the endoscope, but any indicator may be used as long as the direction can be visually recognized.

Further, in the present embodiment, when generating a panoramic image, an image to be synthesized is projectively transformed with respect to a reference image, and image synthesis is performed. may be performed to synthesize images.

In addition to the function of generating image data of a normal image and/or special light image for display or recording based on the image signal obtained by the endoscope 11, and the function of controlling the light source device 12, the present The hardware structure of the processor device 13 having a function of generating a panoramic image from time-series normal images, a function of displaying the normal image and the panoramic image on the display 14, etc., according to the invention, may be various as follows. processor. For various processors, the circuit configuration can be changed after manufacturing such as CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), which is a general-purpose processor that executes software (program) and functions as various control units. Programmable Logic Device (PLD), which is a processor, ASIC (Application Specific Integrated Circuit), etc. be

One processing unit may be composed of one of these various processors, or composed of two or more processors of the same type or different types (for example, a plurality of FPGAs, or a combination of a CPU and an FPGA). may Also, a plurality of control units may be configured by one processor. As an example of configuring a plurality of control units with a single processor, first, as represented by a computer such as a client or a server, a single processor is configured by combining one or more CPUs and software. There is a form in which a processor functions as multiple controllers. Secondly, as typified by System On Chip (SoC), etc., there is a form of using a processor that realizes the functions of the entire system including multiple control units with a single IC (Integrated Circuit) chip. be. In this way, various control units are configured using one or more of the above various processors as a hardware structure.

Furthermore, the present invention is not limited to the embodiments described above, and it goes without saying that various modifications are possible without departing from the spirit of the present invention.

10 endoscope device 11 endoscope scope 12 light source device 12a light source operation section 13 processor device 13a processor operation section 14 display device 16 insertion section 16a insertion section distal end section 16b bending section 16c flexible tube section 17 hand operation section 18 universal cord 21 angle knob 22 operation button 23 forceps inlet 25a connector portion 25b connector portion 31 light source control portion 32 light source unit 32a V-LED
32b B-LED
32c G-LED
32d R-LED
40 light guide 42 illumination lens 44 objective lens 45 imaging device 46 endoscope operating unit 47 endoscope control unit 48, 62 ROM
61 processor control unit 65 image processing unit 66 display control unit 67 storage unit 100 normal images 102, 210 arrow 200 panoramic image 202 frame information H matrix S10 to S65 steps

Claims (14)

a panorama image generation unit that generates a panorama image based on a moving image captured by an endoscope;
a display control unit for displaying a moving image captured by the endoscope and the generated panoramic image on a display;
a first indicator that specifies an arbitrary position on the panoramic image displayed on the display;
The display control unit
displaying on the panorama image displayed on the display device first information indicating the current shooting position of the moving image displayed on the display device;
and a second method for supporting an operation in a moving direction of a distal end of the endoscope for capturing an image of an imaging region including the arbitrary position on the panoramic image with the endoscope. display information,
When the arbitrary position on the panorama image is designated by the first instruction unit, the display control unit is configured to provide a relative position between the arbitrary position on the panorama image and the center position of the current moving image , and calculating a relative angle between the panoramic image and the current moving image , and imaging an imaging region including the arbitrary position with the endoscope based on the calculated relative position and the relative angle. An endoscope apparatus that determines a direction in which a tip of an endoscope should move, and displays an index indicating the determined direction as the second information on a current moving image displayed on the display. a panorama image generation unit that generates a panorama image based on a moving image captured by an endoscope;
a display control unit that displays a moving image captured by the endoscope and the generated panoramic image on a display;
The display control unit
displaying on the panorama image displayed on the display device first information indicating the current shooting position of the moving image displayed on the display device;
and a second method for supporting an operation in a moving direction of a distal end of the endoscope for capturing an image of an imaging region including the arbitrary position on the panoramic image with the endoscope. display information,
The display control unit calculates a relative angle between the panoramic image and the current moving image , and transmits information indicating the vertical direction of the current moving image with respect to the panoramic image based on the calculated relative angle . An endoscope apparatus for displaying second information on the panoramic image displayed on the display . The display control unit detects which region of the panoramic image each frame of the moving image corresponds to by template matching, and uses frame information indicating the region detected by the template matching as the first information. The endoscope apparatus according to claim 1 or 2, which is displayed on a panoramic image. an area designation generation mode setting unit for setting an area designation generation mode for designating an attention area of the moving image;
When the region designation generation mode is set, the panorama image generation unit generates a panorama image based on a plurality of frames including the attention region designated in the region designation generation mode among the frames constituting the moving image. The endoscope apparatus according to any one of claims 1 to 3 . A time-designated generation mode setting unit that sets a time-designated generation mode that indicates the time range of the moving image,
When the time-specified generation mode is set, the panorama image generation unit generates a panorama image based on a plurality of frames within the time range designated in the time-designated generation mode among the frames constituting the moving image. The endoscope apparatus according to any one of claims 1 to 3 . 6. The display controller according to any one of claims 1 to 5 , wherein the display control unit causes the display to display the generated panoramic image generated by the panoramic image generating unit and the panoramic image being generated by the panoramic image generating unit. The endoscope device according to . a second instruction unit for designating an uncaptured expansion area for expanding the panorama image in the middle of generation on the panorama image in the middle of generation displayed on the display;
When the extension area is specified by the second instruction unit, the display control unit specifies an imaging area including the extension area, which is third information indicating a movement direction of the distal end of the endoscope. 7. The method according to claim 6 , wherein third information indicating a direction in which the tip of the endoscope should move for imaging by the endoscope is displayed on the current moving image displayed on the display. endoscopic device. Equipped with a shooting mode setting unit that switches between a normal image shooting mode that acquires a normal image corresponding to white light and a multi-frame shooting mode that alternately acquires a normal image corresponding to white light and a special light image corresponding to special light. ,
When switched to the multi-frame imaging mode, the panorama image generation unit extracts a plurality of sets of feature points between special light images having overlapping imaging areas from a plurality of special light images captured in the multi-frame imaging mode. 8. The panorama image is generated by performing adjustment between the plurality of special light images and a plurality of normal images obtained by imaging the same region based on the plurality of sets of extracted feature points. The endoscope device according to . When switched to the normal image shooting mode, the panorama image generation unit extracts a plurality of sets of feature points between the normal images having overlapping imaging areas from the plurality of normal images captured in the normal image shooting mode, and The photographing mode setting unit adjusts the plurality of normal images based on the plurality of sets of feature points to generate the panorama image, and when the plurality of sets of feature points between the normal images cannot be extracted, The endoscope apparatus according to claim 8 , wherein the mode is switched to the multi-frame imaging mode. A panorama shooting mode setting unit for setting a panorama shooting mode for generating the panorama image,
When the panorama imaging mode is set, the panorama image generation unit captures an image with the endoscope after the panorama imaging mode is set, with reference to the time when the panorama imaging mode is set. The panoramic image based on a moving image, a moving image captured by the endoscope before the panoramic imaging mode is set, or a moving image captured by the endoscope before and after the panoramic imaging mode is set. 10. The endoscope apparatus according to any one of claims 1 to 9 , which generates 11. The panorama image generation unit according to any one of claims 1 to 10 , wherein the panorama image generation unit performs distortion correction for correcting distortion of each frame of the moving image used to generate the panorama image, and generates the panorama image based on the distortion-corrected frames. The endoscopic device according to any one of claims 1 to 3. A method of operating an endoscope apparatus comprising a panoramic image generator, a display controller, and a first instruction unit for designating an arbitrary position on a panoramic image displayed on a display, comprising:
a step in which the panoramic image generation unit generates a panoramic image based on a moving image captured by an endoscope;
a step in which the display control unit displays a moving image captured by the endoscope and the generated panoramic image on the display ;
a step in which the display control unit displays, on the panorama image displayed on the display, first information indicating a current shooting position of the moving image displayed on the display;
a step in which the display control unit determines whether or not the arbitrary position on the panoramic image displayed on the display is specified by the first instruction unit ;
The display control unit controls the moving direction of the tip of the endoscope for imaging an imaging region including the arbitrary position on the panoramic image with the endoscope. and displaying second information to assist the operation;
In the step of displaying the second information, when it is determined that the arbitrary position on the panoramic image is designated by the first instruction unit, the arbitrary position on the panoramic image and the current moving image are displayed. calculating a relative position with respect to a center position and a relative angle between the panoramic image and the current moving image, and determining an imaging region including the arbitrary position based on the calculated relative position and the relative angle; A direction in which the tip of the endoscope should move for imaging with the scope is determined, and an index indicating the determined direction is displayed as the second information on the current moving image displayed on the display. A method of operating an endoscopic device . a panoramic image generation unit generating a panoramic image based on a moving image captured by an endoscope;
a step in which a display control unit displays a moving image captured by the endoscope and the generated panoramic image on a display;
a step in which the display control unit displays, on the panorama image displayed on the display, first information indicating a current shooting position of the moving image displayed on the display;
The display control unit controls a movement direction of a distal end of the endoscope for imaging an imaging region including the arbitrary position on the panoramic image with the endoscope. and causing the supporting second information to be displayed;
The step of displaying the second information includes calculating a relative angle between the panoramic image and the current moving image , and indicating the vertical direction of the current moving image with respect to the panoramic image based on the calculated relative angle. An endoscopic image display method for displaying information as the second information on a panoramic image displayed on the display . The step of displaying the first information includes detecting, by template matching, which area of the panorama image each frame of the moving image corresponds to, and displaying frame information indicating the area detected by the template matching as the first information. 14. The endoscopic image display method according to claim 13 , wherein the information is displayed on the panoramic image.

Images