JP6941233B2 - Image processing equipment, endoscopic system, and image processing method - Google Patents

Description

The present invention relates to an image processing apparatus, an endoscopic system, and an image processing method, and more particularly to an image processing apparatus, an endoscopic system, and an image processing method for acquiring an image with a plurality of observation lights.

In the medical field, images of subjects taken with medical equipment are used for diagnosis, treatment, etc., but "what kind of structure of the subject is clearly (or unclearly) reflected in the taken images?""Depends on the observation light used for photographing. For example, in an image taken under special light such as narrow band light having a strong short wavelength component, blood vessels on the surface layer are depicted with good contrast, which is suitable for detecting lesions, while under special light having a strong long wavelength component. In the image taken with, deep blood vessels are depicted with good contrast. In addition, observations by doctors are often performed with normal light (white light) instead of special light. As described above, in photographing, it is preferable to irradiate the observation light according to the purpose of use and the object of the image.

As a technique for switching the observation light in this way, for example, Patent Document 1 is known. Patent Document 1 describes an oxygen saturation image from an image obtained by alternately irradiating a first white light and a second white light with a normal observation mode in which a second white light is irradiated to display a normal light image. An endoscopic system with a special observation mode to generate and display is described.

JP-A-2017-153978

When switching between narrow-band light with a strong short-wavelength component and normal light, images using special light (including images generated by irradiating multiple narrow-band lights and images generated by irradiating short-wavelength narrow-band light) May not be suitable for observation by a user who is accustomed to observing an image with normal light, and a method of constantly acquiring an image with special light during diagnosis may hinder the user's observation.

The present invention has been made in view of such circumstances, and is an image processing device, an endoscopic system, and an image processing capable of acquiring an image by a plurality of observation lights as needed while suppressing the influence on the user's observation. The purpose is to provide a method.

In order to achieve the above-mentioned object, the image processing apparatus according to the first aspect of the present invention was photographed by a first image captured by the first observation light and a second observation light different from the first observation light. An image acquisition unit for acquiring the second image, an acquisition instruction reception unit for receiving a still image acquisition instruction, an image acquisition control unit for controlling the acquisition of the first image and the second image by the image acquisition unit, and at least a second image acquisition unit. A display control unit for displaying one image on a display device and a classification unit for classifying subjects reflected in at least a second image are provided, and the image acquisition control unit is the first until the acquisition instruction reception unit receives an acquisition instruction. The image acquisition unit performs a moving image acquisition process for sequentially acquiring one image as a moving image, and acquires a first image and a second image as a still image in response to an acceptance of an acquisition instruction. After the still image acquisition process is completed, the image acquisition unit is made to perform the moving image acquisition process.

When acquiring an image with a plurality of observation lights, it is preferable to acquire an image with a plurality of observation lights as necessary while not interfering with the observation by the user. However, in Patent Document 1 described above, in the special observation mode, the first and second white lights are alternately irradiated, and the oxygen saturation image is displayed, so that the observation with normal light is hindered. In contrast to such a conventional technique, in the first aspect, the first image by the first observation light is sequentially acquired and displayed as a moving image until the still image acquisition instruction is received, and the still image acquisition instruction is received. Acquires the first and second images by the first and second observation lights as still images, and returns to the acquisition and display of moving images of the first image by the first observation light after the acquisition of the still images is completed. As a result, the user can continue observing the first image as needed (for example, when there is an instruction from the user or when a region of interest is detected, or when a still image needs to be acquired). A still image can be acquired by the first and second observation lights, and the subject (area of interest, etc.) can be classified together with the observation.

In the first aspect, one frame of a moving image can be acquired as a still image. In addition, when in-vivo imaging is performed, the type of polyp (neoplastic or non-neoplastic), stage diagnosis of cancer, determination of the position in the lumen (imaging position), and the like can be performed as "classification". Further, in the first aspect, the first image is continuously displayed. The second image can be displayed as needed (for example, in response to a user's instruction input or in response to the result of processing the second image).

In the first aspect and each of the following aspects, one of the first observation light and the second observation light may be white light and the other may be narrow band light, or both may be different narrow band light. Further, the light emitted from the light source may be used as it is for the first observation light and the second observation light, or a filter for transmitting a specific wavelength band to the light emitted from the light source (for example, white light) may be applied. It may be the generated light. When narrow-band light is used as the first observation light and / or the second observation light, the narrow-band light used may be narrow-band light emitted from a light source for narrow-band light, or may have a specific wavelength with respect to white light. Narrow-band light generated by applying a filter that transmits a band may be used. In this case, different narrow-band lights may be emitted at different timings by sequentially switching the filters.

In the first aspect, the first image taken by the first observation light and the second image taken by the second observation light are acquired. Since the second observation light is not used for taking the first image and the first observation light is not used for taking the second image, the wavelengths cannot be separated and the image quality of the first image and the second image cannot be separated. Does not decrease.

In the first aspect and each of the following aspects, "the first observation light and the second observation light are different" means that at least one of the wavelength band or the spectral spectrum of the first observation light and the second observation light is different. It means that they are not the same. Further, the first image and the second image may be medical images obtained by photographing a subject such as a living body.

In the first aspect and each of the following aspects, the medical image is also referred to as a medical image. Further, as the light source used for taking a medical image, a light source having a white band, a light having a plurality of wavelengths (narrow band light) as the white band, an infrared light, and a light source generating excitation light can be used. Further, the medical image acquired in the first aspect may be a white band light or a normal light image obtained by irradiating light in a plurality of wavelength bands as light in the white band, or may be a normal light image obtained based on the normal light image. It may be a special optical image having information in a specific wavelength band.

As described above, according to the first aspect, it is possible to acquire an image by a plurality of observation lights according to a purpose (observation, classification of a subject, etc.) while suppressing the influence on the user's observation.

In the first aspect, the image processing apparatus according to the second aspect further includes a region of interest detection unit that detects a region of interest from the first image acquired as a moving image, and the image acquisition control unit detects the region of interest. In this case, the acquisition instruction receiving unit is instructed to acquire the first image and the second image as still images. In the second aspect, the still image can be automatically acquired (without user instruction) in response to the detection of the region of interest. The attention region detection unit can perform detection processing of the attention region on the first image constituting one frame of the moving image. The area of interest is also referred to as the area of interest.

In the second aspect, the image processing apparatus according to the third aspect detects the area of interest as a subject from the first image and / or the second image as a still image, and the display control unit is stationary. The first image and / or the second image as an image is displayed on the display device by emphasizing the detected region of interest. According to the third aspect, since the first image and / or the second image as the still image is displayed with the area of interest emphasized, the user has triggered the acquisition of the first image and / or the second image. It is possible to easily confirm the position of the region of interest and which region the classification was performed on, and if the detection of the region of interest is erroneous, the first image and / or the second image is acquired. Since the target that triggered the detection can be confirmed, it is possible to easily determine that the detection is false. In the third aspect, the emphasis of the region of interest can be performed by marking with a specific figure such as a rectangle, a circle, a cross, an arrow, superimposition processing, changing the color tone or gradation, frequency processing, or the like. It is not limited to the mode.

The image processing apparatus according to the fourth aspect further includes a classification result storage unit that stores the classification result in association with the first image and / or the second image in any one of the first to third aspects. According to the fourth aspect, the relationship between the classification result and the first and second images becomes clear. In a fourth aspect, the classification result can be associated with a first image as a moving image, a first image as a still image and / or a second image.

In any one of the first to fourth aspects of the image processing device according to the fifth aspect, the display control unit causes the display device to display information indicating the result of classification. In the fifth aspect, the information can be displayed by, for example, characters, numbers, figures, symbols, colors, etc. according to the classification result, whereby the user can easily recognize the classification result. The information may be superimposed on the image or displayed separately from the image.

In the first or second aspect of the image processing device according to the sixth aspect, the display control unit causes the display device to display the first image and / or the second image as a still image. According to the sixth aspect, the user can confirm the still image (first image and / or second image) while observing with the first image (moving image), and the still image taken by this is inadequate. If there is, it is possible to make a decision such as re-shooting.

The image processing apparatus according to the seventh aspect further includes an image processing unit that performs image processing on the first image and / or the second image as a still image in any one of the first to third aspects. The display control unit causes the display device to display the image obtained by the image processing. In the seventh aspect, for example, image processing such as color balance adjustment, blood vessel enhancement, feature amount enhancement, difference enhancement, or composition of images subjected to these processes is performed to perform image processing such as observation image, classification (discrimination) image, and the like. Images can be generated and these images can be displayed.

In any one of the first to seventh aspects, the image processing apparatus according to the eighth aspect has a parameter calculation unit for calculating a parameter for aligning the first image and the second image, and a parameter for the first image. The display control unit further includes an image generation unit that generates the first alignment image by applying the image to the display device, and the display control unit displays the first alignment image on the display device at the timing when the second image is acquired. When an image is acquired by irradiating only one of the first observation light and the second observation light, the first image cannot be obtained at the timing of acquiring the second image. However, in the eighth aspect, since the alignment parameter is applied to the first image to generate the alignment first image, it is possible to prevent a substantial decrease in the frame rate of the first image. In addition, changes in the color and structure of the subject can be reduced between frames (between the frames of the first image and the frames of the first image aligned). In addition, in the eighth aspect and each of the following aspects, the "alignment first image" means "the first image at the shooting time of the second image generated by applying the alignment parameter to the first image". It means "image".

In the eighth aspect, the parameter calculation unit may calculate parameters for at least one of relative movement, rotation, and deformation of the first image and the second image as parameters. "Transformation" can include enlargement and reduction. In addition, the parameter calculation unit calculates the parameters that perform the projective transformation between the first image and the second image as parameters, and the image generation unit applies the projective transformation based on the calculated parameters to the first image to align the first image. It may be generated.

In the eighth aspect of the image processing apparatus according to the ninth aspect, the parameter calculation unit is photographed at a shooting time in which the time difference between the shooting time of the second image and the shooting time of the second image is equal to or less than the first threshold value. The parameters for aligning the first image and the first image are calculated. If the time difference of the shooting time exceeds the first threshold value, the shooting range, shooting angle, etc. may change due to the movement of the subject, and the alignment accuracy may decrease. The first image taken at the shooting time when the time difference from the shooting time of the second image is equal to or less than the first threshold value is acquired. As a result, it is possible to generate a first aligned image in which the structure of the subject is less changed than that of the first image. In the ninth aspect, the first threshold value can be set in consideration of conditions such as alignment accuracy.

In the eighth aspect of the image processing apparatus according to the tenth aspect, the parameter calculation unit determines the wavelength of the first observation light and the wavelength of the second observation light among the image signal of the first image and the image signal of the second image. The common wavelength component is extracted and weighted to the image signal component of the first image of the common wavelength component, and the intensity of the image signal component of the first image other than the common wavelength component is more common than the intensity of the image signal component of the common wavelength component. To generate an image signal in which the image signal component of the first image is relatively strong, and to weight the image signal component of the second image having a common wavelength component, the image signal component of the second image other than the common wavelength component is weighted. At least one of generating an image signal in which the image signal component of the second image having a common wavelength component rather than the intensity of the image signal component is relatively strong is performed, and the first image and the second image are aligned. Calculate the parameters to be used. As a result, it is possible to obtain an image (first alignment image) in which the alignment accuracy is improved and the change in color and structure of the subject between frames is small.

In the eighth aspect of the image processing apparatus according to the eleventh aspect, the parameter calculation unit determines the wavelength of the first observation light and the wavelength of the second observation light among the image signal of the first image and the image signal of the second image. The image signal component of the first image of the common wavelength component is generated by extracting the common wavelength component, and the image signal component of the second image of the common wavelength component is generated, and the first image and the second image are generated. Calculate the parameters to align with the image. As a result, it is possible to obtain an image (first alignment image) in which the alignment accuracy is improved and the change in color and structure of the subject between frames is small.

In the first to eleventh aspects of the image processing apparatus according to the twelfth aspect, when the display control unit acquires the second image, the time difference from the shooting time of the second image is equal to or less than the second threshold value. The first image taken at the shooting time is displayed on the display device. According to the twelfth aspect, since the display of the first image is continued at the timing when the second image is acquired, the display of the first image is interrupted or images having different colors are displayed for observation. The effect can be suppressed. In the twelfth aspect, the second threshold value can be set in consideration of the influence on the image (position, orientation, etc. of the subject) due to the deviation of the shooting time.

In any one of the first to twelfth aspects of the image processing apparatus according to the thirteenth aspect, the image acquisition unit captures an image in which light having a center wavelength shorter than that of the first observation light is used as the second observation light. Is acquired as the second image. Since the structure of the subject reflected in the image differs depending on the wavelength of the observation light, it is preferable to use the observation light having a short wavelength in order to photograph and detect a fine structure such as a lesion. In the thirteenth aspect, it is possible to accurately detect a fine structure, classify a subject, and the like using the second image while continuing the observation by displaying the first image.

In any one of the first to thirteenth aspects of the image processing apparatus according to the fourteenth aspect, the acquisition instruction receiving unit receives a still image acquisition instruction by the user as an acquisition instruction. According to the fourteenth aspect, the user can acquire an image at a desired timing.

In order to achieve the above-mentioned object, the endoscopic system according to the fifteenth aspect of the present invention is inserted into an image processing device, a display device, and a subject according to any one of the first to fourteenth aspects. An insertion portion having a hard tip portion, a curved portion connected to the base end side of the hard tip portion, and a soft portion connected to the base end side of the curved portion, and a base of the insertion portion. An endoscope having a hand operation unit connected to the end side, a light source device that irradiates the subject with the first observation light or the second observation light, and a photographing lens for forming an optical image of the subject. It includes an imaging element having an imaging element in which an optical image is formed by the photographing lens, and the photographing lens is provided on a hard tip portion. Since the endoscope system according to the fifteenth aspect includes the image processing device according to any one of the first to the fourteenth aspects, a plurality of observations are performed as necessary while suppressing the influence on the user's observation. Images by light can be acquired.

Further, since the endoscope system according to the fifteenth aspect includes the image processing device according to any one of the first to the fourteenth aspects, the advantageous effect of providing the image processing device is exhibited. That is, since the second image is not acquired even when the need for the second image is low (for example, when the region of interest or the like is not detected and the subject cannot be classified), the observation light is irradiated and the non-irradiation is performed. It is possible to prevent the deterioration of the light source from being unnecessarily accelerated due to the increase in repetition.

In the fifteenth aspect, the light emitted from the light source may be used as it is as the observation light, or the light generated by applying a filter that transmits a specific wavelength band to the light emitted from the light source may be used as the observation light. good. For example, when the narrow band light is used as the first observation light and / or the second observation light, the light emitted from the light source for the narrow band light may be used as the observation light, or a specific wavelength band with respect to the white light. The light generated by applying the filter that transmits the light may be used as the observation light. In this case, different narrow-band light may be irradiated at different timings by sequentially switching the filters applied to the white light.

In the fifteenth aspect of the endoscopic system according to the sixteenth aspect, the light source device irradiates the subject with white light including light in the wavelength bands of red, blue, and green as the first observation light, and the second aspect. The subject is irradiated with narrow-band light corresponding to any of the wavelength bands of red, blue, and green as observation light. According to the sixteenth aspect, while displaying and observing the first image taken with white light (first observation light), the area of interest is made using the second image taken with narrow band light (second observation light). Can be detected and classified. Narrow-band light corresponding to the violet and infrared wavelength bands may be used.

In the sixteenth aspect of the endoscope system according to the seventeenth aspect, the light source device is irradiated with a laser light source for white light that irradiates a laser for white light as excitation light and a laser for white light. A phosphor that emits white light as 1 observation light and a laser light source for narrow band light that irradiates narrow band light as second observation light are provided. When a laser light source for excitation light is used to obtain white light as the first observation light, if the acquisition frequency of the second image is high, the irradiation and non-irradiation of the first observation light are repeated more often, and therefore the white light. There is a possibility that the deterioration of the light source will be accelerated due to the repeated excitation and non-excitation of the laser light source. However, since the endoscope system according to the 17th aspect includes the image processing device according to any one of the 1st to 14th aspects, the advantageous effect of including the image processing device is exhibited. That is, since the second image is not acquired even when the need for the second image is low (for example, when the region of interest is not detected and classification is not necessary), the irradiation of the observation light and the non-irradiation are repeated. It is possible to prevent the deterioration of the light source from being unnecessarily accelerated due to the increase.

In the sixteenth aspect of the endoscope system according to the eighteenth aspect, the light source device includes a white light source that emits white light, a white light filter that transmits white light, and a narrow band light component of the white light. A narrow band light filter for transmitting and a first filter switching control unit for inserting a white light filter or a narrow band light filter into an optical path of white light emitted by a white light source are provided. When switching filters to generate multiple types of observation light (white light, narrow band light), the first image and / or the second image may be out of sync due to the synchronization shift between the filter switching and the read timing of the image sensor (image sensor). Although the color balance may be lost, the endoscopic system according to the eighteenth aspect includes the image processing device according to any one of the first to the fourteenth aspects, so that the image processing device is provided. A favorable effect is produced. That is, by acquiring the second image even when the need for the second image is low (for example, when the region of interest is not detected and classification is not necessary), the number of times the light source or filter is switched increases, and the first image and / Or the degree to which the color balance of the second image is lost can be reduced.

In the fifteenth aspect of the endoscopic system according to the nineteenth aspect, the light source device uses the first narrow band light corresponding to any of the wavelength bands of red, blue, and green as the first observation light as the subject. Is irradiated, and the subject is irradiated with a second narrow-band light corresponding to any of the wavelength bands of red, blue, and green as the second observation light and having a wavelength band different from that of the first narrow-band light. The nineteenth aspect defines an aspect in which a plurality of narrow band lights are used, for example, a plurality of blue narrow band lights having different wavelengths, a blue narrow band light and a green narrow band light, and a plurality of red narrow band lights having different wavelengths. However, the observation light is not limited to these combinations. Narrow-band light corresponding to the violet and infrared wavelength bands may be used.

In the nineteenth aspect of the endoscopic system according to the twentieth aspect, the light source device is a white light source that emits white light including light in the red, blue, and green wavelength bands, and the first narrower of the white light. A first narrow band light filter that transmits a component of band light, a second narrow band light filter that transmits a component of the second narrow band light of white light, and a first narrow band in the optical path of white light emitted by a white light source. A second filter switching control unit for inserting a band light filter or a second narrow band light filter is provided. When the filter is switched by the second filter switching control unit to generate a plurality of types of observation light (first narrow band light, second narrow band light), the filter switching and the reading timing of the image sensor (image sensor) are determined. The color balance of the first image and / or the second image may be lost due to the synchronization shift. However, since the endoscope system according to the twentieth aspect includes the image processing device according to any one of the first to the fourteenth aspects, the advantageous effect of providing the image processing device is exhibited. That is, by acquiring the second image even when the need for the second image is low (for example, when the region of interest is not detected and classification is not necessary), the number of times the light source or filter is switched increases, and the first image and / Or the degree to which the color balance of the second image is lost can be reduced.

In order to achieve the above-mentioned object, the image processing method according to the 21st aspect of the present invention is a first image taken by the first observation light and a second image taken by a second observation light different from the first observation light. It is an image processing method of an image processing apparatus including an image acquisition unit for acquiring two images, that is, an acquisition instruction receiving process for receiving a still image acquisition instruction, and acquisition of a first image and a second image by the image acquisition unit. The image acquisition control step includes an image acquisition control step to be controlled, a display control step to display at least the first image on the display device, and a classification step to classify the subject reflected in at least the second image. Until the acquisition instruction is received in the process, the image acquisition unit is made to perform a moving image acquisition process in which the first image is sequentially acquired as a moving image, and the first image and the second image are used as still images in response to the acceptance of the acquisition instruction. The image acquisition unit is made to perform the still image acquisition process to be acquired, and after the still image acquisition process is completed, the image acquisition unit is made to perform the moving image acquisition process. According to the 21st aspect, as in the 1st aspect, it is possible to acquire an image by a plurality of observation lights as needed while suppressing the influence on the user's observation.

The image processing method according to the 21st aspect may further include the same configurations as those in the 2nd to 14th aspects. Further, a program for causing an image processing device or an endoscopic system to execute the image processing method of these aspects, and a non-temporary recording medium on which a computer-readable code of the program is recorded can also be mentioned as an aspect of the present invention.

As described above, according to the image processing apparatus, the endoscope system, and the image processing method of the present invention, it is possible to acquire an image by a plurality of observation lights as necessary while suppressing the influence on the user's observation. Can be done.

FIG. 1 is an external view of the endoscope system according to the first embodiment. FIG. 2 is a block diagram showing a configuration of an endoscope system. FIG. 3 is a diagram showing a configuration of a hard tip portion of an endoscope. FIG. 4 is a diagram showing a functional configuration of the image processing unit. FIG. 5 is a diagram showing information recorded in the recording unit. FIG. 6 is a flowchart showing the procedure of image processing. FIG. 7 is a flowchart (continuation of FIG. 6) showing a procedure of image processing. FIG. 8 is a diagram showing a state of acquisition of a moving image and a still image. FIG. 9 is a diagram showing an example of displaying a moving image and a still image. FIG. 10 is a diagram showing an example of displaying a still image. FIG. 11 is a diagram showing a state in which the discrimination result of the region of interest is displayed together with the image. FIG. 12 is a diagram showing an example of highlighting the region of interest. FIG. 13 is another diagram showing an example of highlighting the region of interest. FIG. 14 is a diagram showing how the image and the classification result of the region of interest are associated and stored. FIG. 15 is another diagram showing how the image and the classification result of the region of interest are associated and stored. FIG. 16 is a flowchart showing processing for the first alignment image. FIG. 17 is a diagram showing how the first alignment image is created. FIG. 18 is a diagram showing a state in which the blue light component is weighted in the first still image. FIG. 19 is a diagram showing an example of the first alignment image. FIG. 20 is a diagram showing an example of the configuration of the light source. FIG. 21 is a diagram showing another example of the configuration of the light source. FIG. 22 is a diagram showing an example of a rotation filter. FIG. 23 is a diagram showing another example of the rotation filter.

Hereinafter, embodiments of an image processing apparatus, an endoscope system, and an image processing method according to the present invention will be described in detail with reference to the accompanying drawings. In the following description, the moving image acquired by irradiating the first observation light is referred to as the "first moving image", and the still image acquired by irradiating the first and second observation lights is referred to as the "first still image". , "Second still image" may be described.

<First Embodiment>

FIG. 1 is an external view showing an endoscope system 10 (image processing device, diagnostic support device, endoscope system, medical image processing device) according to the first embodiment, and FIG. 2 is an external view showing the endoscope system 10. It is a block diagram which shows the main part structure of. As shown in FIGS. 1 and 2, the endoscope system 10 includes an endoscope main body 100 (endoscope), a processor 200 (processor, image processing device, medical image processing device), and a light source device 300 (light source device). And a monitor 400 (display device).

<Structure of the endoscope body>

The endoscope main body 100 includes a hand operation unit 102 (hand operation unit) and an insertion unit 104 (insertion unit) connected to the hand operation unit 102. The operator (user) grips and operates the hand operation unit 102, inserts the insertion unit 104 into the body of the subject (living body), and observes it. Further, the hand operation unit 102 is provided with an air supply / water supply button 141, a suction button 142, a function button 143 to which various functions are assigned, and a shooting button 144 for receiving a shooting instruction operation (still image, moving image). .. The insertion portion 104 is composed of a soft portion 112 (soft portion), a curved portion 114 (curved portion), and a tip hard portion 116 (tip hard portion) in this order from the hand operation portion 102 side. That is, the curved portion 114 is connected to the base end side of the hard tip portion 116, and the soft portion 112 is connected to the base end side of the curved portion 114. The hand operation unit 102 is connected to the base end side of the insertion unit 104. The user can bend the curved portion 114 and change the direction of the hard tip portion 116 up, down, left and right by operating the hand operating portion 102. The hard tip portion 116 is provided with an imaging optical system 130 (imaging unit), an illumination unit 123, a forceps opening 126, and the like (see FIGS. 1 to 3).

During observation and treatment, by operating the operation unit 208 (see FIG. 2), white light and / or narrow band light (red narrow band light, green narrow band light,) from the illumination lenses 123A and 123B of the illumination unit 123. And one or more of blue narrow band light). Further, by operating the air supply / water supply button 141, cleaning water is discharged from a water supply nozzle (not shown) to clean the photographing lens 132 (photographing lens, imaging unit) of the photographing optical system 130 and the illumination lenses 123A and 123B. Can be done. A conduit (not shown) communicates with the forceps opening 126 opened by the hard tip 116, and a treatment tool (not shown) for removing a tumor or the like is inserted into this conduit, and the patient moves back and forth as appropriate to the subject. You can take the necessary measures.

As shown in FIGS. 1 to 3, a photographing lens 132 (imaging portion) is arranged on the tip end surface 116A of the tip rigid portion 116. A CMOS (Complementary Metal-Oxide Semiconductor) type image sensor 134 (image sensor, image sensor), a drive circuit 136, and an AFE 138 (AFE: Analog Front End) are arranged behind the photographing lens 132, and these elements are used. Output the image signal. The image pickup element 134 is a color image pickup element, and is composed of a plurality of light receiving elements arranged in a matrix (two-dimensional arrangement) in a specific pattern arrangement (Bayer arrangement, X-Trans (registered trademark) arrangement, honeycomb arrangement, etc.). It has a plurality of pixels. Each pixel of the image sensor 134 includes a microlens, a red (R), green (G), or blue (B) color filter and a photoelectric conversion unit (photodiode or the like). The photographing optical system 130 can also generate a color image from pixel signals of three colors of red, green, and blue, and generate an image from pixel signals of any one or two colors of red, green, and blue. You can also do it. Although the case where the image sensor 134 is a CMOS type image sensor will be described in the first embodiment, the image sensor 134 may be a CCD (Charge Coupled Device) type. Each pixel of the image sensor 134 may further include a purple color filter corresponding to a purple light source and / or an infrared filter corresponding to an infrared light source.

The optical image of the subject (tumor part, lesion part) is imaged on the light receiving surface (imaging surface) of the image sensor 134 by the photographing lens 132, converted into an electric signal, and output to the processor 200 via a signal cable (not shown). Is converted into a video signal. As a result, the observation image is displayed on the monitor 400 connected to the processor 200.

Further, on the tip end surface 116A of the tip rigid portion 116, the illumination lenses 123A and 123B of the illumination portion 123 are provided adjacent to the photographing lens 132. The ejection ends of the light guide 170, which will be described later, are arranged behind the illumination lenses 123A and 123B, and the light guide 170 is inserted into the insertion portion 104, the hand operation portion 102, and the universal cable 106, and the light guide 170 is inserted. The incident end is arranged within the light guide connector 108.

<Structure of light source device>

As shown in FIG. 2, the light source device 300 includes a light source 310 for illumination, a diaphragm 330, a condenser lens 340, a light source control unit 350, and the like, and causes observation light to enter the light guide 170. The light source 310 includes a red light source 310R, a green light source 310G, and a blue light source 310B that irradiate narrow-band light of red, green, and blue, respectively, and can irradiate narrow-band light of red, green, and blue. The illuminance of the observation light by the light source 310 is controlled by the light source control unit 350, and the illuminance of the observation light can be lowered and the illumination can be stopped as necessary.

The light source 310 can emit red, green, and blue narrow-band light in any combination. For example, red, green, and blue narrow-band light can be emitted at the same time to irradiate white light (normal light) as observation light, or one or two of them can be emitted to emit narrow-band light (special). It can also be irradiated with light). The light source 310 may further include a purple light source that irradiates purple light (an example of narrow band light) and / or an infrared light source that irradiates infrared light (an example of narrow band light). Further, white light or narrow band light may be irradiated as observation light by a light source that irradiates white light and a filter that transmits white light and each narrow band light (see, for example, FIGS. 20 to 23).

<Wavelength band of light source>

The light source 310 may be a light source having a white band or a light source having a plurality of wavelength bands as the light having a white band, or a light source having a specific wavelength band narrower than the white wavelength band. The specific wavelength band may be a blue band or a green band in the visible region, or a red band in the visible region. When a specific wavelength band is a visible blue band or green band, it includes a wavelength band of 390 nm or more and 450 nm or less, or 530 nm or more and 550 nm or less, and peaks in a wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less. It may have a wavelength. When the specific wavelength band is the red band in the visible region, the light in the specific wavelength band includes the wavelength band of 585 nm or more and 615 nm or less, or 610 nm or more and 730 nm or less, and the light in the specific wavelength band is 585 nm or more and 615 nm or less or 610 nm or more. It may have a peak wavelength in the wavelength band of 730 nm or less.

Even if the above-mentioned light in a specific wavelength band includes a wavelength band in which the oxidation hemoglobin and the reduced hemoglobin have different absorption coefficients and has a peak wavelength in the wavelength band in which the oxidation hemoglobin and the reduced hemoglobin have different absorption coefficients. good. In this case, the specific wavelength band includes a wavelength band of 400 ± 10 nm, 440 ± 10 nm, 470 ± 10 nm, or 600 nm or more and 750 nm, and 400 ± 10 nm, 440 ± 10 nm, 470 ± 10 nm, or 600 nm or more and 750 nm. It may have a peak wavelength in the following wavelength band.

Further, the light generated by the light source 310 may include a wavelength band of 790 nm or more and 820 nm or less, or 905 nm or more and 970 nm or less, and may have a peak wavelength in a wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less.

Further, the light source 310 may include a light source that irradiates excitation light having a peak of 390 nm or more and 470 nm or less. In this case, it is possible to acquire a medical image (in-vivo image) having information on the fluorescence emitted by the fluorescent substance in the subject (living body). When acquiring a fluorescence image, a dye for the fluorescence method (fluorestin, acridine orange, etc.) may be used.

The light source type (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode), etc.), wavelength, presence / absence of filter, etc. of the light source 310 are preferably configured according to the type of subject, the purpose of observation, and the like. When observing, it is preferable to combine and / or switch the wavelength of the observation light according to the type of the subject, the purpose of observation, and the like. When switching the wavelength, for example, by rotating a disk-shaped filter (rotary color filter) arranged in front of the light source and provided with a filter that transmits or blocks light of a specific wavelength, the wavelength of the emitted light is switched. It may be (see FIGS. 20 to 23).

Further, the image sensor used in carrying out the present invention is not limited to a color image sensor in which a color filter is provided for each pixel as in the image sensor 134, and a monochrome image sensor may be used. When a monochrome image sensor is used, the wavelength of the observation light can be sequentially switched to perform surface-sequential (color-sequential) imaging. For example, the wavelength of the emitted observation light may be sequentially switched between (blue, green, red), or the observation that emits by irradiating wideband light (white light) with a rotary color filter (red, green, blue, etc.) The wavelength of light may be switched. Further, the wavelength of the observation light emitted by the rotary color filter (green, blue, etc.) may be switched by irradiating one or a plurality of narrow band lights (green, blue, etc.). The narrow-band light may be infrared light having two or more wavelengths (first narrow-band light, second narrow-band light) having different wavelengths.

By connecting the light guide connector 108 (see FIG. 1) to the light source device 300, the observation light emitted from the light source device 300 is transmitted to the illumination lenses 123A and 123B via the light guide 170, and the illumination lenses 123A, The observation range is irradiated from 123B.

<Processor configuration>

The configuration of the processor 200 will be described with reference to FIG. The processor 200 inputs the image signal output from the endoscope main body 100 via the image input controller 202, performs necessary image processing in the image processing unit 204, and outputs the image signal via the video output unit 206. As a result, the observation image (in-vivo image) is displayed on the monitor 400 (display device). These processes are performed under the control of the CPU 210 (CPU: Central Processing Unit). That is, the CPU 210 functions as an image acquisition unit, an acquisition instruction reception unit, an image acquisition control unit, a display control unit, a classification unit, a region of interest detection unit, a classification result storage unit, an image processing unit, a parameter calculation unit, and an image generation unit. Have. The communication control unit 205 controls communication with an in-hospital system (HIS: Hospital Information System), an in-hospital LAN (Local Area Network), etc., which are not shown. The recording unit 207 records an image of the subject (medical image, captured image), information indicating the detection and / or classification result of the region of interest, and the like. The voice processing unit 209 outputs a message (voice) or the like according to the result of detection and / or classification of the region of interest from the speaker 209A under the control of the CPU 210 and the image processing unit 204.

Further, ROM 211 (ROM: Read Only Memory) is a non-volatile storage element (non-temporary recording medium), and the image processing method according to the present invention is applied to the CPU 210 and / or the image processing unit 204 (image processing device, computer). The computer-readable code of the program to be executed is stored. The RAM 212 (RAM: Random Access Memory) is a storage element for temporary storage during various processes, and can also be used as a buffer for image acquisition.

<Function of image processing unit>

FIG. 4 is a diagram showing a functional configuration of an image processing unit 204 (medical image acquisition unit, medical image analysis processing unit, medical image analysis result acquisition unit). The image processing unit 204 includes an image acquisition unit 204A (image acquisition unit), an acquisition instruction reception unit 204B (acquisition instruction reception unit), an image acquisition control unit 204C (image acquisition control unit), a display control unit 204D (display control unit), and classification. Section 204E (classification section), attention area detection section 204F (focus area detection section), classification result storage section 204G (classification result storage section), image processing section 204H (image processing section), parameter calculation section 204I (parameter calculation section) , And an image generation unit 204J (image generation unit). The classification unit 204E and the area of interest detection unit 204F also operate as a medical image analysis processing unit.

Further, the image processing unit 204 acquires a special optical image having information in a specific wavelength band based on a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in the white band. A special optical image acquisition unit may be provided. In this case, the signal of a specific wavelength band is used for RGB (R: red, G: green, B: blue) or CMY (C: cyan, M: magenta, Y: yellow) color information included in a normal optical image. It can be obtained by the calculation based on.

Further, the image processing unit 204 includes a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in the white band, and a special light image obtained by irradiating light in a specific wavelength band. A feature amount image generation unit that generates a feature amount image by an operation based on at least one of the above may be provided, and a feature amount image as a medical image (medical image) may be acquired and displayed. The image processing unit 204H may have the function of the feature amount image generation unit.

The details of the processing by these functions of the image processing unit 204 will be described later. The processing by these functions is performed under the control of the CPU 210.

The function of the image processing unit 204 described above can be realized by using various processors. Various processors include, for example, a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) to realize various functions. In addition, the various processors described above include programmable logic devices (PLCs) such as GPUs (Graphics Processing Units) and FPGAs (Field Programmable Gate Arrays), which are specialized processors for image processing, whose circuit configurations can be changed after manufacturing. Programmable Logic Device (PLD) is also included. Further, the above-mentioned various processors also include a dedicated electric circuit, which is a processor having a circuit configuration specially designed for executing a specific process such as an ASIC (Application Specific Integrated Circuit).

The functions of each part may be realized by one processor, or may be realized by a plurality of processors of the same type or different types (for example, a plurality of FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Further, a plurality of functions may be realized by one processor. As an example of configuring a plurality of functions with one processor, first, one processor is configured by a combination of one or more CPUs and software, as represented by a computer such as an image processing unit main body and a server. , There is a form in which this processor is realized as a plurality of functions. Secondly, as typified by System On Chip (SoC), there is a form in which a processor that realizes the functions of the entire system with one IC (Integrated Circuit) chip is used. As described above, various functions are configured by using one or more of the above-mentioned various processors as a hardware structure. Further, the hardware structure of these various processors is, more specifically, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.

When the above-mentioned processor or electric circuit executes software (program), the processor (computer) readable code of the software to be executed is stored in a non-temporary recording medium such as ROM (Read Only Memory), and the processor Refers to the software. Software stored on a non-temporary recording medium includes a program for performing image input and subject measurement. The code may be recorded on a non-temporary recording medium such as various optical magnetic recording devices or semiconductor memories instead of the ROM. During processing using software, for example, RAM (Random Access Memory) is used as a temporary storage area, and data stored in EEPROM (Electronically Erasable and Programmable Read Only Memory) (not shown) can also be referred to. can.

<Structure of operation unit>

The processor 200 includes an operation unit 208. The operation unit 208 is provided with an operation mode setting switch or the like (not shown), and can set the wavelength of the observation light (white light, narrow band light, or narrow band light in the case of narrow band light). .. Further, the operation unit 208 includes a keyboard and a mouse (not shown), and the user can perform an operation of setting shooting conditions and display conditions, or a shooting instruction (acquisition instruction) of a moving image or a still image via these devices. The shooting instruction of the moving image and the still image may be given by the shooting button 144). These setting operations may be performed via a foot switch (not shown), or may be performed by voice, line of sight, gesture, or the like.

<Structure of recording unit>

The recording unit 207 (recording device) includes various photomagnetic recording media, a non-temporary recording medium such as a semiconductor memory, and a control unit for these recording media, and includes a first moving image 207A (first image) and a first image. The still image 207B (first image), the second still image 207C (first image), the alignment first image 207D, the observation still image 207E, the attention area classification result 207F, and the like are recorded in association with each other. These images and information are displayed on the monitor 400 by the operation via the operation unit 208 and the control of the CPU 210 and / or the image processing unit 204.

In addition to the above-mentioned images, the analysis results of the area of interest (area of interest), which is a region of interest included in the medical image (medical image), and / or the presence / absence of a mode of interest, are recorded. It may be recorded in the unit 207 (recording device). In this case, the image processing unit 204 (medical image analysis processing unit, medical image analysis result acquisition unit) can acquire the analysis results from the recording unit 207 and display them on the monitor 400.

<Display device configuration>

The monitor 400 (display device) operates the operation unit 208, controls the CPU 210 and / or the image processing unit 204, and controls the first moving image 207A (first image), the first still image 207B (first image), and the first image processing unit 204. 2 Still image 207C (first image), alignment first image 207D, observation still image 207E, attention area classification result 207F, and the like are displayed. Further, the monitor 400 has a touch panel (not shown) for performing a shooting condition setting operation and / or a display condition setting operation.

<Image processing method>

An image processing method using the endoscope system 10 having the above-described configuration will be described. 6 to 7 are flowcharts showing the procedure of the image processing method according to the first embodiment.

<Observation light of the first image and the second image>

In the first embodiment, a white light image (normal image) in which white light is used as observation light (first observation light) is acquired as the first image, and blue light as narrow band light (center wavelength is the first observation light). A case where a blue light image (special light image) in which the observation light (second observation light) is used as the observation light (shorter) is acquired as the second image will be described. However, in the present invention, the observation light is not limited to such a combination. For example, the second image may be a special light image obtained by acquiring green light, red light, infrared light, purple light, or the like as narrow band light as observation light. Further, both the first observation light and the second observation light are designated as narrow-band light (for example, first narrow-band light and second narrow-band light such as blue light and green light, or red light having different wavelengths). An image and a second image may be acquired. In the first embodiment, it is assumed that the first image and the second image are captured by irradiating only the first observation light or only the second observation light in one frame.

<Acquisition of the first moving image>

The image acquisition unit 204A controls the light source control unit 350 to emit the red light source 310R, the green light source 310G, and the blue light source 310B to irradiate the subject with white light (first observation light). The first moving image (first image, normal optical image) of the subject is taken by the element 134 or the like (step S100: image acquisition control step, moving image acquisition process). Specifically, the frame images constituting the moving image are sequentially acquired at a rate of 30 frames / second, 60 frames / second, or the like. The image acquisition unit 204A acquires (inputs) the captured first moving image via the image input controller 202 (step S100: image acquisition step). The display control unit 204D displays the acquired first moving image on the monitor 400 (display device) (step S102: display control step).

<Detection of area of interest>

The attention area detection unit 204F (attention area detection unit) detects the attention area from each frame of the acquired first moving image (step S103: first attention area detection step). The region of interest can be detected by equipping the region of interest 204F with, for example, a known CAD system (CAD: Computer Aided Diagnosis). Specifically, for example, the presence or absence of a region of interest (a region of interest, which is a region of interest) and an object (an object of interest) in the region of interest can be extracted based on the feature amount of the pixels of the medical image. In this case, the region of interest detection unit 204F divides the image to be detected into, for example, a plurality of rectangular regions, and sets each of the divided rectangular regions as a local region. The attention area detection unit 204F calculates the feature amount (for example, hue) of the pixels in the local area for each local area of the detection target image, and determines the local area having a specific hue from each local area as the attention area. In step S103, "detecting the region of interest" means performing detection processing on the image.

<Detection of the region of interest based on the deep learning algorithm>

The detection of the region of interest may be performed using the result of deep learning. For example, every time a new image is recorded in the recording unit 207 (or every time a new image is taken), the region of interest detection unit 204F performs image analysis using deep learning based on the deep learning algorithm. By performing the processing, it is analyzed whether or not the image includes a region of interest. The deep learning algorithm is a known convolutional neural network (convolutional neural network) method, that is, whether or not the image contains a region of interest through the repetition of the convolutional layer and the pooling layer, the fully connected layer, and the output layer. It is an algorithm that recognizes.

In the image analysis process using deep learning, a learning device generated by giving an image labeled as "a region of interest" or "not a region of interest" as teacher data may be used. “Whether or not to perform such machine learning” and / or “whether or not to use the learning result” may be set according to the user's operation via the operation unit 208 and the monitor 400.

Examples of the region of interest (region of interest) detected in step S103 include polyps, cancer, colon diverticulum, inflammation, treatment scars (EMR: Endoscopic Mucosal Resection), ESD scars (ESD: Endoscopic Submucosal Dissection), and clip locations. Etc.), bleeding points, perforations, vascular atypia, etc.

The region of interest detection unit 204F determines whether or not the region of interest has been detected (step S104). If the determination is affirmed (when the region of interest is detected), the process proceeds to step S108, and a still image shooting instruction (acquisition instruction) is given. If the determination is denied (when the region of interest is not detected), the process proceeds to step S106, and it is determined whether or not the user has accepted the shooting instruction of the still image (acquisition instruction acceptance step). The shooting instruction by the user can be given by operating the shooting button 144 or the operation unit 208. If the determination in step S106 is denied, the process returns to step S102 and the acquisition and display of the first moving image are repeated (moving image acquisition process).

<Acceptance of still image shooting instructions and acquisition instructions>

When the determination in step S104 is affirmed (when the region of interest is detected), the image acquisition control unit 204C issues a still image shooting instruction (acquisition instruction) (step S108: still image acquisition instruction step). When a user instruction is received in step S106, a still image shooting instruction (acquisition instruction) is similarly given according to the instruction (step S108: still image acquisition instruction step). The acquisition instruction reception unit 204B receives a still image shooting instruction (acquisition instruction) (step S110: acquisition instruction reception step).

<Shooting still images>

When the still image shooting instruction (acquisition instruction) is received, the image acquisition unit 204A acquires one frame of the first moving image as the first still image under the control of the image acquisition control unit 204C (step S112: still image acquisition). Process, still image acquisition process). The frame to be used as the first still image can be a frame in which the region of interest is detected by the above-mentioned processing, but another frame (for example, another frame in which the shooting time difference from the frame in which the region of interest is detected is equal to or less than the threshold value). Frame) may be used. Further, the image acquisition unit 204A controls the light source control unit 350 under the control of the image acquisition control unit 204C, emits a blue light source 310B instead of the white light (first observation light), and emits blue light as narrow band light (first observation light). 2 Observation light) is applied to the subject, and a second still image of the subject is photographed (acquired) by the photographing optical system 130, the imaging element 134, or the like (step S114: still image acquisition step, still image acquisition process).

<Acquisition pattern of first image and second image>

FIG. 8 is a diagram showing an example of acquisition of a first image (first moving image, first still image) and a second image (second still image) in the first embodiment. These figures show how images are acquired from the left side to the right side of the figure along the time axis t. Part (a) of FIG. 8 shows a state in which the first moving image 500 (first image) by the first observation light (white light, normal light) is continuously photographed at a specified frame rate (frame interval: Δt). Is shown. In the portion (b) of FIG. 8, the still image acquisition instruction is received at the timing of (t = t1), and the image 605 (first image) in the moving image 600 is acquired as the first still image 701 accordingly. (Step S112), the image 606 (second image) is acquired as the second still image 702 (step S114). A plurality of first and second still images may be captured in response to an instruction to acquire a still image.

<Generation of still image for observation>

The image processing unit 204H performs image processing on the first still image 701 and / or the second still image 702 to generate an observation still image (step S116: image processing step). The image processing unit 204H can generate a white light image, a special light image, and an image based on a combination thereof. Further, the image processing unit 204H performs image processing such as color balance adjustment, blood vessel enhancement, feature amount enhancement, difference enhancement, or synthesizing images subjected to these processes, and performs image processing such as observation image, classification (discrimination) image, and the like. Image can be generated. For example, the image processing unit 204H can generate a blue region-enhanced image from a white light image and a blue narrow band light image. In the blue region-enhanced image, fine blood vessels on the surface layer of the mucous membrane of an organ, fine structures of the mucous membrane, and the like can be emphasized and displayed. In addition, the image processing unit 204H can generate a red region-enhanced image.

In the red region-enhanced image, a slight color difference in the red region of the image can be emphasized and displayed. The white light image is an image suitable for normal observation. These observation images allow the user to perform observations efficiently. The image processing may be determined according to the user's instruction, or the image processing unit 204H may determine the image processing without the user's instruction. The image processing unit 204H can record the generated still image for observation as the still image 207E for observation in the recording unit 207.

<Prevention of frame rate reduction of the first image>

In the first embodiment, in order to prevent deterioration of image quality due to wavelength separation, only the first observation light or the second observation light is irradiated as the observation light, and the first observation light and the second observation light are not simultaneously irradiated. The first image cannot be obtained at the irradiation timing of the second observation light. For example, when the first still image and the second still image are acquired by the pattern shown in the portion (b) of FIG. 8, the first still image cannot be acquired at the acquisition timing of the image 606 (second still image). Therefore, in the first embodiment, the "first image for alignment"("the first image at the shooting time of the second image generated by applying the alignment parameter to the first image" as follows. ) Is generated and displayed to prevent a substantial decrease in the frame rate of the first image (step S118: alignment first image generation step). The details of the process of generating the first alignment image will be described later.

<Detection of the region of interest from the second still image>

The attention area detection unit 204F detects the attention area as a subject from the first still image and / or the second still image (step S119: second attention area detection step). The detection of the region of interest can be performed in the same manner as in the first step of detecting the region of interest in step S103. When the frame of the first moving image in which the region of interest is detected in the process of step S103 is acquired as the first still image, the re-detection process for the first still image can be omitted.

<Classification of areas of interest>

The classification unit 204E classifies (identifies) the region of interest (an example of the subject) detected from the second still image in step S119 (step S120: classification step). Examples of classifications include the type of lesion (hyperplastic polyp, adenoma, intramucosal cancer, invasive cancer, etc.), extent of lesion, size of lesion, macroscopic shape of lesion, stage diagnosis of cancer, current location in the lumen. (In the upper part, the pharynx, esophagus, stomach, duodenum, etc., in the lower part, the cecum, ascending colon, transverse colon, descending colon, sigmoid colon, rectum, etc.) and the like. In these classifications, the results of machine learning (deep learning) can be used as in the case of detection. Further, the classification of the region of interest may be performed integrally with the detection. When the first observation light is white light and the second observation light is blue narrow band light, the classification unit 204E is focused on the region of interest (subject) based on at least the second still image of the first still image and the second still image. ) Is preferred. This is because, in the above example, the second still image is taken by blue narrow band light having a shorter central wavelength than the first observation light (white light), and is suitable for classifying fine structures such as lesions. Because. The image to be used for classifying the region of interest may be set based on the user's operation via the operation unit 208, or may be set by the classification unit 204E regardless of the user's operation. May be good.

<Display of still image>

The display control unit 204D displays a still image (first still image, second still image, still image for observation) on the monitor 400 (display device) (step S122: still image display step). The still image to be displayed may be an observation still image generated in step S116, in addition to the acquired first still image and second still image. These still images can be displayed in various patterns. For example, as shown in the portion (a) of FIG. 9, the moving image 800 may be continuously displayed on the monitor 400, and the first still image 802 may be displayed in another display area. Further, as shown in the portion (b) of FIG. 9, the second still image 804 may be displayed in addition to the first still image 802. The number of still images to be displayed is not limited to one, and a plurality of still images may be displayed. When displaying a plurality of still images, a display may be added each time a still image is acquired, and when the display area is full, the old still image may be deleted and the newly acquired still image may be displayed. Further, instead of displaying the moving image and the still image in parallel as shown in FIG. 9, only the still image 806 (the first still image and / or the second still image) is frozen for a certain period as shown in FIG. It may be displayed (the display of the same still image is continued). By displaying the still images illustrated in FIGS. 9 and 10, the user can confirm the still images such as the images used for classification (discrimination) during diagnosis (observation), and the images are blurred, halation, water fog, etc. If there is a defect in the above, it is possible to take measures such as instructing the shooting again.

<Display of classification results>

The display control unit 204D may display information indicating the classification result on the monitor 400 (display device) together with the still image (step S124: classification result display step). FIG. 11 is a diagram showing a display example of the classification result, and shows the still images 808, 810, 812 in addition to the moving image 800 and the classification results for those still images. In FIG. 11, "HP" indicates "Helicobacter Pylori" and "adenoma" indicates "adenoma". By displaying the classification result in this way, the user can evaluate the quality of the still image and the classification (discrimination) result at the same time, so that it is possible to determine which result should be trusted when the discrimination result of the same lesion is different. The classification unit 204E and the attention area detection unit 204F may transmit the detection result of the attention area and / or the information indicating the classification result by voice via the voice processing unit 209 and the speaker 209A.

<Highlight area of interest>

In displaying the image and the classification result, the display control unit 204D, the classification unit 204E, and the attention area detection unit 204F can highlight the attention area. In the information output, the display control unit 204D, the classification unit 204E, the attention area detection unit 204F, and the like output, for example, characters, numbers, symbols, colors, etc. indicating the position and size of the attention area as the first still image and / or the second. This can be done by superimposing and displaying on a still image. FIG. 12 shows an example of such highlighting, and shows a case where a rectangle 820 surrounding a region of interest to be classified is displayed in addition to the display of the classification result shown in FIG. The highlighting may be performed on the display mode shown in FIGS. 9 and 10 (without displaying the classification result). FIG. 13 is a diagram showing an example in which highlighting is performed in the embodiment shown in FIG. When the first still image and / or the second still image is displayed without highlighting the area of interest, the user needs to confirm the area of interest throughout the image. Can easily confirm which area was targeted for detection and classification of the area of interest. Further, if the detection of the region of interest is erroneous detection, it can be easily determined that the first still image and / or the second still image does not include the region of interest and is erroneous detection. In addition to the examples shown in FIGS. 12 and 13 (rectangular display), the area of interest is emphasized by marking with specific figures such as circles, crosses, and arrows, superimposing processing, changing color tone and gradation, frequency processing, and the like. However, it is not limited to such an embodiment.

<Saving classification results and images>

The classification result storage unit 204G associates the classification result with the first still image and / or the second still image, and stores the classification result as the attention area classification result 207F in the recording unit 207 (step S126: classification result storage step). The result of classification may be associated with the above-mentioned still image for observation. 14 and 15 are diagrams showing an example in which the classification result and the image are associated and saved. FIG. 14 shows how the subfolders 1010, 1020, 1030, 1040, 1050, and 1060 associated with the moving image are stored in the main folder 1000 created in the recording unit 207. FIG. 15 is a diagram showing images and information stored in the subfolder 1010, in which the moving image 1011 and the subfolders 1012 and 1013 associated with the moving image 1011 are stored, whereby the moving image 1011 and the subfolders 1012 and 1013 are associated with each other. It shows how it looks. The first still image 1012A, the second still image 1012B, the observation still image 1012C, and the classification result 1012D are stored in the subfolder 1012, and it is shown that these still images are related to the classification result. Similarly, the subfolder 1013 stores the first still image 1013A, the second still image 1013B, the observation still image 1013C, and the classification result 1013D, indicating that these still images and the classification result are related. By saving in such folders, the user can easily grasp the correspondence between the image and the classification result.

As described above, when the image and the classification result are saved, the image in which the region of interest such as a lesion is detected (hereinafter referred to as "lesion image") is a specific lesion (low prevalence, difficult-to-detect case). Etc.) may be saved (recorded) for the inspection in which it was found. For example, when the size of the lesion is small, or when the shape of the lesion is flat and there is almost no ridge, the lesion image (still image, moving image) can be saved as a “lesion that is difficult to detect”. Also, for example, when a pathological biopsy is performed (in this case, it is considered that it is difficult to judge the lesion to be biopsied by endoscopic findings), or the result of the pathological biopsy and the endoscopic findings. If there is an inconsistency (for example, an endoscopic finding was "suspected adenoma" and a biopsy was taken, but the pathological result was a hyperplastic polype, etc.), the lesion image was saved as a "lesion difficult to diagnose". can do. Further, when the learning device is constructed by machine learning using the lesion image as an input, the lesion image may be saved according to the purpose of use of the learning device. For example, when constructing a learning device for the purpose of detecting (picking up) lesions during screening, only the examination for screening purposes is saved (ESD (Endoscopic Submucosal Dissection)). When constructing a learning device for the purpose of stage discrimination (intramucosal cancer, advanced cancer, etc.) of cancer (low utility value for learning, etc.), about examinations for therapeutic purposes such as ESD and EMR (EMR: Endoscopic Mucosal Resection) It is conceivable to save only the lesion image of.

After saving the classification result and the image, the image processing unit 204 (image acquisition control unit 204C) determines whether or not to end the processing of the image processing method (step S128: end determination step). When the process is continued (NO in step S128), the still image acquisition process (acquisition of the first and second still images and their display, etc.) is ended, and the process returns to step S102 to restart the moving image acquisition process.

<Processing for the first alignment image>

Hereinafter, the processing of the alignment first image in step S118 of FIG. 7 will be described in detail with reference to the flowchart of FIG.

<Image used to generate the first alignment image>

For the generation of the first image for alignment, for example, the missing timing of the first still image (such as the image 605 (first still image 701) and the image 606 (second still image 702) in the portion (b) of FIG. 8). The first still image (image 605) acquired before the second still image (shooting timing of the image 606) can be used. Specifically, "the first still image taken at the shooting time before the shooting time of the second still image and the time difference from the shooting time of the second still image is equal to or less than the first threshold value". Can be used. As a result, it is possible to generate a first aligned image in which changes in the color and structure of the subject between frames are small. The threshold value for the shooting time (first threshold value) can be determined according to the alignment accuracy, the allowable time for image generation and display delay, and the like. Hereinafter, a case where the alignment first image is generated by using the image 605 as the first still image (first still image 701) and the image 606 as the second still image (second still image 702) will be described.

In the generation of the alignment first image, in addition to the above-mentioned pattern, a plurality of first still images having different shooting times (for example, images 604 and 605 in the portion (b) of FIG. 8) may be used. , The first still image acquired after the missing timing of the first still image (for example, images 607 and 608 in the part (b) of FIG. 8) may be used. For example, in the case of a system in which there is a delay from image acquisition to display, it is possible to align using the first image (for example, images 607 and 608) taken after the shooting time of the second image, and the first image. By aligning using the first image (for example, images 605 and 607) taken at a time before and after the shooting time of the two images, highly accurate alignment can be performed.

<Correction before alignment (pre-processing)>

The wavelength of the observation light is different between the first still image and the second still image in addition to the shooting timing. As a result, in the first still image 701 using white light as the observation light, for example, as shown in the portion (a) of FIG. 17, the thick blood vessel 601 is clearly shown, but the fine blood vessel 602 is not clearly shown. On the other hand, in the second still image 702 in which the blue narrow band light is used as the observation light, for example, as shown in the portion (b) of FIG. 17, the thick blood vessel 601 is unclear as compared with the first still image 701. The small blood vessels 602 are clearly visible. Therefore, in the first embodiment, the image processing unit 204 (parameter calculation unit 204I) corrects the difference between the first still image and the second still image due to the difference between the first observation light and the second observation light. (Preprocessing) is performed (step S200: image correction step).

Specifically, the parameter calculation unit 204I extracts and extracts a component having a wavelength common to the first observation light and the second observation light from the image signal of the first still image and the image signal of the second still image. The wavelength component is weighted to at least one of the image signal of the first still image and the image signal of the second still image, and the signal intensity of the component of the common wavelength is higher than the signal intensity of the component other than the component of the common wavelength. Also produces a relatively strong image. In the first embodiment, since the first observation light is white light and the second observation light is blue light, the parameter calculation unit 204I uses the common wavelength of the image signal of the first still image and the image signal of the second still image. Increases the weight of a certain blue light component. FIG. 18 is an example showing a state in which the blue light component is weighted in the first still image 701, and the fine blood vessels 602 are relatively emphasized.

In the first embodiment, it is possible to obtain an image (first alignment image) in which the alignment accuracy is improved by such correction (preprocessing) and the change in color and structure of the subject between frames is small. In addition, instead of weighting the common wavelength component (blue light component) as described above, the alignment first image may be generated by using only the common wavelength component.

<Parameter calculation and alignment>

The parameter calculation unit 204I calculates a parameter for aligning the corrected (preprocessed) first still image 701 and the second still image 702 (step S202: parameter calculation step). The parameters to be calculated are parameters for at least one of relative movement, rotation, and deformation, and "deformation" may include enlargement or reduction. The image generation unit 204J applies the generated parameters to the corrected first still image 701 to generate an alignment first image (step S204: image generation step). In steps S202 and S204, the parameter calculation unit 204I calculates the parameters for performing the projection conversion between the first still image and the second still image, and the image generation unit 204J performs the first projection conversion based on the calculated parameters. It can be applied to a still image to generate a first alignment image. An example of the first image for alignment (image 710) is shown in FIG. As described above, the second still image is used when calculating the parameters, but the alignment first image is generated by moving, deforming, etc. the first still image, so that the pixel value of the second still image is The tint of the alignment first image does not change due to the influence.

<Output for the first alignment image>

The display control unit 204D and the image generation unit 204J display the first alignment image on the monitor 400 (display device) (step S206: display control step). Further, the display control unit 204D and the image generation unit 204J record the generated first alignment image as "alignment first image 207D" in the recording unit 207 (step S208: alignment first image recording step). The display and recording of the alignment first image can be performed sequentially following the display and recording of each frame of the first moving image. Such sequential display may be performed in real time during the inspection of the subject, or the first moving image and the first alignment image are recorded in the recording unit 207 (first moving image 207A, first alignment first). Image 207D) This may be done when the user looks back at the image later. When the first image for alignment is generated by performing the above-mentioned correction (weighting of the blue light component) for the first image, the image generation unit 204J is used when outputting (displaying, etc.) the first image for alignment. May restore the balance of the wavelength components and make it the same as white light. As a result, it is possible to prevent the user from feeling uncomfortable when images having different wavelength balances are displayed on the monitor 400.

As described above, in the first embodiment, in addition to the normal display of the first moving image (step S102), the alignment first image can be displayed even at the timing when the first image cannot be acquired (step S206). This prevents a substantial decrease in the frame rate of the first moving image, and the user can continue the observation with the normal light image (first image) taken by the normal light (white light).

In the flowcharts of FIGS. 6 and 7, the mode of displaying the alignment first image at the timing when the first image cannot be acquired has been described, but the display control unit 204D continues to display the first image instead of the alignment first image. You may. In this case, at the timing when the second image is acquired, the display control unit 204D monitors the monitor 400 ( It is preferable to display on a display device). For example, assuming that the second threshold value is (2 × Δt; Δt is the frame interval of the moving image), in the example shown in FIG. 8, the images 605 and 607 (time difference) at the acquisition timing of the image 606 which is the second image. Is Δt), or images 607 and 608 (time difference is 2 × Δt) can be continuously displayed.

As described above, in the endoscope system 10 according to the first embodiment, the user continues the observation by the first image and if necessary (for example, when instructed by the user or the area of interest). The still image can be acquired by the first and second observation lights (at the timing when it is necessary to acquire the still image, such as when it is detected), and the subject can be classified together with the observation. In addition, it is possible to obtain an image in which the change in the color and structure of the subject between frames is small while preventing a substantial decrease in the frame rate of the image display (first image) by generating and displaying the first alignment image. This makes it possible to observe the exact structure of the subject. As described above, in the endoscope system 10, it is possible to acquire images by a plurality of observation lights as needed while suppressing the influence on the user's observation.

<Other configurations of the light source and the effect of applying the present invention>

Examples of other configurations of the light source in the endoscope system of the present invention, and the effect of applying the image processing apparatus of the present invention in that case will be described.

(Example 1)

As shown in FIG. 20, the light source device 320 (light source device) is irradiated with a white light laser light source 312 (white light laser light source) that irradiates a white light laser as excitation light and a white light laser. Thereby, the phosphor 314 (phosphor) that emits white light as the first observation light and the narrow band light as the second observation light (for example, blue narrow band light can be used, but green narrow band light, It includes a narrow band light laser light source 316 (narrow band light laser light source) that irradiates red narrow band light). The light source device 320 is controlled by the light source control unit 350. In FIG. 20, the components other than the light source device 320 and the light source control unit 350 among the components of the endoscope system 10 are not shown.

When the laser light source 312 for white light is used to obtain white light as the first observation light, if the number of acquisitions of the second image is large, the irradiation and non-irradiation of the first observation light are repeated many times, and therefore the white light. There is a possibility that the deterioration of the light source will be accelerated due to the repetition of excitation and non-excitation of the laser light source 312. However, since the endoscope system 10 includes the image processing device according to the present invention, the advantageous effect of including the image processing device is exhibited. That is, the second image is acquired only when there is an instruction to acquire the still image (when the region of interest is detected or when there is an instruction from the user), and when there is no instruction to acquire (for example, the region of interest is not detected). Since the second image is not acquired (when there is no need for classification), it is possible to prevent the repetition of irradiation and non-irradiation of the first observation light from increasing and the deterioration of the light source from being unnecessarily accelerated.

(Example 2)

As shown in FIG. 21, the light source device 322 (light source device) includes a white light source 318 (white light source) that emits white light, a white light region that transmits white light, and a narrow band light region that transmits narrow band light. The rotation filter 360 (white light filter, narrow band light filter) in which the A first filter switching control unit) is provided. The white light source 318 and the rotation filter control unit 363 are controlled by the light source control unit 350. In FIG. 21, the components other than the light source device 322 and the light source control unit 350 among the components of the endoscope system 10 are not shown.

When controlling the rotation of the rotation filter 360 to generate a plurality of types of observation light (for example, white light as the first observation light and narrow band light as the second observation light), the rotation of the rotation filter 360 and the image sensor (for example, There is a possibility that the color balance of the first image and / or the second image will be lost due to the synchronization deviation with the reading timing of the image sensor 134). However, since the endoscope system 10 includes the image processing device according to the present invention, the advantageous effect of including the image processing device is exhibited. That is, the second image is acquired only when there is an instruction to acquire a still image (when a region of interest is detected, when there is a user instruction), and when there is no instruction to acquire (for example, the region of interest is detected). Since the second image is not acquired when there is no need for classification), the number of times the light source or filter is switched increases, and the degree to which the color balance of the first image and / or the second image is lost is reduced. be able to.

In Example 2, the white light source 318 may use a white light source that emits a wide band of light, or may generate white light by simultaneously irradiating a light source that emits red, blue, and green light. Further, such a rotation filter 360 and a rotation filter control unit 363 may be provided in the light source 310 shown in FIG.

FIG. 22 is a diagram showing an example of a rotation filter 360. In the example shown in the portion (a) of FIG. 22, the rotation filter 360 has two circular white light regions 362 (white light filters) that transmit white light and one circular narrow band light region that transmits narrow band light. A 364 (narrow band optical filter) is formed, and the white light region 362 or the narrow band optical region 364 is formed by rotating around the rotation shaft 361 under the control of the rotation filter control unit 363 (first filter switching control unit). It is inserted into the light path of white light, which irradiates the subject with white light or narrow band light. The narrow band light region 364 can be a region that transmits arbitrary narrow band light such as red, blue, green, and purple. Further, the number, shape, and arrangement of the white light region 362 and the narrow band light region 364 are not limited to the example shown in the part (a) of FIG. 22, and are changed according to the irradiation ratio of the white light and the narrow band light. good.

The shapes of the white light region and the narrow band light region are not limited to the circular shape as shown in the portion (a) of FIG. 22, and may be fan-shaped as shown in the portion (b) of FIG. 22. Part (b) of FIG. 22 shows an example in which three-quarters of the rotation filter 360 is a white light region 362 and a quarter is a narrow-band light region 364. The area of the fan shape can be changed according to the irradiation ratio of white light and narrow band light.

In the example of FIG. 22, a plurality of narrow band optical regions corresponding to different narrow band lights may be provided in the rotation filter 360.

FIG. 23 is a diagram showing another example of the rotation filter. As the white light source for the rotation filter shown in FIG. 22, a white light source 318 can be used as in the light source device 322 shown in FIG. 23. Further, unlike the rotation filter 360 shown in FIG. 21, the rotation filter 369 shown in the portion (a) of FIG. 23 is not provided with a white light region for transmitting white light, and is a component of the first narrow band light of the white light. Two circular first narrow-band light regions 365 (first narrow-band optical filter) that transmit light, and one circular second narrow-band light region 367 (second narrow-band light filter) that transmits the components of the second narrow-band light. An optical filter) is provided. By rotating such a rotation filter 369 around the rotation shaft 361 by the rotation filter control unit 363 (see FIG. 21; the second filter switching control unit), the first is the optical path of the white light emitted by the white light source 318. A narrow-band optical region 365 (first narrow-band optical filter) or a second narrow-band optical region 367 (second narrow-band optical filter) is inserted to irradiate a subject with first narrow-band light or second narrow-band light. Can be done.

The shapes of the first narrow-band optical region 365 and the second narrow-band optical region 367 are not limited to a circle as shown in the portion (a) of FIG. 23, and may be fan-shaped as shown in the portion (b) of FIG. 23. .. Part (b) of FIG. 23 shows an example in which two-thirds of the rotation filter 369 is the first narrow-band optical region 365 and one-third is the second narrow-band optical region 367. The area of the fan shape can be changed according to the irradiation ratio of the first narrow band light and the second narrow band light. In the example of FIG. 23, the rotation filter 369 may be provided with three or more types of narrow band optical regions corresponding to different narrow band lights.

When the rotary filter control unit 363 switches the filter to generate a plurality of types of observation light (first narrow band light, second narrow band light), synchronization between the filter switching and the read timing of the image sensor (image sensor 134). The color balance of the first image and / or the second image may be lost due to the deviation. However, since the endoscope system 10 includes the image processing device according to the present invention, the advantageous effect of including the image processing device is exhibited. That is, when there is no instruction to acquire the second image (for example, when the region of interest is not detected and there is no need to classify), the second image is not acquired, so that the number of times the light source or filter is switched increases. , The degree to which the color balance of the first image and / or the second image is lost can be reduced.

(Additional note)

In addition to each aspect of the embodiments described above, the configurations described below are also included in the scope of the present invention.

(Appendix 1)

The medical image analysis processing unit detects a region of interest, which is a region of interest, based on the feature amount of pixels of the medical image.

The medical image analysis result acquisition unit is a medical image processing device that acquires the analysis results of the medical image analysis processing unit.

(Appendix 2)

The medical image analysis processing unit detects the presence or absence of a noteworthy object based on the feature amount of the pixel of the medical image.

The medical image analysis result acquisition unit is a medical image processing device that acquires the analysis results of the medical image analysis processing unit.

(Appendix 3)

The medical image analysis result acquisition department

Obtained from a recording device that records the analysis results of medical images

The analysis result is a medical image processing apparatus that is a region of interest that is a region of interest included in a medical image, and / or a presence / absence of an object of interest.

(Appendix 4)

A medical image is a medical image processing apparatus that is a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in a white band.

(Appendix 5)

A medical image is an image obtained by irradiating light in a specific wavelength band.

A medical image processing device in which a specific wavelength band is narrower than the white wavelength band.

(Appendix 6)

A medical image processing device in which a specific wavelength band is a blue or green band in the visible range.

(Appendix 7)

A medical treatment in which a specific wavelength band includes a wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less, and light in a specific wavelength band has a peak wavelength in a wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less. Image processing device.

(Appendix 8)

A medical image processing device in which a specific wavelength band is a red band in the visible range.

(Appendix 9)

The specific wavelength band includes a wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less, and the light of the specific wavelength band has a peak wavelength in the wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less. Image processing device.

(Appendix 10)

The specific wavelength band includes a wavelength band in which the absorption coefficient differs between the oxidized hemoglobin and the reduced hemoglobin, and the light in the specific wavelength band has a peak wavelength in the wavelength band in which the absorption coefficient differs between the oxidized hemoglobin and the reduced hemoglobin. Medical image processing device.

(Appendix 11)

The specific wavelength band includes a wavelength band of 400 ± 10 nm, 440 ± 10 nm, 470 ± 10 nm, or 600 nm or more and 750 nm or less, and the light of the specific wavelength band is 400 ± 10 nm, 440 ± 10 nm, 470 ±. A medical image processing apparatus having a peak wavelength in a wavelength band of 10 nm or 600 nm or more and 750 nm or less.

(Appendix 12)

A medical image is an in-vivo image that shows the inside of a living body.

An in-vivo image is a medical image processing device that has information on fluorescence emitted by a fluorescent substance in the living body.

(Appendix 13)

Fluorescence is a medical image processing device obtained by irradiating a living body with excitation light having a peak of 390 or more and 470 nm or less.

(Appendix 14)

A medical image is an in-vivo image that shows the inside of a living body.

A specific wavelength band is a medical image processing device that is a wavelength band of infrared light.

(Appendix 15)

A medical image in which a specific wavelength band includes a wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less, and light in a specific wavelength band has a peak wavelength in a wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less. Processing equipment.

(Appendix 16)

The medical image acquisition unit acquires a special optical image having information in a specific wavelength band based on a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in the white band. Equipped with an optical image acquisition unit

A medical image is a medical image processing device that is a special optical image.

(Appendix 17)

A medical image processing device that obtains a signal in a specific wavelength band by calculation based on RGB or CMY color information included in a normal optical image.

(Appendix 18)

By calculation based on at least one of a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in a white band and a special light image obtained by irradiating light in a specific wavelength band. Equipped with a feature amount image generation unit that generates feature amount images

A medical image is a medical image processing device that is a feature image.

(Appendix 19)

The medical image processing apparatus according to any one of Supplementary notes 1 to 18.

An endoscope that irradiates at least one of light in a white wavelength band or light in a specific wavelength band to acquire an image, and

Endoscope device equipped with.

(Appendix 20)

A diagnostic support device including the medical image processing device according to any one of Supplementary notes 1 to 18.

(Appendix 21)

A medical business support device including the medical image processing device according to any one of Supplementary notes 1 to 18.

Although the embodiments and other aspects of the present invention have been described above, the present invention is not limited to the above-described aspects, and various modifications can be made without departing from the spirit of the present invention.

10 Endoscopic system

100 Endoscope body

102 Hand operation unit

104 Insert

106 universal cable

108 Light Guide Connector

112 Flexible part

114 Curved part

116 Hard tip

116A Tip side end face

123 Lighting unit

123A Illumination lens

123B Illumination lens

126 Forceps mouth

130 Imaging optical system

132 Shooting lens

134 Image sensor

136 drive circuit

138 AFE

141 Air supply Water supply button

142 suction button

143 function button

144 Shoot button

170 Light Guide

200 processors

202 image input controller

204 Image processing unit

204A Image acquisition section

204B Acquisition instruction reception department

204C Image acquisition control unit

204D display control unit

204E Classification Department

204F Area of interest detector

204G classification result storage unit

204H Image processing section

204I Parameter calculation unit

204J image generator

205 Communication control unit

206 Video output

207 Recording unit

207A 1st moving image

207B 1st still image

207C 2nd still image

207D Alignment 1st image

207E Still image for observation

207F Area of interest classification result

208 Operation unit

209 Speech processing unit

209A speaker

210 CPU

211 ROM

212 RAM

300 light source device

310 light source

310B blue light source

310G green light source

310R red light source

312 Laser light source for white light

314 Fluorescent material

316 Laser light source for narrow band light

318 white light source

320 light source device

322 Light source device

330 Aperture

340 Condensing lens

350 Light source control unit

360 rotation filter

361 rotating shaft

362 White light region

363 Rotation filter control unit

364 Narrow band optical region

365 1st narrow band optical region

367 2nd narrow band optical region

369 rotation filter

400 monitor

500 1st moving image

600 video

601 blood vessels

602 blood vessels

604 image

605 images

606 images

607 image

608 images

701 First still image

702 Second still image

710 images

800 moving image

802 1st still image

804 Second still image

806 Still image

808 Still image

810 Still image

812 Still image

820 rectangle

1000 main folder

1010 subfolder

1011 moving image

1012 subfolder

1012A 1st still image

1012B 2nd still image

1012C Still image for observation

1012D classification result

1013 subfolder

1013A 1st still image

1013B 2nd still image

1013C Still image for observation

1013D classification result

1020 subfolder

1030 subfolder

1040 subfolder

1050 subfolder

1060 subfolder

S100 to S208 Each step of the image processing method

t time axis

Claims (21)

An image acquisition unit that acquires a first image taken by the first observation light and a second image taken by a second observation light different from the first observation light.

The acquisition instruction reception unit that accepts still image acquisition instructions,

An image acquisition control unit that controls the acquisition of the first image and the second image by the image acquisition unit.

A display control unit that displays at least the first image on a display device,

At least a classification unit that classifies the subject reflected in the second image,

With

The image acquisition control unit

Until the acquisition instruction receiving unit receives the acquisition instruction, the image acquisition unit is made to perform a moving image acquisition process for sequentially acquiring the first image as a moving image.

In response to the acceptance of the acquisition instruction, the image acquisition unit is made to perform a still image acquisition process for acquiring the first image and the second image as still images.

An image processing device that causes the image acquisition unit to perform the moving image acquisition process after the still image acquisition process is completed.

A region of interest detection unit that detects a region of interest from the first image acquired as the moving image is further provided.

The image processing according to claim 1, wherein the image acquisition control unit instructs the acquisition instruction receiving unit to acquire the first image and the second image as the still image when the region of interest is detected. Device.

The area of interest detection unit detects the area of interest as the subject from the first image and / or the second image as the still image.

The display control unit displays the first image and / or the second image as the still image.

The image processing device according to claim 2, wherein the detected region of interest is emphasized and displayed on the display device.

The image processing apparatus according to any one of claims 1 to 3, further comprising a classification result storage unit that stores the classification results in association with the first image and / or the second image.

The image processing device according to any one of claims 1 to 4, wherein the display control unit displays information indicating the result of the classification on the display device.

The image processing device according to claim 1 or 2, wherein the display control unit displays the first image and / or the second image as the still image on the display device.

An image processing unit that performs image processing on the first image and / or the second image as the still image is further provided.

The image processing device according to any one of claims 1 to 3, wherein the display control unit displays an image obtained by the image processing on the display device.

A parameter calculation unit that calculates a parameter for aligning the first image and the second image,

An image generation unit that applies the parameters to the first image to generate a first alignment image is further provided.

The image processing device according to any one of claims 1 to 7, wherein the display control unit displays the alignment first image on the display device at the timing when the second image is acquired.

The parameter calculation unit aligns the second image and the first image taken at a shooting time when the time difference between the shooting time of the second image is equal to or less than the first threshold value. The image processing apparatus according to claim 8.

The parameter calculation unit extracts a wavelength component common to the wavelength of the first observation light and the wavelength of the second observation light from the image signal of the first image and the image signal of the second image, and the above-mentioned The image of the first image having the common wavelength component rather than the intensity of the image signal component of the first image other than the common wavelength component by weighting the image signal component of the first image of the common wavelength component. An image of the second image other than the common wavelength component is generated by generating an image signal in which the signal component is relatively strong, and by weighting the image signal component of the second image of the common wavelength component. At least one of generating an image signal in which the image signal component of the second image having the common wavelength component is relatively stronger than the intensity of the signal component is performed, and the first image and the second image are displayed. The image processing apparatus according to claim 8, wherein the parameters for aligning the images are calculated.

The parameter calculation unit extracts a wavelength component common to the wavelength of the first observation light and the wavelength of the second observation light from the image signal of the first image and the image signal of the second image, and the above-mentioned The image signal component of the first image having a common wavelength component is generated, and the image signal component of the second image having the common wavelength component is generated, and the first image and the second image are aligned with each other. The image processing apparatus according to claim 8, wherein the parameters to be used are calculated.

At the timing when the second image is acquired, the display control unit displays the first image captured at the shooting time when the time difference from the shooting time of the second image is equal to or less than the second threshold value. The image processing apparatus according to any one of claims 1 to 11.

The method according to any one of claims 1 to 12, wherein the image acquisition unit acquires an image captured as the second observation light with light having a center wavelength shorter than that of the first observation light as the second observation light. Image processing device.

The image processing device according to any one of claims 1 to 13, wherein the acquisition instruction receiving unit receives a still image acquisition instruction by a user as the acquisition instruction.

The image processing apparatus according to any one of claims 1 to 14.

With the display device

An insertion portion to be inserted into a subject, which has a hard tip portion, a curved portion connected to the proximal end side of the rigid tip end portion, and a soft portion connected to the proximal end side of the curved portion. An endoscope having a portion and a hand operation portion connected to the proximal end side of the insertion portion.

A light source device that irradiates the subject with the first observation light or the second observation light.

An imaging unit having a photographing lens for forming an optical image of the subject and an image pickup element for forming the optical image with the photographing lens.

With

The photographing lens is an endoscope system provided on the hard tip portion.

The light source device irradiates the subject with white light including light in the wavelength bands of red, blue, and green as the first observation light, and any one of red, blue, and green as the second observation light. The endoscopic system according to claim 15, wherein the subject is irradiated with narrow band light corresponding to the wavelength band of.

The light source device includes a white light laser light source that irradiates a white light laser as excitation light, and a phosphor that emits the white light as the first observation light when irradiated with the white light laser. The endoscope system according to claim 16, further comprising a laser light source for narrow band light that irradiates the narrow band light as the second observation light.

The light source device includes a white light source that emits white light, a white light filter that transmits the white light, a narrow band light filter that transmits a component of the narrow band light among the white light, and the white light source. The endoscope system according to claim 16, further comprising a first filter switching control unit that inserts the white light filter or the narrow band light filter into the light path of the white light that emits light.

The light source device irradiates the subject with first narrow band light corresponding to any one of red, blue, and green wavelength bands as the first observation light, and red, blue, and the second observation light. The endoscope system according to claim 15, wherein the subject is irradiated with a second narrow-band light corresponding to any of the wavelength bands of green and green and having a wavelength band different from that of the first narrow-band light.

The light source device includes a white light source that emits white light including light in the red, blue, and green wavelength bands, and a first narrow band light filter that transmits a component of the first narrow band light among the white light. A second narrow band light filter that transmits a component of the second narrow band light among the white light, and the first narrow band light filter or the second narrow band in the optical path of the white light emitted by the white light source. The endoscopic system according to claim 19, further comprising a second filter switching control unit into which an optical filter is inserted.

It is an image processing method of an image processing apparatus including an image acquisition unit for acquiring a first image captured by the first observation light and a second image captured by a second observation light different from the first observation light. hand,

The acquisition instruction reception process that accepts still image acquisition instructions,

An image acquisition control step for controlling the acquisition of the first image and the second image by the image acquisition unit, and

At least a display control step of displaying the first image on a display device, and

At least a classification process for classifying the subject reflected in the second image, and

With

In the image acquisition control step,

Until the acquisition instruction is received in the acquisition instruction receiving step, the image acquisition unit is made to perform a moving image acquisition process for sequentially acquiring the first image as a moving image.

In response to the acceptance of the acquisition instruction, the image acquisition unit is made to perform a still image acquisition process for acquiring the first image and the second image as still images.

An image processing method for causing the image acquisition unit to perform the moving image acquisition process after the still image acquisition process is completed.

Images