JP6042798B2 - Image processing apparatus and endoscope system - Google Patents

Description

  The present invention relates to a technique for separating and displaying an endoscope video signal acquired by an endoscope apparatus for each wavelength of light irradiated on an object.

  In an endoscope system, light emitted from a light source device is irradiated into a body cavity of a subject, which is a target, through a scope of the endoscope device, and reflection of light that has entered the inside of the stomach or intestines from the surface. Observe the light. For observation, light in a predetermined wavelength region may be used in addition to normal light including light from the ultraviolet region to the visible region and the infrared region.

  In normal light observation using normal light, the surface of the subject, such as the stomach and intestine, is recognized as a skin color or pink color. On the other hand, in special light observation using special light, for example, when the observation object is irradiated with light having a short wavelength on the blue to green side, the light having the short wavelength is applied to the surface or surface of the subject. By reaching only a very shallow part, the other parts become inconspicuous. Thereby, in the special light observation image, it becomes possible to emphasize the change of the surface or clarify the internal structure close to the surface.

  In this way, the lesions of the stomach and intestines are emphasized, and the internal structure peculiar to the lesions is clearly displayed in the image. ing.

  However, in the special light observation, the wavelength region to be used is narrower than that in the case of the normal light observation, so that the obtained image is dark. For this reason, in endoscopy, observation is performed using both normal light and special light. In general, as illustrated in FIG. 6, the filter is configured to switch between normal light and special light by limiting the wavelength region of light output from the light source device using a filter. A doctor who is a user of the endoscope apparatus performs the operation using a user operation unit such as a button provided on the operation unit at hand. When the light source is filtered, a special light observation image is obtained, and when the light source is not filtered, a normal light observation image is obtained.

  As described above, according to the special light observation, it becomes easy to confirm the lesioned part. On the other hand, in special light observation, since an image becomes dark, it is difficult to always perform inspection by special light observation from the start to the end of the inspection. For this reason, conventionally, a doctor usually performs special light observation at a place suspected of being a lesion and performs normal light observation at other places. However, from the actual medical field, in endoscopy, both normal light observation video that provides a bright and clear image and special light observation video that provides an image with the lesion highlighted are simultaneously observed. Possible technology is required.

  As a known technique for automatically switching between normal light observation and special light observation, for example, the light output from the light source device is switched to normal light or special light according to the generated timing signal, and the normal light video signal and special light observation are switched. It is disclosed that an optical video signal is obtained (for example, Patent Documents 1 and 2).

JP 2004-321244 A JP 2008-86759 A

  According to the above known technique, the light output from the light source device is switched according to the timing signal, and the normal light observation image and the special light observation image are output in parallel. However, according to such a method, it is necessary to perform timing control so that the timing of switching the light output from the light source device matches the timing of image capturing. When the timing is shifted between light switching and imaging, it is not possible to appropriately acquire two types of observation images in parallel. Further, in order to appropriately perform such timing control, extremely complicated processing is required. For this reason, in practice, it is extremely difficult to implement a technique for simultaneously acquiring and displaying two types of observation images in parallel by the above-described known technique.

  An object of the present invention is to provide a technique capable of separating according to wavelengths by a simple method even when a plurality of types of video signals having different wavelengths of light to be irradiated are simultaneously acquired. .

  According to the image processing device of one aspect of the present invention, each of the image signals obtained by alternately irradiating the target with the first and second lights having different wavelengths output from the light source device, An extraction unit that extracts a frame image from a video signal from one video stream in which the first and second video signals by the first and second lights are mixed, and the frame image is analyzed. A detection unit for determining whether the frame image is the first or second video signal by determining which of the features each of the first and second video signals has; And an output unit that separates and outputs the video stream of the first video signal and the video stream of the second video signal from the video stream based on the detection result.

  According to the present invention, even when a plurality of types of video signals having different wavelengths of light to be irradiated are simultaneously acquired, it is possible to separate them according to the wavelength by a simple method.

1 is an overall block diagram of an endoscope system including an image processing device according to an embodiment. It is a whole explanatory view about the separation method of the video stream of the observation picture by the endoscope system concerning an embodiment. It is a figure which illustrates the data structure of the observation image | video in each process step by an endoscope system. It is a figure explaining the 1st method of determining the kind of observation image. It is a figure explaining the 2nd method of determining the kind of observation image. It is a figure explaining the technique which displays the video stream of the observation video in the past.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is an overall block diagram of an endoscope system including an image processing apparatus according to the present embodiment. An endoscope system 100 illustrated in FIG. 1 includes a video processor 1, a scope 2, a light source device 3, and a monitor 4.

  The light source device 3 includes a normal light observation light source 31, an NBI (Narrow Band Imaging) light source 32, an NBI filter 33, a light source control unit 34, and a light source changeover switch unit 35. As shown in FIG. 1, in the present embodiment, the light source device 3 includes two light sources 31 and 32, and the light source control unit 34 controls the light source changeover switch unit 35 according to a command from the video processor 1. To do. The light source changeover switch unit 35 performs switching so that light is output from either the normal light observation light source 31 or the NBI light source 32.

  An NBI filter 33 is attached to the NBI light source 32 of the two light sources 31 and 32. Thus, the light output from the NBI light source 32 to which the NBI filter 33 is attached is limited to G (green) and B (blue) of RGB.

The light source device 3 transmits the output light to the scope 2 through a light guide cable not shown in FIG.
The scope 2 is inserted into the observation target 5, that is, the body cavity of the subject, and irradiates the observation target 5 with the light supplied from the light source device 3 through the irradiation port 21. The lens 22 forms an image of the reflected light of the light irradiated on the observation object 5, and a CCD (Charge Coupled Device) 23 converts this into an electric signal and outputs it to the video processor 1. The scope control unit 24 controls the drive timing of the CCD 23 and the like.

  The video processor 1 is configured by integrating a processor 10 and FPGAs (field-programmable gate arrays) 6 and 8, processes a video signal acquired by the scope 2, and outputs the processed video signal to the monitor 4. In the present embodiment, the video processor 1, when set to that effect via the user operation unit 7 including buttons or the like, is a video stream in which the normal light observation video stream and the NBI video stream are mixed. The input can be received and separated, and two types of observation images can be simultaneously output to the monitor 4 in parallel.

  The processor 10 includes a shared memory 11, a front-end processing unit 12, an image processing unit (an ISP (Image Signal Processing) processing unit in FIG. 1) 13, a memory 14, a system control unit 15, a video composition processing unit 16, and a back-end process. The video signal input from the FPGA 6 is subjected to necessary processing, and is output to the monitor 4 through the FPGA 8.

  Among these, the shared memory 11 temporarily holds the RAW data (video stream) input from the FPGA 6 and passes the RAW data to the front end processing unit 12. As will be described later in detail, when the user operation unit 7 switches the output of the light source device 3 and the photographing mode for alternately photographing both the normal light observation video and the NBI video is set, the signal is input to the processor 10. RAW data is a mixture of two types of observation video signals. Hereinafter, a shooting mode for acquiring a video stream in which a plurality of types (two types in the embodiment) of observation video signals are mixed is referred to as “parallel shooting mode”.

  The front end processing unit 12 performs pre-processing when performing image processing in the image processing unit 13. For example, processing such as reduction or enlargement of the image size is executed for display on the monitor 4.

  The image processing unit 13 performs necessary image processing on the video signal that has been subjected to necessary pre-processing by the front-end processing unit 12. The details of the image processing executed in the image processing unit are well-known techniques, and a detailed description thereof will be omitted. Further, when the shooting mode is set to the above-described shooting mode, the image processing unit 13 extracts a frame image from the input video stream. Then, the image processing unit 13 analyzes each frame image, and based on which of the features of the video signal of each observation video the frame image has, which of the normal light observation video stream or the NBI video stream It is determined whether it is included in. Then, when the shooting mode is set to the parallel shooting mode, the image processing unit 13 separates the normal light observation video stream and the NBI video stream and outputs the separated video stream to the video synthesis processing unit 16.

  The video composition processing unit 16 composes the video stream output from the image processing unit 13 and creates a video to be displayed on the monitor 4. Specifically, endoscope images and information to be displayed on the monitor 4 other than the endoscope images, for example, patient information, date, time, and other information related to various examinations are respectively displayed at predetermined positions on the screen. Create the placed video. The video composition processing unit 16 outputs the created video signal for display on the monitor 4 to the back-end processing unit 17.

  In the embodiment, the image processing unit 13 is configured to separate the two video streams for each type of observation video, but is not limited thereto. For example, in the image processing unit 13, information indicating which of the two types of observation images the frame image is included in is given to each frame image, and this information is referred to in the video composition processing unit 16. Thus, it may be configured to be separated into two video streams.

  The back-end processing unit 17 performs necessary processing such as interlace / progressive conversion on the video signal for display on the monitor 4 and outputs it to the FPGA 8 at the subsequent stage. In this way, the endoscopic image captured by the scope 2 is output to the monitor 4 through the FPGA 8.

The monitor 4 performs necessary processing on the video signal input from the video processor 1 and displays the video on the screen.
Thus, according to the endoscope system 100 according to the present embodiment, when the parallel photographing mode is set, the light source device 3 alternately outputs light having different wavelengths. In the parallel shooting mode, the image processing unit 13 of the video processor 1 analyzes a video stream in which two types of observation video input from the scope 2 are mixed, separates each type of observation video, and outputs the separated video to the monitor 4. . The monitor 4 can output and display two video streams received from the video processor 1 simultaneously.

  Although FIG. 1 illustrates the case where two images are displayed in parallel on one monitor 4, the present invention is not limited to this. For example, the video processor 1 may be connected to two monitors 4A and 4B, and a video stream of each observation video may be output to each of the monitors 4A and 4B. According to this configuration, the video processor 1 includes the image processing unit 13A for normal light observation video and the image processing unit 13B for NBI video. Each image processing unit 13 outputs the video stream of the observation video detected by the own device to each of the monitors 4A and 4B through the subsequent video composition processing units 16A and 16B and FPGAs 8A and 8B.

  FIG. 2 is an overall explanatory diagram of a method for separating a video stream of an observation video by the endoscope system 100 according to the present embodiment, and FIG. 3 is a data configuration of the observation video at each processing stage by the endoscope system 100. FIG. Referring to FIGS. 2 and 3, the video processor 1 separates a video stream in which two types of observation video input from the scope 2 are mixed according to the type of the observation video, and simultaneously displays them on the monitor 4 in parallel. This will be described in detail.

  As described above, the light source control unit 34 of the light source device 3 transmits a switching control signal to the changeover switch unit 35 in accordance with the shooting mode set via the user operation unit 7. When the user selects a mode for capturing one of the normal light observation image and the NBI image, the light source control unit 34 selects one of the light sources 31 and 32 that corresponds to one of the image capturing modes. A change control signal indicating this is generated and transmitted to the changeover switch unit 35. When the above-described parallel photographing mode is selected, the light source control unit 34 generates a switching control signal for switching the connection destination between the normal light observation light source 31 and the NBI light source 32 at a high speed. To the unit 35. The changeover switch unit 35 determines a connection destination light source according to the switching control signal received from the light source control unit 34, and when the parallel shooting mode is set, the changeover switch unit 35 speeds up between the two light sources 31 and 32 at high speed. Switch the connection destination.

  In the case where either the normal light observation mode or the NBI shooting mode is set, the light source corresponding to the shooting mode is set as the connection destination among the two light sources 31 and 32 by the changeover switch unit 35. Other than this, it is the same as the conventional one. For this reason, in the following, the description related to the prior art is omitted, and the operation of each device constituting the endoscope system 1 when the parallel imaging mode is set will be mainly described.

When the changeover switch unit 35 switches the connection destination between the normal light observation light source 31 and the NBI light source 32 at a high speed, the image of each frame acquired by the imaging unit of the scope 2 is the normal light observation image _IO or NBI image. it is one of the I _N. The video stream S supplied from the scope 2 to the video processor 1, so that the normal light observation image I _O and NBI image I _N are mixed. In FIG. 2, the lens 22 and the CCD 23 in FIG.

The video detection unit 19 in FIG. 2 is one of the configurations of the image processing unit 13 of the video processor 1. Image detection unit 19, among the frame image I, by analyzing the fields of the "video data", determines whether it is a it to either NBI image I _N a normal light observation image I _O. Based on the "image data" in the frame image I, for how a the image I to determine which one of the normal light observation image I _O, or NBI image I _N is later, to FIGS It will be described with reference to it.

When the image processing unit 19 in the image processing unit 13 performs a video detection process for determining which observation image is being detected, the image processing unit 13 converts each frame image I into a normal light observation video according to the processing result. It distributes the stream _{S O} and NBI video stream _{S N.} The video streams S _{O and} S _N separated into two in this way are simultaneously displayed on the monitors 4A and 4B, respectively. As described in the description of FIG. 1, two types of observation images may be displayed side by side on one monitor 4.

In the present embodiment, the timing at which the changeover switch unit 35 of the light source device 3 switches between the two light sources 31 and 32 and the timing at which the CCD 23 of the scope 2 performs imaging are not synchronized. For this reason, as illustrated in FIGS. 2 and 3, two types of observation images are not always acquired alternately. Therefore, the image detecting unit 19 of the image processing section 13 analyzes each frame image I, for each frame image I, will determine either a is whether the normal light observation image I _O, or NBI image I _N respectively There is a need. This will be described with reference to FIGS.

FIG. 4 is a diagram for explaining a first method for determining the type of observation image.
According to the first method, the type of the frame image is determined using the characteristics related to the color of the frame image. Specifically, the video detection unit 19 of the image processing unit 13 performs demosaic processing on the RAW data for each frame image of the video stream S input to the image processing unit 13, and generates color data (RGB format) for each pixel. obtain. Then, using the obtained color data (RGB), and determines whether the frame image is a one of the normal light observation image I _O, or NBI image I _N.

In general, as schematically shown in FIG. 4, if the pixel _{P N} of the NBI image _{I N,} with respect to R (red), while G (green) and B (blue) becomes strong, normal light In the case of the pixel P _O of the observation image _IO , R is slightly stronger than G or B. Using this feature, R, G, and B values are obtained from the RGB data of all the pixels of the frame image, and the ratio of R to G and B is compared with a predetermined threshold value.

When the ratio R of the G and B exceeds the threshold value, the frame image is determined to be intended to be within the normal light observation image stream S _O. When the ratio of R to G or B is equal to or less than the threshold value, it is determined that the frame image is included in the NBI video stream _SN .

For example, in the case of full HD, 1920 × 1080 pixels = 2,073,600 pixels are converted into RGB data. Then, for R, G, and B of all the pixels, for example, a total value or an average value is obtained, and the above comparison is performed to determine which of the two types of video streams S _O or S _N is a frame image. To do.

  As described above, by utilizing the characteristics of RGB data that are clearly different between the normal light observation video and the NBI video, it is possible to easily separate the video streams for each type of the observation video.

FIG. 5 is a diagram for explaining a second method for determining the type of observation image.
According to the second method, the type of the frame image is determined using the feature related to the luminance of the frame image. Specifically, the video detection unit 19 of the image processing unit 13 obtains RGB format color data obtained by demosaicing the RAW data for each frame image of the video stream S input to the image processing unit 13. Is further converted to YUV format. Then, using the obtained color data (YUV), and determines whether the frame image is a one of the normal light observation image I _O, or NBI image I _N.

In general, special light such as NBI irradiates only a part of the wavelength of light. Therefore, the special light observation image is an overall dark image compared to the normal light observation image. Using this feature, Y (luminance) data is extracted from the color data (YUV) of each pixel of the frame image, and Y is compared with a predetermined threshold value. If the value of Y exceeds the threshold value, the frame image is relatively bright image, i.e., determines that are included in the normal light observation image stream S _O. The value of Y is 16 ≦ Y ≦ 235 in ITU-R BT709. Set an appropriate threshold within this range.

If the value of Y is less than or equal to the threshold value, it is determined that the frame image is included in a relatively dark video, that is, the NBI video stream _SN . As described above, in the embodiment, the RGB format color data obtained by demosaicing the RAW data is converted into the YUV format by a known calculation formula. Using the obtained YUV format data, the above comparison is made based on the total value or average value of Y, and it is determined whether the image is a frame image included in the two types of video streams _SO or _SN , and separated. Output. Regarding the conversion to the YUV format, a configuration other than the video detection unit 19 may execute this, and the converted YUV data may be provided to the video detection unit 19.
As described above, by using the characteristic of Y (luminance) that is clearly different between the normal light observation video and the NBI video, the video stream can be easily separated for each type of the observation video.

  Thus, according to the endoscope system 100 according to the present embodiment, the light source device 3 alternately irradiates the subject that is the observation target 5 with normal light and special light (narrowband light) having different wavelengths. . The scope 2 acquires one video stream in which two types of video signals of normal light observation video and NBI video are mixed. Which of the characteristics of the normal light observation video signal and the NBI video signal is included in each frame image constituting the video stream input from the scope 2 by the image processing unit 13 (the video detection unit 19) of the video processor 1? Determine. Accordingly, the image processing unit 13 (the video detection unit 19 thereof) determines whether each frame image is a normal light observation video signal or an NBI video signal. In this way, for each determined type of observation video, it is separated into two video streams and output. Therefore, even when video signals of a plurality of types of endoscopes having different wavelengths of light are acquired by alternately switching wavelengths, a simple method without requiring complicated processing such as timing control, Depending on the wavelength of the irradiated light, that is, depending on the type of observation image, it can be separated and output in parallel.

  In FIG. 1 and the like, the light source device 3 includes two light sources 31 and 32. The light source 32 is provided with an NBI filter 33, and each light source outputs light having a different wavelength. However, the present invention is not limited to this. For example, the light source device 3 may be configured to include only one light source and perform control for inserting / removing an NBI filter on the optical path to obtain a video stream including a normal light observation video and an NBI video. The video processor 1 can perform separation according to the type of observation video by performing the same processing as described above on the input video stream.

  Further, in the above description, in the endoscopic examination or endoscopic operation, when displaying the video acquired by the scope 2 on the monitor 4 in real time, the video stream is separated and displayed according to the type of observation video. Although described, the present invention is not limited to this. For example, when the light source device 3 is switched between the two light sources 31 and 32 having different wavelengths, the video streams for the two observation videos are acquired at the same time, stored in the storage unit, etc., and then reproduced. In addition, the playback apparatus can be applied to a method of displaying a video stream separated according to the type of observation video by the above method.

  Further, the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, all the constituent elements shown in the embodiments may be appropriately combined. Furthermore, constituent elements over different embodiments may be appropriately combined. It goes without saying that various modifications and applications are possible without departing from the spirit of the invention.

1 Video Processor 2 Scope 3 Light Source Device 4 Monitor 5 Observation Object (Subject)
10 processor 13 image processing unit (ISP processing unit)
DESCRIPTION OF SYMBOLS 19 Image | video detection part 34 Light source control part 35 Light source changeover switch part 100 Endoscope system

Claims (4)

A video signal obtained by alternately irradiating an object with first and second lights having different wavelengths output from a light source device, wherein the first and second lights are respectively obtained by the first and second lights. An extraction unit for extracting a frame image from the video signal from one video stream in which the video signals are mixed;
By analyzing the frame image and determining which of the features each of the first and second video signals has, the frame image is either the first or second video signal. A detection unit for determining whether there is,
An output unit that separates and outputs the video stream of the first video signal and the video stream of the second video signal from the video stream based on the detection result of the detection unit;
An image processing apparatus comprising: The first and second lights are light in a normal light region and a special light region, respectively.
The detection unit determines whether the frame image is a video signal based on normal light or a special signal based on whether the ratio of the R value to the G and B values in the color information of the frame image exceeds a predetermined threshold. The image processing apparatus according to claim 1, wherein the image processing apparatus determines which of the video signals by light corresponds. The first and second lights are light in a normal light region and a special light region, respectively.
The detection unit identifies a video signal based on normal light or a video signal based on special light based on whether or not the value of luminance Y exceeds a predetermined threshold among the color information of the frame image. The image processing apparatus according to claim 1. First and second light source devices for outputting the first and second lights, respectively;
A display unit for displaying video streams of the first and second video signals separated by the output unit;
An endoscope system comprising the image processing device according to any one of claims 1 to 3.

Images