Classifications

• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/06—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements
  • A61B1/063—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements for monochromatic or narrow-band illumination
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/00002—Operational features of endoscopes
  • A61B1/00043—Operational features of endoscopes provided with output arrangements
  • A61B1/00045—Display arrangement
  • A61B1/0005—Display arrangement combining images e.g. side-by-side, superimposed or tiled
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/00163—Optical arrangements
  • A61B1/00186—Optical arrangements with imaging filters
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
  • A61B1/042—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances characterised by a proximal camera, e.g. a CCD camera
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
  • A61B1/043—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances for fluorescence imaging
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/06—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements
  • A61B1/0638—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements providing two or more wavelengths
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/06—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements
  • A61B1/0655—Control therefor
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/06—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements
  • A61B1/07—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements using light-conductive means, e.g. optical fibres

Definitions

• This disclosure relates to a medical observation system and a medical observation method.
• fluorescence observation makes it possible to grasp tissue conditions through fluorescence, which would be difficult to ascertain when observing the reflected light of visible light such as white light irradiated onto the biological tissue being observed. Therefore, fluorescence observation can be used for a variety of purposes and applications, such as identifying diseased areas.
• drugs can be used depending on the characteristics of the biological tissue being observed.
• drugs that emit fluorescence in the visible light wavelength range or in wavelength ranges other than the visible light range can be injected into the biological tissue.
• This disclosure provides a technology that is advantageous for observing an object through multiple types of observation light with different wavelength bands.
• One aspect of the present disclosure relates to a medical observation system that includes a light source device that emits broadband light in a first wavelength band, a first narrowband light that excites a first substance that emits a first fluorescence in a wavelength band included in the first wavelength band, and a second narrowband light that excites a second substance that emits a second fluorescence in a wavelength band not included in the first wavelength band, and a control unit that controls the light source device, where the control unit controls the light source device in a first mode so that the broadband light and the first narrowband light are irradiated to an observation object in a time-division manner, and controls the light source device in a second mode different from the first mode so that the broadband light and the second narrowband light are irradiated to the observation object.
• Another aspect of the present disclosure relates to a medical observation method including a step of emitting from a light source device at least one of broadband light in a first wavelength band, first narrowband light that excites a first substance that emits a first fluorescence in a wavelength band included in the first wavelength band, and second narrowband light that excites a second substance that emits a second fluorescence in a wavelength band not included in the first wavelength band, and emitting the broadband light and the first narrowband light from the light source device in a first mode such that the broadband light and the first narrowband light are irradiated to an observation object in a time-division manner, and emitting the broadband light and the second narrowband light from the light source device in a second mode different from the first mode such that the broadband light and the second narrowband light are irradiated to the observation object.
• FIG. 1A illustrates an example of a medical observation system.
• FIG. 1B is a diagram illustrating another example of a medical observation system.
• FIG. 2 is a diagram showing a schematic configuration of an example of a light source device.
• FIG. 3 is a diagram showing a schematic configuration of an example of an imaging system of a camera head.
• FIG. 4 is a diagram showing a schematic configuration of another example of the imaging system of the camera head.
• FIG. 5 is a diagram showing a schematic configuration of another example of the imaging system of the camera head.
• FIG. 6 is a diagram showing a schematic configuration of another example of the imaging system of the camera head.
• FIG. 7 is a block diagram showing an example of the configuration of the camera head and the control device.
• FIG. 1A illustrates an example of a medical observation system.
• FIG. 1B is a diagram illustrating another example of a medical observation system.
• FIG. 2 is a diagram showing a schematic configuration of an example of a light source device.
• FIG. 8 shows an example of an output image displayed on a display device.
• FIG. 9 shows another example of an output image displayed on the display device.
• FIG. 10 shows another example of an output image displayed on the display device.
• FIG. 11 shows another example of an output image displayed on the display device.
• FIG. 12 is a diagram illustrating the types of light incident on the imaging elements (first imaging element and second imaging element) according to the first embodiment.
• FIG. 13 shows an example of a timing chart of light source emission and image sensor exposure in the first mode of the first embodiment.
• FIG. 14 shows an example of a timing chart of light source emission and image sensor exposure in the second mode of the first embodiment.
• FIG. 15 shows another example of a timing chart of light source emission and image sensor exposure in the second mode of the first embodiment.
• FIG. 13 shows an example of a timing chart of light source emission and image sensor exposure in the first mode of the first embodiment.
• FIG. 16 is a diagram illustrating the types of light incident on the imaging elements (first imaging element and second imaging element) according to the second embodiment.
• FIG. 17 is a diagram illustrating the types of light incident on the imaging elements (first imaging element and second imaging element) according to the third embodiment.
• FIG. 18 shows an example of a timing chart of light source emission and image sensor exposure in the first mode of the third embodiment.
• FIG. 19 is a diagram illustrating the types of light incident on the imaging elements (first imaging element and second imaging element) according to the fourth embodiment.
• FIG. 20 is a diagram illustrating the types of light incident on the imaging elements (first imaging element and second imaging element) according to the fifth embodiment.
• FIG. 21 is a diagram illustrating the types of light incident on the imaging elements (the first imaging element, the second imaging element, and the third imaging element) according to the sixth embodiment.
• FIG. 22 shows an example of a timing chart of light source emission and image sensor exposure in the third mode of the seventh embodiment.
• FIG. 23 is a diagram illustrating the types of light incident on the imaging elements (first imaging element, second imaging element, and third imaging element) according to the eighth embodiment.
• FIG. 24 is a diagram illustrating the types of light incident on the imaging elements (first imaging element, second imaging element, and third imaging element) according to the ninth embodiment.
• FIG. 25 shows an example of a timing chart of light source emission and image sensor exposure in the fifth mode of the ninth embodiment.
• FIG. 26 is a diagram illustrating the types of light incident on the imaging elements (first imaging element, second imaging element, and third imaging element) according to the tenth embodiment.
• FIG. 27 is a flowchart showing an example of a medical observation method performed by the medical observation system.
• FIG. 28 shows an example of the relationship in physical size between the imaging region (effective pixel region) of the first imaging element and the imaging region (effective pixel region) of the second imaging element.
• FIG. 29 is a schematic diagram showing an example of a first image generated based on an image signal from the first imaging element shown in FIG.
• FIG. 30 is a diagram for explaining an example of a generation process of a first image, a second image, and a superimposed image in the first image generation example.
• FIG. 31 shows an example of a first output image generated from a first image (image captured under normal light) which is an image captured mainly by the first imaging element.
• FIG. 32 shows an example of a second output image generated mainly from a second image (fluorescence image) captured by the second imaging element.
• FIG. 33 is a diagram for explaining an example of a generation process of an output image in the first image generation example.
• FIG. 34A is a flowchart showing an example of a process for generating a superimposed output image.
• FIG. 34B is a flowchart showing an example of a process for generating a superimposed output image.
• FIG. 34C is a flowchart showing an example of a process for generating a superimposed output image.
• FIG. 34A is a flowchart showing an example of a process for generating a superimposed output image.
• FIG. 34B is a flowchart showing an example of a process for generating a superimposed output image.
• FIG. 34C is
• FIG. 35 is a diagram for explaining an example of a generation process of a first image, a second image, and a superimposed image in the second image generation example.
• FIG. 36 is a diagram for explaining an example of the generation process of an output image in the second image generation example.
• FIG. 37 is a conceptual diagram showing an example of a medical observation system configured as an operating field illumination observation device in which a ring light (an open field illumination device for observing a living body) is connected to a light source device.
• FIG. 38 is a diagram showing an example of a medical observation system configured as a microscope system.
• FIG. 1A is a diagram illustrating an example of a medical observation system 100 .
• the medical observation system 100 shown in FIG. 1A is configured as an endoscope device (endoscope system) for observing light from a subject (i.e., observation light) via a captured image.
• an endoscope device endoscope system
• the medical observation system 100 of this example is also capable of directly viewing the observation light with the naked eye without the use of a captured image.
• the medical observation system 100 shown in FIG. 1A includes a light source device 10, an insertion device 20 (endoscope body), a light guide 30, a camera head (imaging unit) 50, a display device 70, a transmission cable 80, and a control device 90.
• the light source device 10 is a device that emits light to be irradiated onto the subject being observed, and emits light under the control of the control device 90 (particularly the "control unit” described below (see FIG. 7)).
• the light source device 10 can emit multiple types of light with different wavelength bands, and can emit, for example, one or both of visible light (such as white light) and invisible light (such as infrared light or ultraviolet light).
• the light source device 10 shown in FIG. 1A has a broadband light source (first light source) 11 that emits broadband light, a first narrowband light source (second light source) 12 that emits first narrowband light, and a second narrowband light source (third light source) 13 that emits second narrowband light.
• first light source first light source
• second light source first narrowband light source
• third light source second narrowband light source
• the broadband light emitted from the broadband light source 11 contains light in a relatively wide wavelength band (i.e., the first wavelength band) as the main light component compared to the narrowband light.
• the first wavelength band of the broadband light may be a single continuous wavelength band, or may contain multiple discrete wavelength bands.
• the first narrowband light is partially or entirely included in the first wavelength band, which is the wavelength band of the broadband light, and has a bandwidth narrower than the first wavelength band, and can act as light that excites the first substance that emits the first fluorescence.
• the first fluorescence also has a bandwidth narrower than the first wavelength band.
• the wavelength band of the second narrowband light is not entirely included in the first wavelength band and has a narrower bandwidth than the first wavelength band, and can act as light that excites a second substance that emits a second fluorescence.
• the wavelength band of the second narrowband light may be a wavelength band on the longer wavelength side or a wavelength band on the shorter wavelength side than the first wavelength band of the broadband light and the wavelength band of the first narrowband light.
• the second fluorescence also has a narrower bandwidth than the first wavelength band and may be a wavelength band on the longer wavelength side or a wavelength band on the shorter wavelength side than the first wavelength band of the broadband light and the wavelength band of the first narrowband light.
• the light source device 10 is connected to the insertion device 20 via a light guide 30, and light emitted from the light source device 10 is transmitted to the insertion device 20 via the light guide 30.
• the light guide 30 is detachably connected to the light source device 10 and the insertion device 20.
• the insertion device 20 has an insertion section 21, and an optical connection section 22 and an imaging connection section 23 provided on the base end side of the insertion section 21.
• the insertion device 20 shown in FIG. 1A is configured as a rigid endoscope, and the insertion section 21 has a rigid, elongated shape.
• the insertion device 20 can have any structure, and may be configured as a flexible endoscope having a flexible insertion section 21, for example.
• a light transmission section (light guide) and an objective lens are provided on the end face of the tip 21a of the insertion section 21, which is located opposite the base end side.
• Light transmitted from the light source device 10 to the insertion device 20 via the light guide 30 is emitted from the light transmission section on the end face of the tip 21a of the insertion section 21 towards the observation object.
• Light from the observation object then enters the objective lens and is guided through the inside of the insertion section 21 to the imaging connection section 23.
• the general light from the observation object that is guided to the imaging connection section 23 via the objective lens and the inside of the insertion section 21 in this way is called observation light, and for example, reflected light from the observation object and fluorescence emitted from the observation object can be included in the observation light.
• the imaging connection section 23 is detachably connected to the connection section of the camera head 50.
• the observation light transmitted through the objective lens passes through the imaging connection section 23 and enters the camera head 50, where it is received.
• the imaging connection part 23 can also function as an eyepiece. In other words, when the imaging connection part 23 is detached from the camera head 50, the operator of the insertion device 20 can directly view the observation light through the imaging connection part 23.
• the camera head 50 is an imaging device that is detachably connected to the insertion device 20 and receives the observation light transmitted through the insertion device 20, and captures an image of an observation target illuminated with light emitted by the light source device 10 to obtain an image.
• the camera head 50 is connected to the control device 90 via a transmission cable 80.
• the transmission cable 80 is capable of transmitting various signals (e.g., image signals, control signals, synchronization signals, and clock signals) and power between the camera head 50 and the control device 90.
• the camera head 50 outputs an image signal corresponding to the received observation light, and the image signal is transmitted from the camera head 50 to the control device 90 via the transmission cable 80.
• the signal transmission method in the transmission cable 80 is not limited, and various signals can be transmitted as electrical signals or optical signals via the transmission cable 80. Also, instead of a wired signal transmission method via the transmission cable 80, various signals may be transmitted between the camera head 50 and the control device 90 by a wireless signal transmission method (e.g., wireless LAN (Local Area Network), Bluetooth (registered trademark), infrared communication, etc.).
• a wireless signal transmission method e.g., wireless LAN (Local Area Network), Bluetooth (registered trademark), infrared communication, etc.
• the control device 90 is connected to the camera head 50, the light source device 10, and the display device 70 by wire or wirelessly, and controls the camera head 50, the light source device 10, and the display device 70 in an integrated manner.
• the control device 90 controls the light emission of the light source device 10 as described below, generates an image from an image signal sent from the camera head 50, and displays the image on the display device 70.
• the control device 90 can also control the insertion device 20 connected to the camera head 50 via the camera head 50.
• control device 90 is provided separately from the light source device 10, the camera head 50, and the display device 70, but it may be provided integrally with the light source device 10, the camera head 50, and/or the display device 70.
• the display device 70 has a display of any configuration (e.g., a liquid crystal display or an organic electro-luminescence (EL) display) and displays an image on the display under the control of the control device 90.
• a display of any configuration e.g., a liquid crystal display or an organic electro-luminescence (EL) display
• EL organic electro-luminescence
• the display device 70 is shown as a single block, but one or more display devices 70 can be connected to the control device 90 (image generating device).
• the same output image may be synchronously displayed on the multiple display devices 70, or different output images may be synchronously displayed on the multiple display devices 70.
• an output image showing an overhead view of the entire observation target may be displayed on one display device 70, while an output image showing an enlarged view of the entire observation target or a portion of the observation target may be displayed on another display device 70.
• FIG. 1B is a diagram showing another example of a medical observation system 100.
• the medical observation system 100 shown in FIG. 1B further includes a third narrowband light source (fourth light source) 14 that emits a third narrowband light.
• the other configuration of the medical observation system 100 shown in FIG. 1B is similar to that of the medical observation system 100 shown in FIG. 1A described above.
• the wavelength band of the third narrowband light is not entirely included in the first wavelength band and has a narrower bandwidth than the first wavelength band, and can act as light that excites a third substance that emits a third fluorescence in a wavelength band that is at least partially different from the wavelength band of the second fluorescence.
• the wavelength band of the third narrowband light may be a wavelength band on the longer wavelength side or a wavelength band on the shorter wavelength side than the first wavelength band of the broadband light and the wavelength band of the first narrowband light.
• the wavelength band of the second narrowband light and the wavelength band of the third narrowband light may partially overlap each other, or may not overlap each other at all.
• the third fluorescence also has a narrower bandwidth than the first wavelength band, and may be a wavelength band on the longer wavelength side or a wavelength band on the shorter wavelength side than the first wavelength band of the broadband light and the wavelength band of the first narrowband light.
• FIG. 2 is a diagram showing a schematic configuration of an example of a light source device 10.
• a lens optical system 40 and a mirror optical system 41 are provided for each of the broadband light source 11, the first narrowband light source 12, and the second narrowband light source 13.
• the lens optical system 40 includes a collimator lens that converts incident light into parallel light.
• the mirror optical system 41 reflects light incident through the lens optical system 40 and transmits light incident through other mirror optical systems 41.
• the broadband light L1 emitted from the broadband light source 11 is collimated by the corresponding lens optical system 40, reflected by the corresponding mirror optical system 41, passes through the mirror optical system 41 associated with the other light sources 12 and 13, and enters the light guide 30.
• the first narrowband light L2 emitted from the first narrowband light source 12 is collimated by the corresponding lens optical system 40, reflected by the corresponding mirror optical system 41, passes through the mirror optical system 41 associated with the second narrowband light source 13, and enters the light guide 30.
• the second narrowband light L3 emitted from the second narrowband light source 13 is collimated by the corresponding lens optical system 40, reflected by the corresponding mirror optical system 41, and enters the light guide 30.
• the broadband light L1, first narrowband light L2, and second narrowband light L3 emitted from each of the light sources 11, 12, and 13 are reflected by the corresponding mirror optical system 41 and then enter the light guide 30 through a common optical path. Therefore, when two or more light sources among the broadband light source 11, the first narrowband light source 12, and the second narrowband light source 13 emit light simultaneously, multiple types of light enter the light guide 30 and are then irradiated onto the object of observation.
• the light source device 10 shown in FIG. 2 is merely an example, and the light source device 10 can have any other configuration, for example, a third narrowband light source 14 (see FIG. 1B) may be provided.
• a third narrowband light source 14 see FIG. 1B
• the specific device configurations of the broadband light source 11, the first narrowband light source 12, the second narrowband light source 13, and the third narrowband light source 14 are not limited, and each of the light sources 11, 12, 13, and 14 may have a single light-emitting device or may have multiple light-emitting devices.
• the wavelength bands of the broadband light L1, the first narrowband light L2, the second narrowband light L3, and the third narrowband light source 14 emitted by the broadband light source 11, the first narrowband light source 12, the second narrowband light source 13, and the third narrowband light source 14, respectively, are not limited.
• the broadband light source 11 may emit visible light as the broadband light L1, or may emit light in a wavelength band corresponding to the color gamut that can be displayed by the display device 70.
• Visible light is light that can be perceived by healthy human eyes, and may have a wavelength band of 380 to 780 nm, as an example, with an upper wavelength limit of 760 to 830 nm and a lower wavelength limit of 360 to 400 nm.
• light other than visible light is referred to as invisible light
• wavelength bands other than the visible light wavelength band are referred to as invisible light wavelength bands.
• the broadband light source 11 may emit white light as the broadband light L1, and may be composed of an LED (Light-Emitting Diode), a xenon lamp, or any other device.
• the broadband light source 11 may be composed of a white LED, which is a monochromatic light-emitting device, or may be composed of multiple colored LEDs (e.g., a red LED, a green LED, and a blue LED) that emit light of different colors.
• the broadband light source 11 may also have a white LED and multiple colored LEDs that emit colored light other than white light.
• the light emitted from the broadband light source 11 may also include at least a portion of light in the visible light wavelength band, and may, for example, include light that corresponds to the standard color gamut of the display device 70, or may include light in a specific wavelength band (e.g., purple or green light).
• a specific wavelength band e.g., purple or green light
• the first narrowband light source 12 may emit, as the first narrowband light L2, light that excites a first substance that emits a first fluorescence in a wavelength band included in the wavelength band (first wavelength band) of the broadband light L1 from the broadband light source 11.
• the second narrowband light source 13 may emit, as the second narrowband light L3, light that excites a second substance that emits a second fluorescence in a wavelength band not included in the wavelength band (first wavelength band) of the broadband light L1 from the broadband light source 11.
• the third narrowband light source 14 may emit, as the third narrowband light, light that excites a third substance that emits a third fluorescence in a wavelength band not included in the wavelength band (first wavelength band) of the broadband light L1 from the broadband light source 11.
• Each of the first narrowband light source 12, the second narrowband light source 13, and the third narrowband light source 14 may have, for example, a laser light source, an LED light source, a xenon lamp, or any other light-emitting device.
• each of the first narrowband light source 12, the second narrowband light source 13, and the third narrowband light source 14 may be configured by combining such a light-emitting device (e.g., a xenon lamp) with a filter (bandpass filter) that transmits narrowband light of a desired wavelength band from the light emitted from the light-emitting device.
• the broadband light L1 from the broadband light source 11 may be visible light, or may be visible light in a portion of the visible light wavelength band. Furthermore, the first to third fluorescent lights emitted by the first to third substances may each emit light in the visible or invisible light wavelength band (e.g., infrared or ultraviolet light).
• the visible or invisible light wavelength band e.g., infrared or ultraviolet light.
• the substances excited by the light emitted from the light source device 10 may be drugs or fluorescent dyes applied to the object of observation, or may be fluorescent substances that constitute the object of observation itself.
• Such drugs that can be administered to the subject of observation include, for example, 5-ALA (PP-IX), ADS780WS, ADS830WS, aggregation-induced emission dots allophycocyanin (APC), boron-dipyrromethane (BODIPY), CLR 1502, Flavins, fluorescamine, fluorescein, fluoro-gold, green fluorescence protein, ICG (indocyanine green), IRDye 78, IR-PEG nanoparticles, isothiocyanate, rose Bengal, trypan blue, and SGM-101.
• 5-ALA PP-IX
• ADS780WS ADS780WS
• ADS830WS aggregation-induced emission dots allophycocyanin
• BODIPY boron-dipyrromethane
• CLR 1502 Flavins, fluorescamine, fluorescein, fluoro-gold, green fluorescence protein, ICG (indocyanine green), IRDye 78,
• Fluorescent dyes that can be added to the object of observation include, for example, coumarine, Cy3, DyLight547, GE3126, metal nanoclusters, oxacarbocyanine, rhodamine, riboflavin, fluorescein, AlexaFluor 488, AlexaFluor660, AlexaFluor680, AlexaFluor700, Cy5, Cy5.5, Dy677, Dy682, Dy752, DyLight647, HiLyte Fluor 647, HiLyte Fluor 680, IRDye 700DX, methylene blue, porphyrins, porphysomes, VivoTag-680, VivoTag-S680, AlexaFluor750, AlexaFluor790, carbocyanine, conjugated copolymers, CW800-CA, Cy7, Cy7.5, cyanine dyes, Dy780, HiLyte
• fluorescent substances derived from the observation subject that make up the observation subject itself include collagen, elastin, and NADH.
• FIG. 3 shows a schematic diagram of an example of the imaging system of the camera head 50.
• the camera head 50 shown in FIG. 3 has an excitation light cut filter FC, a branching optical system Bs, a first image sensor 522a, and a second image sensor 522b.
• the observation light Lf transmitted through the insertion device 20 is incident on the branching optical system Bs after a specific wavelength band is cut by the excitation light cut filter FC.
• the wavelength band of light cut by the excitation light cut filter FC includes the wavelength band of excitation light that may be irradiated to the observation object.
• the excitation light cut filter FC partially, substantially, or completely suppresses at least the light in the wavelength bands of the first narrowband light, the second narrowband light, and the third narrowband light.
• the excitation light cut filter FC can be configured using a known wavelength selection filter, etc., and prevents the reflected light of the excitation light irradiated to the observation object from being received by the imaging element (in this example, the first imaging element 522a and the second imaging element 522b).
• the excitation light cut filter FC may be composed of a single filter or multiple filters.
• the excitation light cut filter FC may be a single filter that partially, substantially, or completely suppresses light in the wavelength bands of these excitation lights, or may include multiple filters that partially, substantially, or completely suppress light in the wavelength bands of each excitation light.
• the excitation light cut filter FC is not limited to the position shown in FIG. 3, but can be installed at any position on the optical path of the observation light Lf (including the separated light beam) from the observation object to the imaging elements (the first imaging element 522a and the second imaging element 522b in this example). That is, the excitation light cut filter FC may be provided in the insertion device 20, or may be provided, for example, upstream or downstream in the traveling direction of the observation light Lf with respect to the objective lens on the end face of the tip 21a of the insertion section 21. The excitation light cut filter FC may also be provided separately from the medical observation system 100.
• the excitation light cut filter FC does not have to be provided. If the effect of reflected excitation light on the captured image is sufficiently small, the system configuration may be simplified by not providing the excitation light cut filter FC.
• the branching optical system Bs is an optical element 15 that separates the observation light Lf from the observation object into a first light beam Lf1 and a second light beam Lf2 (multiple light beams), and guides the first light beam Lf1 to the first imaging element 522a and the second light beam Lf2 to the second imaging element 522b.
• the branching optical system Bs shown in FIG. 3 reflects the first light beam Lf1 toward the first imaging element 522a, while transmitting the second light beam Lf2 and guiding it to the second imaging element 522b.
• the branching optical system Bs can be configured, for example, based on a combination of a dichroic mirror and a wavelength selection filter, but the specific configuration of the branching optical system Bs is not limited.
• the branching optical system Bs may have an optical device that separates (disperses) the incident light into multiple light beams based on wavelength, for example, separating the light in the visible wavelength band and the light in the invisible wavelength band in the incident light into separate light beams.
• the branching optical system Bs may also have an optical device that separates the incident light into multiple light beams not based on wavelength, for example, separating the incident light into multiple light beams each having the same wavelength characteristics.
• each beam after separation may contain light in a wavelength band different from the wavelength band of light intended to be contained as the main light component in each beam (i.e., an unintended wavelength band).
• the second beam Lf2 may contain visible light equivalent to about 3% to 20% of the amount of light in the visible light wavelength band in the observation light Lf.
• the optical element 15 may contain a filter that partially, substantially, or completely removes the leakage light from the light beam.
• the light contained in the separated light beam is weak fluorescence, the fluorescence that is the intended target for reception may not be properly received by the corresponding imaging element (e.g., second imaging element 522b) if the light beam contains leakage light.
• the leakage light is partially, substantially, or completely removed from the light beam by the filter, so that the fluorescence that is the intended target for reception can be more appropriately received by the corresponding imaging element.
• the branching optical system Bs splits the incident light into multiple light beams without being based on wavelength
• the multiple light beams after splitting may have approximately equal amounts of light, or may have unequal amounts of light.
• the first light beam Lf1 and the second light beam Lf2 split by the branching optical system Bs may each have an amount of light that is about 50% of the amount of light of the observation light Lf, or may have different amounts of light (for example, a difference in light amount of about ⁇ 10% between the two).
• the branching optical system Bs may be designed so that the difference in light amount between the multiple light beams after splitting is determined based on the difference in sensitivity between the imaging elements that receive the multiple light beams after splitting.
• the branching optical system Bs may have a wavelength selection filter that reflects light with wavelengths shorter than the ICG fluorescence wavelength of around 820 to 870 nm, while transmitting light in other wavelength bands.
• FIG. 4 shows the schematic configuration of another example of the imaging system of the camera head 50.
• the branching optical system Bs of the optical element 15 shown in FIG. 4 has a color separation prism PR based on a combination of multiple (three) prisms, and a wavelength selection filter FL.
• the wavelength selection filter FL is provided at the joint surface between a first prism located at the most upstream position of the color separation prism PR and a second prism adjacent to the first prism.
• a portion of the observation light Lf incident on the optical element 15 (first light beam Lf1) is reflected by the bonding surface between the first prism and the second prism of the color separation prism PR, and is then further reflected and directed to the first image sensor 522a. Meanwhile, at least a portion of the other light of the observation light Lf (second light beam Lf2) passes through the optical element 15 and is directed to the second image sensor 522b.
• AR coat Anti-Reflection Coating
• the color separation prism PR is not limited to the example shown in FIG. 4, and can have any configuration. For example, a two-plate prism formed by combining two prisms (a first prism and a second prism) may be used as the color separation prism PR.
• the camera head shown in Figures 3 and 4 above is a typical example of a two-chip type imaging module that captures images using two imaging elements, but the camera head 50 may capture images using three or more imaging elements.
• FIG. 5 shows the schematic configuration of another example of the imaging system of the camera head 50.
• the camera head 50 shown in FIG. 5 has a similar configuration to the camera head 50 shown in FIG. 3, but is provided with a first branching optical system Bs1 and a second branching optical system Bs2 as the optical element 15, and the observation light Lf is separated into a first light beam Lf1 to a third light beam Lf3.
• the observation light Lf that passes through the excitation light cut filter FC is incident on the first branching optical system Bs1.
• the first branching optical system Bs1 reflects the first light beam Lf1 toward the first image sensor 522a while transmitting other light in the observation light Lf.
• the observation light Lf that passes through the first branching optical system Bs1 is incident on the second branching optical system Bs2.
• the second branching optical system Bs2 reflects the second light beam Lf2 toward the second image sensor 522b while transmitting the third light beam Lf3, which is the other light in the observation light Lf, and directs it to the third image sensor 522c.
• the first branching optical system Bs1 and the second branching optical system Bs2 can be configured, for example, based on a combination of a dichroic mirror and a wavelength selection filter, but the specific configuration of the first branching optical system Bs1 and the second branching optical system Bs2 is not limited.
• FIG. 6 shows the schematic configuration of another example of the imaging system of the camera head 50.
• the optical element 15 shown in FIG. 6 has a similar configuration to the optical element 15 shown in FIG. 4, but has a color separation prism PR, a first wavelength selection filter FL1, and a second wavelength selection filter FL2 as the branching optical system Bs.
• the first wavelength selection filter FL1 is provided on the bonding surface between a first prism located at the most upstream position of the color separation prism PR and a second prism adjacent to the first prism.
• the second wavelength selection filter FL2 is provided on the bonding surface between the second prism and a third prism adjacent to the second prism.
• a portion of the observation light Lf incident on the optical element 15 is reflected by the bonding surface between the first prism and the second prism of the color separation prism PR, and is then further reflected and emitted from the optical element 15 toward the first image sensor 522a as the first light beam Lf1. Meanwhile, the other light of the observation light Lf passes through the bonding surface between the first prism and the second prism. Then, a portion of the observation light Lf is reflected by the bonding surface between the second prism and the third prism, and is then emitted from the optical element 15 toward the second image sensor 522b as the second light beam Lf2. Meanwhile, the other light of the observation light Lf passes through the bonding surface between the second prism and the third prism, and is then emitted from the optical element 15 toward the third image sensor 522c as the third light beam Lf.
• the observation light Lf can be separated into multiple light beams (including a first light beam Lf1 and a second light beam Lf2). Then, by appropriately selecting the optical characteristics, such as the transmission wavelength band, of the wavelength selection filter FL, each of the multiple light beams separated from the observation light Lf can contain light of a desired wavelength band.
• the two-plate type imaging module (see Figures 3 and 4) is particularly advantageous in terms of miniaturizing the structure of the camera head 50 and reducing the cost.
• the three-plate type imaging module (see Figures 5 and 6) allows three types of light to be received simultaneously by three imaging elements (first imaging element 522a to third imaging element 522c).
• FIG. 7 is a block diagram showing an example configuration of the camera head 50 and the control device 90.
• the camera head 50 has a lens unit 51, an imaging unit 52, and a communication unit 53.
• the lens unit 51 includes one or more lenses, and focuses the observation light transmitted through the insertion device 20 (see FIG. 1A) and guides it to the imaging unit 52.
• the imaging unit 52 receives the observation light transmitted through the lens unit 51 and outputs a corresponding image signal.
• the imaging unit 52 shown in FIG. 7 has a light entrance unit 521, an imaging element 522, and a signal processing unit 523.
• the image sensor 522 is a photoelectric conversion element that receives the observation light transmitted through the light receiving section 521 and creates an image signal under the control of the control device 90 (particularly the control section 94 described below).
• the image sensor 522 is a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.
• the exposure to the image sensor 522 can be controlled by any shutter method, and either a mechanical shutter or an electronic shutter may be used. In the embodiments described below, a rolling shutter method is used, but other shutter methods (e.g., a global shutter method) may also be used.
• the image sensor 52 may have a single image sensor 522. If the image sensor 52 has a single image sensor 522 and there is no need to separate the observation light into multiple light beams, an optical element (see reference numeral "15" in Figures 3 to 6) for separating the observation light into multiple light beams is not required.
• a single image sensor 522 capable of receiving light of multiple wavelength bands may also be used.
• an image sensor also called a direct image sensor may be used in which the vertical color separation characteristics of the image sensor (e.g., silicon) are utilized to receive multiple types of observation light of different wavelength bands by each of multiple light receiving layers arranged vertically in the image sensor.
• a single image sensor 522 having multiple light receiving layers capable of receiving visible light (e.g., red light, green light, and blue light) and one or more light receiving layers capable of receiving fluorescence in the invisible light wavelength band (e.g., infrared light or ultraviolet light) can receive both a light flux containing the visible light as the main light component and a light flux containing the fluorescence as the main light component.
• two or more image sensors 522 having different characteristics may be provided.
• two or more image sensors 522 having the same characteristics may be provided.
• the light entrance section 521 is an optical system device that directs observation light from the lens unit 51 to the light receiving surface of the image sensor 522.
• the light entrance section 521 includes the above-mentioned optical element 15 (see Figures 3 to 6) that separates the observation light into multiple light beams and directs the multiple light beams to the multiple image sensors 522.
• a lens that optically enlarges or reduces the image so as to align the angles of view may be provided for both or one of the two or more image sensors 522.
• Such an angle of view adjustment lens may be provided as the light entrance section 521, or may be provided as part of the image sensor 522.
• the signal processing unit 523 performs signal processing (e.g., AGC (Auto Gain Control) processing and AD (Analog-to-Digital) conversion processing) on the image signal generated by the image sensor 522 under the control of the control device 90 (control unit 94).
• signal processing e.g., AGC (Auto Gain Control) processing and AD (Analog-to-Digital) conversion processing
• the communication unit 53 communicates with the control device 90 (particularly the communication unit 91) via the transmission cable 80 under the control of the control device 90 (particularly the control unit 94).
• the image signal (digital signal) output from the imaging unit 52 is sent from the communication unit 53 to the control device 90 (particularly the communication unit 91) via the transmission cable 80.
• the communication standard between the communication unit 53 of the camera head 50 and the communication unit 91 of the control device 90 is not limited, and the communication unit 53 and the communication unit 91 may be configured as a high-speed serial interface.
• the control device 90 has a communication unit 91, a memory 92, an image generation unit 93, a control unit 94, an input unit 95, an output unit 96, and a storage unit 97.
• the communication unit 91 Under the control of the control unit 94, the communication unit 91 sends the image signal sent from the camera head 50 via the transmission cable 80 to the image generation unit 93.
• the image generating unit 93 performs various image processing under the control of the control unit 94 to generate an image based on the image signal from the imaging unit 52.
• the image generating unit 93 shown in FIG. 7 has a memory controller 930, an image processing unit 931, a superimposed image generating unit 934, and a display control unit 935.
• the memory controller 930 writes and reads data to the memory 92.
• the memory controller 930 writes an image signal sent from the camera head 50 via the communication unit 91 into the memory 92 as image data. Then, the memory controller 930 reads image data from the memory 92 as necessary, and provides the image data to the image processing unit 931, the superimposed image generating unit 934, and/or the display control unit 935.
• the memory 92 can be configured, for example, from a volatile memory or a non-volatile memory, and can be used as a device capable of temporarily storing various data. There are no limitations on the data that can be stored in the memory 92. Therefore, the memory controller 930 may write image data and other data sent from the image processing unit 931, the superimposed image generating unit 934, and/or the display control unit 935 to the memory 92 or read it from the memory 92.
• the image processing unit 931 performs image processing on the image signal sent from the camera head 50 to generate a corresponding image (captured image).
• the image processing unit 931 can perform any image processing.
• the image processing unit 931 may perform one or more of the following processing, for example: AGC processing, optical black subtraction processing, white balance adjustment processing, demosaic processing, gradation correction processing, image (video) signal level adjustment processing, color correction processing, gamma correction processing, and fluorescent image signal processing.
• the specific configuration of the image processing unit 931 is not limited, and the image processing unit 931 may have a single processing unit regardless of the number of image sensors 522, or may have multiple processing units corresponding to each of the multiple image sensors 522.
• multiple image sensors 522 are assigned to one processing unit of the image processing unit 931, multiple corresponding images may be generated by sequentially processing multiple image signals from the multiple image sensors 522 by the one processing unit.
• the image processing unit 931 may have a first processing unit that generates an image based on an image signal from one imaging element 522, and a second processing unit that generates an image based on an image signal from the other imaging element 522.
• the image processing unit 931 has multiple processing units that are exclusively assigned to each of the multiple imaging elements 522, it is possible to perform image generation processing based on image signals from the multiple imaging elements 522 simultaneously in parallel.
• the superimposed image generating unit 934 generates a superimposed image by synthesizing multiple images. There are no limitations on the types of multiple images that are the basis of the superimposed image generated by the superimposed image generating unit 934 or on the specific synthesis method.
• a superimposed image may be generated by combining a captured image based on reflected light from an observation target irradiated with white light (broadband light) (also referred to as a "normal light captured image”) and a captured image based on fluorescence from an observation target irradiated with excitation light (narrowband light) (also referred to as a "fluorescence captured image”).
• a superimposed image may be generated by combining multiple captured images based on fluorescence of different wavelength bands from a common observation target.
• the superimposed image generating unit 934 may perform a synthesis process on the entire area of each of the multiple images that are the basis of the superimposed image, or may perform a synthesis process on only a partial area.
• the superimposed image can be generated by performing a synthesis process on the entire area of the multiple images without performing relative size and position adjustment between the multiple images. Also, if the total number of pixels of the multiple images that are the basis of the superimposed image is the same but the imaging ranges (including the observation target) shown in the multiple images are not completely the same, the superimposed image can be generated by performing either relative position adjustment or size adjustment between the multiple images and performing a synthesis process on the images of the imaging range common to the multiple images (for example, a part of a common observation target).
• the superimposed image generating unit 934 may perform a synthesis process for generating the superimposed image after adjusting the angles of view and total numbers of pixels of the images of the two or more image sensors using any image processing technology such as digital zoom.
• the superimposed image can be generated by combining the first image and the second image so that they are superimposed on each other over the entire area without adjusting the size or position.
• the size of one or both of the first image and the second image is adjusted (enlarged or reduced), and the first image and the second image after the size adjustment are combined so that they are superimposed on each other to generate a superimposed image.
• a superimposed image of a first image e.g., an image photographed under normal light
• a second image e.g., a fluorescent image relating to the fluorescence emitted by a first fluorescent substance (e.g., a first fluorescent agent)
• a third image e.g., a fluorescent image relating to the fluorescence emitted by a second fluorescent substance (e.g., a second fluorescent agent)
• a third image e.g., a fluorescent image relating to the fluorescence emitted by a second fluorescent substance (e.g., a second fluorescent agent)
• the superimposed image can be generated by combining the first to third images so that they are superimposed on each other over the entire area without adjusting the size or position. Furthermore, if the total number of pixels differs among the first to third images, the size of one or more of the first to third images is adjusted (enlarged or reduced), and the first to third images after size adjustment are combined so that they are superimposed on each other to generate a superimposed image.
• the superimposed image generating unit 934 may automatically generate a superimposed image of multiple images related to each other under the control of the control unit 94, or may generate a superimposed image using multiple images specified by the user as original images.
• the user may specify multiple original images via the input unit 95, and the control unit 94 may control the superimposed image generating unit 934 to generate a superimposed image from the multiple original images specified via the input unit 95.
• the superimposed image generating unit 934 may change the color of the subject, including the observation target, in the superimposed image.
• the superimposed image generating unit 934 may change the fluorescent portion in the superimposed image to a color with high visibility.
• the "highly visible color” referred to here is a color that is easy for a user viewing the superimposed image to identify, and may be, for example, a color that is not inherently shown by the observation target or a color that is inherently not often shown (e.g., yellow-green), or a color that is not used in other parts of the superimposed image.
• the fluorescent image when generating a superimposed image from a fluorescent image based on luminance signal information without color information and an image taken under normal light, the fluorescent image may be composited with the image taken under normal light as an image without color information, or may be changed to a certain color (e.g., a color with high visibility) and composited with the image taken under normal light.
• the fluorescent image taken under normal light is a color image with color information
• the color information of the fluorescent image taken under normal light may be changed to luminance signal information and composited with the image taken under normal light.
• the image taken under normal light may be changed to a monochrome image based on luminance signal information without color information, while the fluorescent image taken under normal light may be changed to a certain color (e.g., a color with high visibility).
• the display control unit 935 generates an output image under the control of the control unit 94.
• the display control unit 935 can generate an output image based on the captured image generated by the image processing unit 931 and/or the superimposed image generated by the superimposed image generation unit 934.
• the output image generated by the display control unit 935 may include only one of the captured image and the superimposed image, or may include multiple images.
• the output image includes multiple images
• the arrangement of the multiple images in the output image is not limited.
• the output image can be in the form of PiP (picture-in-picture; see FIG. 11 below), in which one image is superimposed on another, or PbP (picture-by-picture), in which multiple images are arranged side by side. Therefore, the display control unit 935 may generate an output image in the PiP format, in which one or more related captured images are reduced and arranged in a partial area of the superimposed image.
• the output image generated by the display control unit 935 may be a still image or a video (image).
• the output image generated by the display control unit 935 is transmitted to the display device 70 and displayed on the display of the display device 70.
• the display control unit 935 When multiple display devices 70 are connected to the control device 90, the display control unit 935 generates an output image corresponding to the characteristics of each of the multiple display devices 70 and outputs it to each of the multiple display devices 70. For example, when two or more display devices 70 with different screen resolutions (i.e., the total number of pixels displayed on the display) are connected to the control device 90, the display control unit 935 may generate multiple types of output images with the number of pixels corresponding to the respective screen resolutions of the two or more display devices 70. As a result, an output image having an optimized size is displayed on each of the two or more display devices 70.
• the display control unit 935 may automatically generate an output image under the control of the control unit 94, or may generate an output image according to a user specification.
• the user may specify one or more images to be included in the output image via the input unit 95, and the control unit 94 may control the display control unit 935 to generate an output image from the one or more images specified via the input unit 95 and send it to the display device 70. This allows the user to decide the output image to be displayed on the display device 70 based on their own will, and to switch as necessary.
• the control unit 94 controls the light source device 10, the camera head 50, and the display device 70, and also controls each part of the control device 90 (the communication unit 91, the image generation unit 93, the input unit 95, the output unit 96, the memory unit 97, etc.).
• the control unit 94 controls the light emission in the light source device 10, the reading of image data from the image sensor 522, and the image processing in the image generation unit 93 are performed under the control of the control unit 94.
• control unit 94 may control the image sensor 522 to reduce the frame rate of the image signal output from the image sensor 522, thereby lengthening the exposure time of the image sensor 522.
• the control unit 94 may also control the image generation unit 93 (particularly the image processing unit 931) to add up pixel data (pixel values) of two or more image sensors in the process of generating a captured image.
• control unit 94 may control the image sensor 522 and the signal processing unit 523 to perform thinning readout of pixel data or area-specified readout.
• data (pixel values) of only a portion of the multiple pixels that the image sensor 522 has is used to generate an image.
• the input unit 95 acts as an instruction receiving unit that receives instructions from the user, and under the control of the control unit 94, receives instructions from the user and transmits the instructions to the control unit 94.
• the input unit 95 can take any form, and may be configured as a device (such as a touch panel or operation buttons) that is directly operated by the user.
• the input unit 95 may be configured as a connection unit to which a device operated by a user (e.g., a mouse, a keyboard, or a portable device) is connected in a wired or wireless manner.
• a device operated by a user e.g., a mouse, a keyboard, or a portable device
• instructions from the user input via the operation unit of the light source device 10, the operation unit of the camera head 50, the operation unit of the insertion device 20, and the operation unit of the display device 70 may be transmitted to the control unit 94 via the input unit 95 of the control device 90.
• the output unit 96 outputs various information under the control of the control unit 94.
• the output unit 96 can take any form, and may be provided as, for example, a speaker, a printer, a communication device, and/or an application.
• the memory unit 97 is used to write and read various data by the control unit 94, and stores, for example, programs executed by the control unit 94 and information necessary for the processing of the control unit 94.
• the identification target here is, for example, a diseased area, a blood vessel, a nerve, etc.
• the first identification target contains a first substance that emits fluorescence when excited by the first narrowband light, but does not contain a second substance that emits fluorescence when excited by the second narrowband light.
• the second identification target does not contain the first substance, but does contain the second substance.
• FIG. 8 shows an example of an output image 210 displayed on the display device 70.
• the output image 210 shown in FIG. 8 is a captured image based on the reflected light of broadband light (particularly white light) irradiated onto the object of observation.
• the first material in the first identification target and the second material in the second identification target are not excited or are only weakly excited when irradiated with white light, and therefore do not emit fluorescence or emit only weak fluorescence. Therefore, in the output image 210 shown in FIG. 8, the first identification target and the second identification target displayed as the first identification target image 213 and the second identification target image 214 are difficult to distinguish compared to the observation target (e.g., the outline) displayed as the clear observation target image 211.
• the observation target e.g., the outline
• FIG. 9 shows another example of an output image 210 displayed on the display device 70.
• the output image 210 shown in FIG. 9 is a superimposed image of a captured image based on the reflected light of broadband light (particularly white light) irradiated onto the observation object, and a captured image based on the fluorescence from the observation object irradiated with the first narrowband light (excitation light).
• the first substance in the first identification object is excited by irradiation with the first narrowband light and emits strong fluorescence.
• the second substance in the second identification object is not excited or is only weakly excited when irradiated with white light and the first narrowband light, and therefore does not emit fluorescence or emits only weak fluorescence. Therefore, in the output image 210 shown in FIG. 9, the second identification object displayed as the second identification object image 214 is difficult to identify compared to the observation object (e.g., the outline) and the first identification object displayed as the clear observation object image 211 and the first identification object image 213.
• the user can clearly see the observation target image 211 and the first identification target image 213 at the same time, but has difficulty or cannot clearly see the second identification target image 214.
• FIG. 10 shows another example of an output image 210 displayed on the display device 70.
• the output image 210 shown in FIG. 10 is a superimposed image of a captured image based on the reflected light of broadband light (particularly white light) irradiated onto the object of observation, a captured image based on the fluorescence from the object of observation irradiated with a first narrowband light (excitation light), and a captured image based on the fluorescence from the object of observation irradiated with a second narrowband light (excitation light).
• the first substance in the first identification object is excited by irradiation with the first narrowband light and emits strong fluorescence
• the second substance in the second identification object is excited by irradiation with the second narrowband light and emits strong fluorescence. Therefore, in the output image 210 shown in FIG. 10, the observation object, the first identification object, and the second identification object are displayed as a clear observation object image 211, a first identification object image 213, and a second identification object image 214, respectively. Therefore, the user can clearly view the observation object image 211, the first identification object image 213, and the second identification object image 214 at the same time in the output image 210 of FIG. 10.
• FIG. 11 shows another example of an output image displayed on the display device 70.
• the output image 210 shown in FIG. 11 is displayed in PiP format, and includes a main image 220 and one or more thumbnail display images (in this example, a first thumbnail display image 221 and a second thumbnail display image 222) that occupy a partial area of the main image 220.
• the main image 220 shown in FIG. 11 is a captured image (see FIG. 8) based on the reflected light of broadband light (particularly white light) irradiated onto the object of observation.
• the first reduced-size image 221 is a reduced-size image (see FIG. 9) of a superimposed image of a captured image based on the reflected light of broadband light (particularly white light) irradiated onto the object of observation and a captured image based on fluorescence from the object of observation irradiated with first narrowband light (excitation light).
• the second reduced-size image 222 is a reduced-size image of a superimposed image of a captured image based on the reflected light of broadband light (particularly white light) irradiated onto the object of observation and a captured image based on fluorescence from the object of observation irradiated with second narrowband light (excitation light).
• the image displayed as the output image 210 on the display device 70 may be changed based on instructions from the user input via the input unit 95 under the control of the control device 90 (particularly the control unit 94 (see Figure 7)).
• control unit 94 may control the display control unit 935 to change the images displayed as the main image 220 and the thumbnail images 221 and 222 of the output image 210 in PiP format based on instructions from the user input via the input unit 95.
• the control unit 94 may also control the display control unit 935 to swap the image displayed as the main image 220 with the images displayed as the thumbnail images 221 and 222, and to swap the displayed images between the thumbnail images 221 and 222, based on instructions from the user input via the input unit 95.
• the control device 90 may also control the display control unit 935 to switch the display format of the output image 210 between a single image display format (see Figures 8 to 10) and a multiple image display format (see Figure 11) based on instructions from the user input via the input unit 95.
• the user can determine the condition and characteristics of the observation target based on the output image 210 (see Figures 8 to 11) as described above, which can be displayed on the display device 70.
• the user can accurately and easily determine the condition and characteristics of the observation target by simultaneously or by switching between multiple types of images of the same observation target with different observation light characteristics.
• displaying a superimposed image obtained by superimposing a single or multiple fluorescent images on a broadband light image on the display device 70 in this way the user can grasp the entire affected area of the observation target based on the broadband light image in the superimposed image, while confirming the positions of blood vessels, lesions, etc. that the user wishes to identify during surgery based on the fluorescent image, thereby supporting smooth surgery.
• the following first to fifth embodiments are typical examples in which the imaging section 52 of the camera head 50 uses a two-plate type imaging module equipped with two imaging elements 522 (see Figures 3 and 4).
• the fifth to tenth embodiments are typical examples in which the imaging section 52 of the camera head 50 uses a three-plate type imaging module equipped with three imaging elements 522 (see Figures 5 and 6).
• the observation mode can be switched based on instructions from a user, such as the operator of the medical observation system 100.
• a user such as the operator of the medical observation system 100.
• the instruction is transmitted from the input unit 95 to the control unit 94.
• the control unit 94 controls each part of the light source device 10, the camera head 50, and the control device 90, thereby achieving the desired observation mode.
• the broadband light emitted from the broadband light source 11 is white light
• the first wavelength band which is the wavelength band of the broadband light
• the first fluorescence emitted from the first substance excited by the first narrowband light from the first narrowband light source 12 is visible light (particularly visible light whose wavelength band overlaps with that of the broadband light (white light)).
• the second fluorescence and the third fluorescence emitted from the second substance and the third substance excited by the second narrowband light and the third narrowband light from the second narrowband light source 13 and the third narrowband light source 14 are invisible light (e.g., infrared light) included in the invisible light wavelength band.
• the wavelength bands of the second fluorescence and the third fluorescence are included in the invisible light wavelength band and do not overlap with the wavelength bands of the broadband light (white light) and the first fluorescence.
• the infrared light referred to here includes near-infrared light, mid-infrared light, and far-infrared light, and is light (electromagnetic waves) in a wavelength band of approximately 700 nm to 1000 ⁇ m.
• the broadband light (white light) emitted from the broadband light source 11 is used as illumination light to brightly illuminate the object of observation.
• the first narrowband light to the third narrowband light are used as excitation light for the fluorescent substances (first to third substances).
• the object of observation is irradiated with the broadband light (white light)
• it reflects at least a portion of the broadband light as broadband reflected light.
• the excitation light first to third narrowband light
• wavelength bands of the broadband light and the first to third fluorescent light are not limited to the above examples, and the embodiments described below can also be applied as appropriate when the broadband light and the first to third fluorescent light are lights of other wavelength bands.
• FIG. 12 is a diagram illustrating the types of light incident on the imaging elements (the first imaging element 522a and the second imaging element 522b) according to the first embodiment.
• the camera head 50 (particularly the imaging unit 52) of this embodiment is equipped with a two-plate type imaging module (see Figures 3 and 4), and is equipped with a first imaging element 522a having a color filter CF and a second imaging element 522b not having a color filter CF.
• the resolution of the first imaging element 522a may be higher than the resolution of the second imaging element 522b, and the sensitivity of the second imaging element 522b may be higher than the sensitivity of the first imaging element 522a.
• the resolution and sensitivity of the first imaging element 522a and the second imaging element 522b are not limited to this, and the relative relationship of the resolution and sensitivity between the first imaging element 522a and the second imaging element 522b is also not limited to this.
• the color filter CF can have any color filter and any filter arrangement as long as it can cause light of a desired wavelength band to be incident on the corresponding imaging element, and may include a primary color filter (RGB filter) or a complementary color filter (CMYG filter).
• the color filter CF provided on the first imaging element 522a in this embodiment transmits the broadband reflected light Lw1 and the first fluorescence Lw2 received by the first imaging element 522a as described below, but may or may not transmit light of a wavelength band different from the broadband reflected light Lw1 and the first fluorescence Lw2 (e.g., the second fluorescence Lw3).
• the color filter CF may be provided for all pixels of the first imaging element 522a, or the color filter CF may not be provided for some pixels of the first imaging element 522a.
• the second imaging element 522b in this example does not have a color filter CF, but may have a color filter CF that can transmit the second fluorescence Lw3 as described below.
• the second imaging element 522b may be provided with an R filter that can transmit not only red light in the visible light range but also infrared light (particularly the second fluorescence Lw3).
• the second imaging element 522b may or may not have a color filter CF, but if it does not have a color filter CF, a captured image can be obtained with higher light receiving sensitivity.
• the first imaging element 522a has an RGB filter as a color filter CF
• the second imaging element 522b may also have an RGB filter similar to that of the first imaging element 522a.
• the light source device 10 (see FIG. 1A) emits light from at least one of a broadband light source 11, a first narrowband light source 12, and a second narrowband light source 13, and can irradiate the object of observation S with at least one of the broadband light, the first narrowband light, and the second narrowband light.
• the observation light Lf from the observation target S may include broadband reflected light Lw1, which is reflected light of broadband light, a first fluorescence Lw2 emitted from a first substance excited by the first narrowband light, and a second fluorescence Lw3 emitted from a second substance excited by the second narrowband light.
• the broadband reflected light Lw1 and the first fluorescence Lw2 are light contained in a first wavelength band
• the second fluorescence Lw3 is light contained in a second wavelength band outside the first wavelength band.
• the observation light Lf incident on the optical element 15 is separated into a first light beam Lf1 and a second light beam Lf2 by the optical element 15.
• the optical element 15 guides the light included in the first wavelength band to the first image sensor 522a as the first light beam Lf1, and guides the light included in the second wavelength band to the second image sensor 522b as the second light beam Lf2.
• the first light beam Lf1 in which the light in the wavelength band of the second fluorescence Lw3 is partially, substantially, or completely suppressed, of the observation light Lf, is guided to the first image sensor 522a. That is, light including at least the broadband reflected light Lw1 and the first fluorescence Lw2 is guided to the first image sensor 522a as the first light beam Lf1.
• the second light beam Lf2 in which the light in the first wavelength band of the observation light Lf is partially, substantially, or completely suppressed by the optical element 15, is guided to the second image sensor 522b. That is, light including at least the second fluorescence Lw3 included in the second wavelength band is guided to the second image sensor 522b as the second light beam Lf2.
• the medical observation system 100 of this embodiment having the above-mentioned configuration can acquire various captured images of the observation subject S according to the following observation modes (first to third modes).
• the first mode of this embodiment is an observation mode in which broadband light and first narrowband light are irradiated onto the observation target S, and a captured image is obtained based on the broadband reflected light Lw1 and the first fluorescent light Lw2 from the observation target S.
• the control device 90 controls the light source device 10 (broadband light source 11 and first narrowband light source 12 (see FIG. 1A)), and broadband light and first narrowband light are emitted from the light source device 10 in a time-division manner, and the broadband light and first narrowband light are irradiated onto the observation object S in a time-division manner.
• the light source device 10 sequentially emits broadband light and first narrowband light
• the observation object S is sequentially irradiated with the broadband light and the first narrowband light.
• the emission duration per pulse in time-division emission is not limited.
• the emission duration may be the same or different between pulses emitted from a single light source, and the emission duration may be the same or different between pulses emitted from multiple light sources.
• the irradiation interval which is the time interval between one pulse and the next, is also not limited.
• the irradiation interval may be the same or different between pulses emitted from a single light source, and the irradiation interval may be the same or different between pulses emitted from multiple light sources.
• the optical element 15 sequentially guides the first light beam Lf1 including the broadband reflected light Lw1 from the observation object S irradiated with the broadband light, and the first light beam Lf1 including the first fluorescence Lw2 from the observation object S irradiated with the first narrowband light, to the first image sensor 522a.
• the first image sensor 522a sequentially receives the first light beam Lf1 including the broadband reflected light Lw1 and the first light beam Lf1 including the first fluorescence Lw2.
• the first image sensor 522a repeatedly outputs an image signal based on the broadband reflected light Lw1 and an image signal based on the first fluorescent light Lw2 in sequence.
• the image generation unit 93 (particularly the image processing unit 931 (see FIG. 7)) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 from the image signal based on the broadband reflected light Lw1 output from the first imaging element 522a.
• the image generation unit 93 (image processing unit 931) also generates a captured image (high-resolution color image) of the observation target S based on the first fluorescence Lw2 from the image signal based on the first fluorescence Lw2 output from the first imaging element 522a.
• the "high-resolution color image” referred to here is a color image acquired by the high-resolution imaging element, and it is not necessarily required that the output image sent from the high-resolution imaging element to a downstream stage have the same high resolution as the "color image acquired by the high-resolution imaging element.”
• FIG. 13 shows an example of a timing chart of light source emission and image sensor exposure in the first mode of the first embodiment.
• FIG. 13 shows the exposure state of the first image sensor 522a, with the vertical axis showing the horizontal lines of the first image sensor 522a and the horizontal axis showing time.
• the top row in FIG. 13(a) shows the topmost horizontal line (i.e. the first line), and the bottom row shows the bottommost horizontal line (i.e. the last line).
• the lines (diagonal lines) indicated with the symbol “R1" indicate the timing for starting to read out pixel data for each horizontal line for each image frame.
• the “broadband light image frame” between the lines R1 is an image frame for receiving (exposing) the broadband reflected light Lw1 from the observation target S.
• the “first fluorescence image frame” between the lines R1 is an image frame for receiving (exposing) the first fluorescence Lw2 from the observation target S.
• FIG. 13 shows the emission timing of broadband light from the broadband light source 11, and (c) shows the emission timing of the first narrowband light from the first narrowband light source 12.
• the light emitted from the light source device 10 reaches the observation object S instantaneously after emission. Therefore, the timing at which the broadband light from the broadband light source 11 is irradiated onto the observation object S, and the timing at which the broadband reflected light Lw1 from the observation object S is received by the first image sensor 522a, are approximately the same as the timing at which the broadband light is emitted from the broadband light source 11.
• the light source device 10 emits broadband light and first narrowband light and irradiates the observation target S in a time-division manner.
• the first image sensor 522a sequentially exposes (receives) the broadband reflected light Lw1 and the first fluorescent light Lw2 from the observation target S and reads out image data (pixel values) from the first image sensor 522a. Therefore, the control device 90 (control unit 94) controls the timing of the time-division emission by the light source device 10 and the timing of the readout of image data from the first image sensor 522a so that they are mutually linked.
• the broadband light source 11 and the first narrowband light source 12 emit broadband light and the first narrowband light in a time-division manner, and exposure and image data reading are performed at the first image sensor 522a. More specifically, the broadband light and the first narrowband light are emitted sequentially so as not to overlap and so that the first image sensor 522a is not exposed to both the broadband reflected light Lw1 and the first fluorescent light Lw2 at the same time.
• image data is read out from the first image sensor 522a so that a broadband light image frame exposed by the broadband reflected light Lw1 and a first fluorescent light image frame exposed by the first fluorescent light Lw2 are alternately output from the first image sensor 522a.
• exposure at the first image sensor 522a and readout of image data are performed based on the global shutter method. That is, in capturing an image frame, exposure is started for each row of the multiple pixels of the first image sensor 522a, starting from the first row to the last row. Then, after the exposure time has elapsed, pixel data is output for each row of the multiple pixels, starting from the first row to the last row.
• the exposure time for each image frame which means "the period from the start to the end of charge accumulation in each pixel,” is not limited, but is generally often set to 1/60 or 1/50 seconds.
• the timings of starting and ending the emission of the broadband light and the first narrowband light are not limited to the example shown in FIG. 13, and can be set to any timing.
• the emission of one or both of the broadband light and the first narrowband light may be started or ended.
• the intensity of the fluorescence that is the light receiving object in the first fluorescence image frame is weak (i.e., when the amount of fluorescence emission is small)
• the first narrowband light may be emitted from the first narrowband light source 12 while the image data of the broadband light image frame is being read out.
• the exposure time of the fluorescence in the first fluorescence image frame can be extended, which is advantageous for obtaining bright image data of the first fluorescence image frame.
• the first narrowband light source 12 may constantly emit the first narrowband light.
• Continuous illumination here means continuous illumination.
• the first image sensor 522a alternately outputs the image signal of the broadband light image frame (i.e., the image signal based on the broadband reflected light Lw1) and the image signal of the first fluorescence image frame (i.e., the image signal based on the first fluorescence Lw2). Then, based on the image signal of the broadband light image frame, a normal light captured image of the observation object S, which is a reflected image of broadband light (white light), is generated. Also, based on the image signal of the first fluorescence image frame, a first fluorescence captured image, which is an image of the observation object S based on the first fluorescence Lw2, is generated.
• a normal light image which is a reflected image of visible light (white light)
• a first fluorescent image in which the first substance in the observation target S is emphasized, i.e., an image in which fluorescence is emitted from the first substance in the observation target S excited by narrowband light
• the user can compare the normal light image and the first fluorescent image of the observation target S via the display device 70 (see FIG. 1A), and can also observe a composite image (superimposed image) created from these images.
• the second mode of this embodiment is an observation mode in which the object S is irradiated with broadband light and second narrowband light, and an image is captured based on the broadband reflected light Lw1 and second fluorescent light Lw3 from the object S.
• the control device 90 controls the light source device 10 (broadband light source 11 and second narrowband light source 13), and the broadband light and second narrowband light are continuously emitted from the light source device 10, and the broadband light and second narrowband light are continuously irradiated onto the object of observation S.
• the optical element 15 continuously guides the first light beam Lf1, which includes the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, to the first image sensor 522a.
• the optical element 15 also continuously guides the second light beam Lf2, which includes the second fluorescence Lw3 from the observation target S irradiated with the second narrowband light, to the second image sensor 522b.
• the first imaging element 522a continuously receives the first light beam Lf1 including the broadband reflected light Lw1
• the second imaging element 522b continuously receives the second light beam Lf2 including the second fluorescent light Lw3.
• the first imaging element 522a continuously and repeatedly outputs an image signal based on the broadband reflected light Lw1
• the second imaging element 522b continuously and repeatedly outputs an image signal based on the second fluorescent light Lw3.
• the image generation unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 from the image signal based on the broadband reflected light Lw1 output from the first imaging element 522a.
• the image generation unit 93 (image processing unit 931) also generates a captured image (high-sensitivity monochrome image) of the observation target S based on the second fluorescence Lw3 from the image signal based on the second fluorescence Lw3 output from the second imaging element 522b.
• the "high-resolution color image” here refers to a color image acquired by the high-resolution imaging element
• the "high-sensitivity monochrome image” refers to a monochrome image acquired by the high-sensitivity imaging element.
• FIG. 14 shows an example of a timing chart of light source emission and image sensor exposure in the second mode of the first embodiment.
• FIG. 14(a) shows the exposure state of the first imaging element 522a
• (c) shows the exposure state of the second imaging element 522b
• the vertical axis shows the horizontal lines of the first imaging element 522a and the second imaging element 522b
• the horizontal axis shows time
• line R1 shows the timing at which pixel data readout of each horizontal line for each image frame begins.
• the "broadband light image frame" between lines R1 is an image frame for receiving (exposing) the broadband reflected light Lw1 from the observation target S
• the "second fluorescence image frame” between lines R1 is an image frame for receiving (exposing) the second fluorescence Lw3 from the observation target S.
• FIG. 14(b) shows the emission timing of the broadband light from the broadband light source 11, and (d) shows the emission timing of the second narrowband light from the second narrowband light source 13.
• the light source device 10 continuously emits broadband light and second narrowband light and irradiates the observation target S.
• the first image sensor 522a continuously exposes (receives) the broadband reflected light Lw1 from the observation target S.
• the second image sensor 522b continuously exposes (receives) the second fluorescence Lw3 from the observation target S.
• the reading of image data from the first imaging element 522a and the second imaging element 522b can be performed at any timing while the light source device 10 is emitting the broadband light and the second narrowband light.
• exposure and image data reading at the first imaging element 522a and the second imaging element 522b are performed based on the timing determined by a common synchronization signal.
• the pixel data read start timing R1 of the first imaging element 522a coincides with the pixel data read start timing R1 of the second imaging element 522b.
• the pixel data read start timing R1 of the first imaging element 522a does not have to coincide with the pixel data read start timing R1 of the second imaging element 522b.
• FIG. 15 shows another example of a timing chart of light source emission and image sensor exposure in the second mode of the first embodiment.
• the pixel data read start timing R1 of the first image sensor 522a and the pixel data read start timing R1 of the second image sensor 522b are shifted from each other by 1/2 the exposure time of each image frame and do not overlap in time.
• the exposure time of each image frame of the first image sensor 522a and the second image sensor 522b in the example shown in FIG. 15 is twice the exposure time of each image frame in the example shown in FIG. 14.
• the first image sensor 522a continuously and repeatedly outputs an image signal of a broadband light image frame exposed by the broadband reflected light Lw1 (i.e., an image signal based on the broadband reflected light Lw1).
• the second image sensor 522b continuously and repeatedly outputs an image signal of a second fluorescence image frame exposed by the second fluorescence Lw3 (i.e., an image signal based on the second fluorescence Lw3).
• a normal light photographed image of the observation target S which is a reflected image of broadband light (white light)
• a second fluorescence photographed image which is a photographed image of the observation target S based on the second fluorescence Lw3 is generated.
• a normal light image which is a reflected image of visible light (white light)
• a second fluorescent image in which the second substance in the observation target S is emphasized, i.e., an image in which fluorescence is emitted from the second substance in the observation target S excited by narrowband light
• the user can compare the normal light image and the second fluorescent image of the observation target S via the display device 70, and can also observe a composite image (superimposed image) created from these images.
• the broadband light and the second narrowband light are constantly emitted (i.e., continuously emitted) from the light source device 10, but the light source device 10 may turn off the emission of each of the broadband light and the second narrowband light midway under the control of the control device 90 (control unit 94).
• the light source device 10 may repeatedly turn on and off the emission of the broadband light and the second narrowband light, or may emit the broadband light and the second narrowband light in a time-division manner.
• the first image sensor 522a can continuously receive the broadband reflected light Lw1
• the second image sensor 522b can continuously receive the first fluorescent light.
• the amount of charge stored in the first image sensor 522a and the second image sensor 522b increases with long exposure times, making it possible to capture bright images and suppressing increases in noise caused by gain adjustment.
• all frames of the first image sensor 522a and the second image sensor 522b can be used to generate captured images. In this way, by preventing the occurrence of frames in which no actual imaging is performed, it is possible to avoid a decrease in the effective frame rate and provide smooth images.
• the third mode of this embodiment is an observation mode in which broadband light, first narrowband light, and second narrowband light are irradiated onto the object of observation S, and a captured image is obtained based on the broadband reflected light Lw1, first fluorescence Lw2, and second fluorescence Lw3 from the object of observation S.
• control device 90 controls the light source device 10 (broadband light source 11, first narrowband light source 12, and second narrowband light source 13), and the light source device 10 emits broadband light and first narrowband light in a time-division manner, and emits the second narrowband light continuously.
• the broadband light and first narrowband light are irradiated to the observation object S in a time-division manner, and the second narrowband light is irradiated to the observation object S continuously.
• the optical element 15 sequentially guides the first light beam Lf1 including the broadband reflected light Lw1 from the observation object S irradiated with the broadband light, and the first light beam Lf1 including the first fluorescence Lw2 from the observation object S irradiated with the first narrowband light, to the first image sensor 522a.
• the optical element 15 also continuously guides the second light beam Lf2 including the second fluorescence Lw3 from the observation object S irradiated with the second narrowband light, to the second image sensor 522b.
• the first image sensor 522a sequentially receives the first light beam Lf1 including the broadband reflected light Lw1 and the first light beam Lf1 including the first fluorescent light Lw2. Then, under the control of the control device 90 (control unit 94), the first image sensor 522a repeatedly outputs an image signal based on the broadband reflected light Lw1 and an image signal based on the first fluorescent light Lw2.
• the second image sensor 522b continuously receives the second light beam Lf2 containing the second fluorescence Lw3, and under the control of the control device 90 (control unit 94), continuously and repeatedly outputs an image signal based on the second fluorescence Lw3.
• the image generation unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 from an image signal based on the broadband reflected light Lw1 output from the first imaging element 522a.
• the image generation unit 93 (image processing unit 931) also generates a captured image (high-resolution color image) of the observation target S based on the first fluorescence Lw2 from an image signal based on the first fluorescence Lw2 output from the first imaging element 522a.
• the image generation unit 93 (image processing unit 931) also generates a captured image (high-sensitivity monochrome image) of the observation target S based on the second fluorescence Lw3 from an image signal based on the second fluorescence Lw3 output from the second imaging element 522b.
• the "high-resolution color image” referred to here is a color image acquired by the high-resolution imaging element
• the "high-sensitivity monochrome image” is a monochrome image acquired by the high-sensitivity imaging element.
• a normal light image which is a reflected image of visible light (white light)
• a first fluorescent image in which a first substance in the observation target S is emphasized i.e., an image in which fluorescence is emitted from a first substance in the observation target S excited by narrow band light
• a second fluorescent image in which a second substance in the observation target S is emphasized i.e., an image in which fluorescence is emitted from a second substance in the observation target S excited by narrow band light
• the user can compare and observe the normal light image, first fluorescent image, and second fluorescent image of the observation target S via the display device 70, and can also observe a composite image (superimposed image) created from these images.
• FIG. 16 is a diagram illustrating the types of light incident on the imaging elements (first imaging element 522a and second imaging element 522b) according to the second embodiment.
• the camera head 50 (particularly the imaging unit 52) of this embodiment is equipped with a two-plate type imaging module (see Figures 3 and 4), and is equipped with a first imaging element 522a having a color filter CF and a second imaging element 522b not having a color filter CF.
• the color filter CF provided on the first image sensor 522a in this embodiment transmits the broadband reflected light Lw1 and the first fluorescence Lw2 received by the first image sensor 522a as described below, but may or may not transmit light of a wavelength band different from the broadband reflected light Lw1 and the first fluorescence Lw2 (e.g., the second fluorescence Lw3 and the third fluorescence Lw4).
• the second image sensor 522b does not have a color filter CF in this example, but may have a color filter CF that can transmit the second fluorescence Lw3 and the third fluorescence Lw4 as described below.
• the resolution of the first imaging element 522a may be higher than the resolution of the second imaging element 522b, and the sensitivity of the second imaging element 522b may be higher than the sensitivity of the first imaging element 522a.
• the resolution and sensitivity of the first imaging element 522a and the second imaging element 522b are not limited to this, and the relative relationship of the resolution and sensitivity between the first imaging element 522a and the second imaging element 522b is also not limited to this.
• the light source device 10 (see FIG. 1B) emits light from at least one of the broadband light source 11, the first narrowband light source 12, the second narrowband light source 13, and the third narrowband light source 14, and can irradiate the observation object S with at least one of the broadband light, the first narrowband light, the second narrowband light, and the third narrowband light.
• the observation light Lf from the observation target S may therefore include broadband reflected light Lw1, which is reflected light of broadband light, a first fluorescence Lw2 emitted from a first substance excited by the first narrowband light, a second fluorescence Lw3 emitted from a second substance excited by the second narrowband light, and a third fluorescence Lw4 emitted from a third substance excited by the third narrowband light.
• the broadband reflected light Lw1 and the first fluorescence Lw2 are light contained in a first wavelength band
• the second fluorescence Lw3 and the third fluorescence Lw4 are light contained in a second wavelength band outside the first wavelength band.
• the observation light Lf incident on the optical element 15 is separated into a first light beam Lf1 and a second light beam Lf2 by the optical element 15.
• the optical element 15 guides the light included in the first wavelength band to the first image sensor 522a as the first light beam Lf1, and guides the light included in the second wavelength band to the second image sensor 522b as the second light beam Lf2.
• the first light beam Lf1 in which the light in the wavelength bands of the second fluorescence Lw3 and the third fluorescence Lw4 of the observation light Lf is partially, substantially, or completely suppressed, is guided to the first image sensor 522a. That is, light including at least the broadband reflected light Lw1 and the first fluorescence Lw2 is guided to the first image sensor 522a as the first light beam Lf1.
• the second light beam Lf2 in which the light in the first wavelength band of the observation light Lf is partially, substantially, or completely suppressed by the optical element 15, is guided to the second image sensor 522b. That is, light including at least the second fluorescence Lw3 and the third fluorescence Lw4 included in the second wavelength band is guided to the second image sensor 522b as the second light beam Lf2.
• the medical observation system 100 of this embodiment having the above-mentioned configuration can acquire various captured images of the observation subject S according to the following observation modes (first mode to fifth mode).
• the light source device 10 In the first mode, the light source device 10 emits broadband light and the first narrowband light in a time-division manner, and the broadband reflected light Lw1 and the first fluorescent light Lw2 are received by the first image sensor 522a. In the second mode, the light source device 10 continuously emits broadband light and the second narrowband light, the broadband reflected light Lw1 is received by the first image sensor 522a, and the second fluorescent light Lw3 is received by the second image sensor 522b.
• the light source device 10 emits broadband light and the first narrowband light in a time-division manner and continuously emits the second narrowband light, the broadband reflected light Lw1 and the first fluorescent light Lw2 are received by the first image sensor 522a, and the second fluorescent light Lw3 is received by the second image sensor 522b.
• the fourth mode of this embodiment is an observation mode in which the object S is irradiated with broadband light and third narrowband light, and an image is captured based on the broadband reflected light Lw1 and the third fluorescent light Lw4 from the object S.
• control device 90 controls the light source device 10 (broadband light source 11 and third narrowband light source 14), and the broadband light and third narrowband light are continuously emitted from the light source device 10, and the broadband light and third narrowband light are continuously irradiated onto the observation object S.
• the first narrowband light source 12 and the second narrowband light source 13 are kept in an off state, and the first narrowband light and the second narrowband light are not emitted from the light source device 10.
• the optical element 15 continuously guides the first light beam Lf1, which includes the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, to the first image sensor 522a.
• the optical element 15 also continuously guides the second light beam Lf2, which includes the third fluorescence Lw4 from the observation target S irradiated with the third narrowband light, to the second image sensor 522b.
• the first image sensor 522a continuously receives the first light beam Lf1 including the broadband reflected light Lw1, and under the control of the control device 90 (controller 94), continuously and repeatedly outputs an image signal based on the broadband reflected light Lw1.
• the second image sensor 522b continuously receives the second light beam Lf2 including the third fluorescence Lw4, and under the control of the control device 90 (controller 94), continuously and repeatedly outputs an image signal based on the third fluorescence Lw4.
• the image generation unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 from the image signal based on the broadband reflected light Lw1 output from the first imaging element 522a.
• the image generation unit 93 (image processing unit 931) also generates a captured image (high-sensitivity monochrome image) of the observation target S based on the third fluorescence Lw4 from the image signal based on the third fluorescence Lw4 output from the second imaging element 522b.
• the "high-resolution color image” here refers to a color image acquired by the high-resolution imaging element
• the "high-sensitivity monochrome image” refers to a monochrome image acquired by the high-sensitivity imaging element.
• a normal light image which is a reflected image of visible light (white light)
• a third fluorescent image in which the third substance in the observation target S is highlighted, i.e., an image in which fluorescence is emitted from the third substance in the observation target S excited by narrowband light
• the user can compare the normal light image and the third fluorescent image of the observation target S via the display device 70, and can also observe a composite image (superimposed image) created from these images.
• the fifth mode of this embodiment is an observation mode in which broadband light and first to third narrowband light are irradiated onto the observation target S, and an image is acquired based on the broadband reflected light Lw1 and the first to third fluorescent light Lw2 to Lw4 from the observation target S.
• control device 90 controls the light source device 10 (broadband light source 11 and first to third narrowband light sources 12 to 14), and the light source device 10 emits broadband light and first narrowband light in a time-division manner, and also emits second narrowband light and third narrowband light in a time-division manner.
• the broadband light and first narrowband light are irradiated to the observation object S in a time-division manner
• the second narrowband light and third narrowband light are irradiated to the observation object S in a time-division manner.
• each of the broadband light and the first narrowband light may be emitted simultaneously with one of the second narrowband light and the third narrowband light, and irradiated to the observation object S at the same time.
• the optical element 15 then sequentially guides the first light beam Lf1 including the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, and the first light beam Lf1 including the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light, to the first image sensor 522a.
• the optical element 15 also sequentially guides the second light beam Lf2 including the second fluorescence Lw3 from the observation target S irradiated with the second narrowband light, and the second light beam Lf2 including the third fluorescence Lw4 from the observation target S irradiated with the third narrowband light, to the second image sensor 522b.
• the first image sensor 522a sequentially receives the first light beam Lf1 including the broadband reflected light Lw1 and the first light beam Lf1 including the first fluorescent light Lw2. Then, under the control of the control device 90 (control unit 94), the first image sensor 522a repeatedly outputs an image signal based on the broadband reflected light Lw1 and an image signal based on the first fluorescent light Lw2.
• the second image sensor 522b sequentially receives the second light beam Lf2 including the second fluorescence Lw3 and the second light beam Lf2 including the third fluorescence Lw4. Then, under the control of the control device 90 (control unit 94), the second image sensor 522b repeatedly outputs an image signal based on the second fluorescence Lw3 and an image signal based on the third fluorescence Lw4.
• the image generation unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 from the image signal based on the broadband reflected light Lw1 output from the first imaging element 522a.
• the image generation unit 93 (image processing unit 931) also generates a captured image (high-resolution color image) of the observation target S based on the first fluorescence Lw2 from the image signal based on the first fluorescence Lw2 output from the first imaging element 522a.
• the "high-resolution color image” referred to here is a color image acquired by the high-resolution imaging element.
• the image generation unit 93 also generates a captured image (high-sensitivity monochrome image) of the observation target S based on the second fluorescence Lw3 from an image signal based on the second fluorescence Lw3 output from the second imaging element 522b.
• the image generation unit 93 also generates a captured image (high-sensitivity monochrome image) of the observation target S based on the third fluorescence Lw4 from an image signal based on the third fluorescence Lw4 output from the second imaging element 522b.
• the "high-sensitivity monochrome image” referred to here is a monochrome image acquired by the high-sensitivity imaging element.
• a normal light image which is a reflected image of visible light (white light)
• first to third fluorescent images which are images in which the first to third substances in the observation target S are highlighted, i.e., images in which fluorescence is emitted from the first to third substances in the observation target S excited by narrowband light
• the user can compare and observe the normal light image, first fluorescent image, second fluorescent image, and third fluorescent image of the observation target S via the display device 70, and can also observe a composite image (superimposed image) created from these images.
• the broadband reflected light Lw1 and the first fluorescent light Lw2 of the observation light Lf are imaged by the first image sensor 522a, while the second fluorescent light Lw3 and the third fluorescent light Lw4 are imaged by the second image sensor 522b.
• a high-resolution color image is obtained as the captured image from the broadband reflected light Lw1 in the visible light wavelength band and the first fluorescent light Lw2, which contains color information.
• a high-sensitivity monochrome image is obtained as the captured image from the second fluorescence Lw3 and the third fluorescence Lw4 in the invisible light wavelength band that does not have color information. Therefore, even if it is difficult to capture an appropriate image using the first imaging element 522a because the light quantity of the second fluorescence Lw3 and the third fluorescence Lw4 is small, the second imaging element 522b, which has excellent sensitivity, can capture the second fluorescence Lw3 and the third fluorescence Lw4 appropriately.
• FIG. 17 is a diagram illustrating the types of light incident on the imaging elements (first imaging element 522a and second imaging element 522b) according to the third embodiment.
• the camera head 50 (particularly the imaging unit 52) of this embodiment is equipped with a two-plate type imaging module (see Figures 3 and 4), and is equipped with a first imaging element 522a having a color filter CF, and a second imaging element 522b not having a color filter CF.
• the resolution of the first imaging element 522a may be higher than the resolution of the second imaging element 522b
• the sensitivity of the second imaging element 522b may be higher than the sensitivity of the first imaging element 522a.
• the resolution and sensitivity of the first imaging element 522a and the second imaging element 522b are not limited to this, and the relative relationship of the resolution and sensitivity between the first imaging element 522a and the second imaging element 522b is not limited to this.
• the color filter CF provided on the first image sensor 522a in this embodiment transmits the broadband reflected light Lw1 and the first fluorescence Lw2 received by the first image sensor 522a as described below, but may or may not transmit light of a wavelength band different from the broadband reflected light Lw1 and the first fluorescence Lw2 (e.g., the second fluorescence Lw3).
• the second image sensor 522b does not have a color filter CF in this example, but may have a color filter CF that can transmit the broadband reflected light Lw1, the first fluorescence Lw2, and the second fluorescence Lw3 as described below.
• the light source device 10 (see FIG. 1A) emits light from at least one of a broadband light source 11, a first narrowband light source 12, and a second narrowband light source 13, and can irradiate the object of observation S with at least one of the broadband light, the first narrowband light, and the second narrowband light.
• the observation light Lf from the observation target S may include broadband reflected light Lw1, which is reflected light of broadband light, a first fluorescence Lw2 emitted from a first substance excited by the first narrowband light, and a second fluorescence Lw3 emitted from a second substance excited by the second narrowband light.
• the broadband reflected light Lw1 and the first fluorescence Lw2 are light contained in a first wavelength band
• the second fluorescence Lw3 is light contained in a second wavelength band outside the first wavelength band.
• the observation light Lf incident on the optical element 15 is separated into a first light beam Lf1 and a second light beam Lf2 by the optical element 15.
• the optical element 15 guides a portion of the light included in the first wavelength band to the first image sensor 522a as the first light beam Lf1, and guides a portion of the light included in the first wavelength band and the light included in the second wavelength band to the second image sensor 522b as the second light beam Lf2.
• the first luminous flux Lf1 in which the light in the first wavelength band of the observation light Lf is partially suppressed and the light in the wavelength band of the second fluorescence Lw3 is partially, substantially, or completely suppressed, is guided to the first imaging element 522a. That is, light including at least a portion of the broadband reflected light Lw1 is guided to the first imaging element 522a as the first luminous flux Lf1.
• the second luminous flux Lf2 in which the light in the first wavelength band of the observation light Lf is partially suppressed, is guided to the second imaging element 522b. That is, light including at least a portion of the broadband reflected light Lw1 and the first and second fluorescence Lw3 is guided to the second imaging element 522b as the second luminous flux Lf2.
• the medical observation system 100 of this embodiment having the above-mentioned configuration can acquire various captured images of the observation subject S according to the following observation modes (first to third modes).
• the first mode of this embodiment is an observation mode in which broadband light and first narrowband light are irradiated onto the observation target S, and a captured image is obtained based on the broadband reflected light Lw1 and the first fluorescent light Lw2 from the observation target S.
• the control device 90 controls the light source device 10 (broadband light source 11 and first narrowband light source 12 (see FIG. 1A)), and the light source device 10 emits broadband light and first narrowband light in a time-division manner, and irradiates the broadband light and first narrowband light on the observation object S in a time-division manner.
• the second narrowband light source 13 is kept in an off state, and the light source device 10 does not emit the second narrowband light.
• the optical element 15 then sequentially guides the first light beam Lf1 including the broadband reflected light Lw1 from the observation object S irradiated with the broadband light, and the first light beam Lf1 including the first fluorescence Lw2 from the observation object S irradiated with the first narrowband light, to the first image sensor 522a.
• the optical element 15 also sequentially guides the second light beam Lf2 including the broadband reflected light Lw1 from the observation object S irradiated with the broadband light, and the second light beam Lf2 including the first fluorescence Lw2 from the observation object S irradiated with the first narrowband light, to the second image sensor 522b.
• the first image sensor 522a sequentially receives the first light beam Lf1 containing the broadband reflected light Lw1 and the first light beam Lf1 containing the first fluorescent light Lw2.
• the second image sensor 522b sequentially receives the second light beam Lf2 containing the broadband reflected light Lw1 and the second light beam Lf2 containing the first fluorescent light Lw2.
• each of the first image sensor 522a and the second image sensor 522b sequentially and repeatedly outputs an image signal based on the broadband reflected light Lw1 and an image signal based on the first fluorescent light Lw2 under the control of the control device 90 (control unit 94 (see FIG. 7)).
• the image generation unit 93 (particularly the image processing unit 931 (see FIG. 7)) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 based on the image signal output from the first image sensor 522a which has received the first light beam Lf1 including the broadband reflected light Lw1. Also, under the control of the control unit 94, the image generation unit 93 (image processing unit 931) generates a captured image (high-sensitivity monochrome image) of the observation target S based on the first fluorescence Lw2 based on the image signal output from the second image sensor 522b which has received the second light beam Lf2 including the first fluorescence Lw2.
• the "high-resolution color image” referred to here is a color image acquired by the high-resolution image sensor
• the "high-sensitivity monochrome image” is a monochrome image acquired by the high-sensitivity image sensor.
• FIG. 18 shows an example of a timing chart of light source emission and image sensor exposure in the first mode of the third embodiment.
• FIG. 18(a) shows the exposure state of the first imaging element 522a
• (c) shows the exposure state of the second imaging element 522b
• the vertical axis shows the horizontal lines of the first imaging element 522a and the second imaging element 522b
• the horizontal axis shows time
• line R1 shows the timing at which pixel data readout of each horizontal line for each image frame begins.
• the "broadband light image frame" between lines R1 is an image frame for receiving (exposing) the broadband reflected light Lw1 from the observation target S
• the "first fluorescence image frame” between lines R1 is an image frame for receiving (exposing) the first fluorescence Lw2 from the observation target S.
• FIG. 18(b) shows the emission timing of the broadband light from the broadband light source 11, and (d) shows the emission timing of the first narrowband light from the first narrowband light source 12.
• the light source device 10 emits broadband light and first narrowband light and irradiates the observation target S in a time-division manner.
• the broadband reflected light Lw1 and the first fluorescent light Lw2 from the observation target S are both guided to both the first imaging element 522a and the second imaging element 522b, and are sequentially received by each of the first imaging element 522a and the second imaging element 522b.
• the captured image based on the broadband reflected light Lw1 is generated based on the image signal from the first imaging element 522a
• the captured image based on the first fluorescent light Lw2 is generated based on the image signal from the second imaging element 522b.
• control device 90 controls the timing of time-division light emission by the light source device 10 and the timing of image data readout from the first image sensor 522a and the second image sensor 522b so that they are correlated. Specifically, based on a common synchronization signal, the broadband light source 11 and the first narrowband light source 12 emit broadband light and the first narrowband light in a time-division manner, and the first image sensor 522a and the second image sensor 522b are exposed and image data are read out.
• the broadband light and the first narrowband light are emitted alternately and separately in time so that the first image sensor 522a and the second image sensor 522b are not exposed to both the broadband reflected light Lw1 and the first fluorescent light Lw2 at the same time.
• image data is read out so that an image signal of a broadband light image frame exposed by the broadband reflected light Lw1 and an image signal of a first fluorescence image frame exposed by the first fluorescence Lw2 are output from the first imaging element 522a and the second imaging element 522b.
• each of the first imaging element 522a and the second imaging element 522b repeatedly outputs an image signal of the broadband light image frame and an image signal of the first fluorescence image frame alternately.
• the image generating unit 93 (image processing unit 931) generates a normal light captured image of the observation target S, which is a reflected image of broadband light (white light), from the image signal of the broadband light image frame output from the first imaging element 522a.
• the image generating unit 93 (image processing unit 931) also generates a first fluorescent captured image, which is a captured image based on the first fluorescence Lw2 of the observation target S, from the image signal of the first fluorescent image frame output from the second imaging element 522b.
• a normal light image which is a reflected image of visible light (white light)
• a first fluorescent image in which the first substance in the observation target S is emphasized, i.e., an image in which fluorescence is emitted from the first substance in the observation target S excited by narrowband light
• the user can compare the normal light image and the first fluorescent image of the observation target S via the display device 70 (see FIG. 1A), and can also observe a composite image (superimposed image) created from these images.
• the emission of broadband light and first narrowband light is started and ended while image data is not being read out from the first imaging element 522a and the second imaging element 522b. Therefore, in the broadband light image frame of the first imaging element 522a and the second imaging element 522b, it is possible to prevent the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light from entering the first imaging element 522a. Also, in the first fluorescence image frame of the first imaging element 522a and the second imaging element 522b, it is possible to prevent the broadband reflected light Lw1 from the observation target S irradiated with the broadband light from entering the first imaging element 522a and the second imaging element 522b.
• the timings for starting and ending the emission of the broadband light and the first narrowband light are not limited to the example shown in FIG. 18, and can be set to any timing.
• the emission of one or both of the broadband light and the first narrowband light may be started or ended.
• the intensity of the fluorescence to be received in the first fluorescence image frame is weak (i.e., when the amount of fluorescence emission is small)
• the first narrowband light may be emitted from the first narrowband light source 12 while the image data of the broadband light image frame is being read out.
• the exposure time of the fluorescence in the first fluorescence image frame can be lengthened, which is advantageous for obtaining bright image data of the first fluorescence image frame.
• the first narrowband light source 12 may continuously emit the first narrowband light.
• the first image sensor 522a also outputs an image signal of the first fluorescent image frame
• the second image sensor 522b also outputs an image signal of the broadband light image frame, but in the above example, these image frames are not used to generate the captured image.
• image data of these image frames that are not used to generate the captured image are also repeatedly output as image signals from the imaging elements (first imaging element 522a and second imaging element 522b), but image data of image frames that are not used to generate the captured image do not have to be output as image signals from the imaging elements.
• image data of image frames that are used to generate the captured image are output as image signals from the imaging elements
• the accumulated charge of the imaging elements is reset for image frames that are not used to generate the captured image, and they do not have to be output as image signals from the imaging elements.
• these image frames that are not used to generate the captured image may be used to generate the captured image.
• the captured image generated from the first fluorescent image frame from the first image sensor 522a and the broadband light image frame from the second image sensor 522b may or may not be used to generate the output image.
• Captured images that are not used to generate an output image can be used for any purpose.
• a correction process for the brightness of the output image may be performed by the image generation unit 93 (see FIG. 7) based on the results of image analysis of captured images that are not used to generate an output image.
• a process for adjusting the focal position such as contrast AF (autofocus) may be performed based on the results of image analysis of captured images that are not used to generate an output image, and the focal position may be adjusted based on the contrast and spatial frequency of such captured images.
• a captured image based on the first fluorescence Lw2 may be generated based on the image signal of the first fluorescence image frame from the first image sensor 522a.
• a captured image based on the first fluorescence Lw2 may be generated based on the image signals of the first fluorescence image frame from both the first image sensor 522a and the second image sensor 522b.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signal of the broadband light image frame from the second imaging element 522b.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signals of the broadband light image frame from both the first imaging element 522a and the second imaging element 522b.
• the second mode of this embodiment is an observation mode in which the object S is irradiated with broadband light and second narrowband light, and an image is captured based on the broadband reflected light Lw1 and second fluorescent light Lw3 from the object S.
• control device 90 controls the light source device 10 (broadband light source 11 and second narrowband light source 13), and the broadband light and second narrowband light are emitted from the light source device 10 in a time-division manner, and the broadband light and second narrowband light are irradiated to the observation object S in a time-division manner.
• the first narrowband light source 12 is kept in an off state, and the first narrowband light is not emitted from the light source device 10.
• the optical element 15 guides the first light beam Lf1, which includes the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, to the first image sensor 522a.
• the optical element 15 also guides the second light beam Lf2, which includes the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, and the second light beam Lf2, which includes the second fluorescence Lw3 from the observation target S irradiated with the second narrowband light, to the second image sensor 522b in sequence.
• the first image sensor 522a receives a first light beam Lf1 that includes the broadband reflected light Lw1.
• the second image sensor 522b sequentially receives a second light beam Lf2 that includes the broadband reflected light Lw1 and a second light beam Lf2 that includes the second fluorescent light Lw3.
• the first image sensor 522a repeatedly outputs an image signal based on the broadband reflected light Lw1 under the control of the control device 90 (controller 94).
• the second image sensor 522b repeatedly outputs an image signal based on the broadband reflected light Lw1 and an image signal based on the second fluorescence Lw3 under the control of the control device 90 (controller 94).
• the image generation unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 based on the image signal output from the first imaging element 522a which has received the first light beam Lf1 including the broadband reflected light Lw1. Also, under the control of the control unit 94, the image generation unit 93 (image processing unit 931) generates a captured image (high-sensitivity monochrome image) of the observation target S based on the second fluorescence Lw3 based on the image signal output from the second imaging element 522b which has received the second light beam Lf2 including the second fluorescence Lw3.
• the "high-resolution color image” referred to here is a color image acquired by the high-resolution imaging element
• the "high-sensitivity monochrome image” is a monochrome image acquired by the high-sensitivity imaging element.
• a normal light image which is a reflected image of visible light (white light)
• a second fluorescent image in which the second substance in the observation target S is emphasized, i.e., an image in which fluorescence is emitted from the second substance in the observation target S excited by narrowband light
• the user can compare the normal light image and the second fluorescent image of the observation target S via the display device 70, and can also observe a composite image (superimposed image) created from these images.
• the broadband light and the second narrowband light are emitted in a time-division manner, but the light source device 10 may also emit the second narrowband light continuously under the control of the control device 90 (control unit 94) while repeatedly turning the broadband light on and off.
• the light source device 10 (broadband light source 11) emits broadband light so that the first imaging element 522a is exposed to the broadband reflected light Lw1 in the broadband light image frame of the first imaging element 522a.
• the light source device 10 (broadband light source 11) stops emitting broadband light so that the second imaging element 522b is not exposed to the broadband reflected light Lw1 in the second fluorescence image frame of the second imaging element 522b. This allows the first imaging element 522a to properly output an image signal of the broadband light image frame, and allows the second imaging element 522b to properly output an image signal of the second fluorescence image frame.
• the timings for starting and ending the emission of the broadband light and the second narrowband light are not limited and can be set to any timing. For example, while image data is being read out from the first image sensor 522a and the second image sensor 522b (see “R1" in FIG. 18), the emission of one or both of the broadband light and the second narrowband light may be started or ended. For example, while image data of the second fluorescence image frame is being read out, the broadband light source 11 may emit broadband light.
• the second imaging element 522b also outputs an image signal of a broadband light image frame, but in the above example, the broadband light image frame from the second imaging element 522b is not used to generate the captured image.
• image data of the broadband light image frame that is not used to generate the captured image is repeatedly output as an image signal from the second imaging element 522b, but image data of the broadband light image frame that is not used to generate the captured image does not have to be output as an image signal from the second imaging element 522b.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signal of a broadband light image frame from the second imaging element 522b that is not used to generate the captured image.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signals of the broadband light image frames from both the first imaging element 522a and the second imaging element 522b.
• the captured image generated in this manner may or may not be used to generate an output image.
• Captured images that are not used to generate the output image can be used for any purpose (for example, to correct the brightness of the output image or to adjust the focus position).
• the third mode of this embodiment is an observation mode in which broadband light, first narrowband light, and second narrowband light are irradiated onto the object of observation S, and a captured image is obtained based on the broadband reflected light Lw1, first fluorescence Lw2, and second fluorescence Lw3 from the object of observation S.
• control device 90 controls the light source device 10 (broadband light source 11, first narrowband light source 12, and second narrowband light source 13), and the light source device 10 emits broadband light, first narrowband light, and second narrowband light in a time-division manner.
• the broadband light, first narrowband light, and second narrowband light are irradiated to the observation object S in a time-division manner.
• the optical element 15 sequentially guides the first light beam Lf1 including the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, and the first light beam Lf1 including the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light, to the first image sensor 522a.
• the optical element 15 also sequentially guides the second light beam Lf2 including the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, the second light beam Lf2 including the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light, and the second light beam Lf2 including the second fluorescence Lw3 from the observation target S irradiated with the second narrowband light, to the second image sensor 522b.
• the first image sensor 522a sequentially receives the first light beam Lf1 including the broadband reflected light Lw1 and the first light beam Lf1 including the first fluorescent light Lw2. Then, under the control of the control device 90 (control unit 94), the first image sensor 522a repeatedly outputs an image signal based on the broadband reflected light Lw1 and an image signal based on the first fluorescent light Lw2.
• the second image sensor 522b sequentially receives the second light beam Lf2 including the broadband reflected light Lw1, the second light beam Lf2 including the first fluorescence Lw2, and the second light beam Lf2 including the second fluorescence Lw3. Then, under the control of the control device 90 (control unit 94), the second image sensor 522b repeatedly outputs an image signal based on the broadband reflected light Lw1, an image signal based on the first fluorescence Lw2, and an image signal based on the second fluorescence Lw3.
• the image generating unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1, based on an image signal output from the first imaging element 522a that has received the first light beam Lf1 including the broadband reflected light Lw1, under the control of the control unit 94.
• the image generating unit 93 (image processing unit 931) also generates a captured image (high-sensitivity monochrome image) of the observation target S based on the first fluorescence Lw2, based on an image signal output from the second imaging element 522b that has received the second light beam Lf2 including the first fluorescence Lw2, under the control of the control unit 94.
• the image generating unit 93 (image processing unit 931) also generates a captured image (high-sensitivity monochrome image) of the observation target S based on the second fluorescence Lw3, based on an image signal output from the second imaging element 522b that has received the second light beam Lf2 including the second fluorescence Lw3, under the control of the control unit 94.
• a "high-resolution color image” here refers to a color image captured by a high-resolution imaging element
• a "high-sensitivity monochrome image” refers to a monochrome image captured by a high-sensitivity imaging element.
• a normal light image which is a reflected image of visible light (white light)
• a first fluorescent image and a second fluorescent image which are images in which the first and second substances in the observation target S are highlighted, i.e., images in which fluorescence is emitted from the first and second substances in the observation target S excited by narrowband light
• the user can compare the normal light image, the first fluorescent image, and the second fluorescent image of the observation target S, and can also observe a composite image (superimposed image) created from these images.
• the first imaging element 522a also outputs an image signal of the first fluorescent image frame
• the second imaging element 522b also outputs an image signal of the broadband light image frame, but in the above example, these image frames are not used to generate the captured image.
• image data of these image frames that are not used to generate the captured image are also repeatedly output as image signals from the imaging elements (first imaging element 522a and second imaging element 522b), but image data of image frames that are not used to generate the captured image do not have to be output as image signals from the imaging elements.
• a captured image based on the first fluorescence Lw2 may be generated based on the image signal of the first fluorescence image frame from the first imaging element 522a that is not used to generate the captured image.
• a captured image based on the first fluorescence Lw2 may be generated based on the image signals of the first fluorescence image frame from both the first imaging element 522a and the second imaging element 522b.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signal of the broadband light image frame from the second imaging element 522b.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signals of the broadband light image frame from both the first imaging element 522a and the second imaging element 522b.
• the captured image generated in this manner may or may not be used to generate an output image.
• Captured images that are not used to generate the output image can be used for any purpose (for example, to correct the brightness of the output image or to adjust the focus position).
• FIG. 19 is a diagram explaining the types of light incident on the imaging elements (first imaging element 522a and second imaging element 522b) according to the fourth embodiment.
• the camera head 50 (particularly the imaging unit 52) of this embodiment has a two-plate type imaging module (see Figures 3 and 4) and is equipped with a first imaging element 522a having a color filter CF and a second imaging element 522b having no color filter CF.
• the first imaging element 522a has a relatively low sensitivity and a high resolution (e.g., 4K resolution) compared to the second imaging element 522b, while the second imaging element 522b has a relatively high sensitivity and a low resolution (e.g., HD resolution) compared to the first imaging element 522a.
• the color filter CF provided on the first imaging element 522a in this embodiment transmits the broadband reflected light Lw1 received by the first imaging element 522a as described below, but may or may not transmit light of a wavelength band different from the broadband reflected light Lw1 (e.g., the second fluorescence Lw3).
• the second imaging element 522b in this example does not have a color filter CF, but may have a color filter CF that can transmit the first fluorescence Lw2 and the second fluorescence Lw3 as described below.
• the light source device 10 (see FIG. 1A) emits light from at least one of a broadband light source 11, a first narrowband light source 12, and a second narrowband light source 13, and can irradiate the object of observation S with at least one of the broadband light, the first narrowband light, and the second narrowband light.
• the observation light Lf from the observation target S may include broadband reflected light Lw1, which is reflected light of broadband light, a first fluorescence Lw2 emitted from a first substance excited by the first narrowband light, and a second fluorescence Lw3 emitted from a second substance excited by the second narrowband light.
• the broadband reflected light Lw1 and the first fluorescence Lw2 are light contained in a first wavelength band
• the second fluorescence Lw3 is light contained in a second wavelength band outside the first wavelength band.
• the observation light Lf incident on the optical element 15 is separated into a first light beam Lf1 and a second light beam Lf2 by the optical element 15.
• the optical element 15 guides a portion of the light contained in the first wavelength band to the first image sensor 522a as the first light beam Lf1, and guides at least the first fluorescence Lw2 and the light contained in the second wavelength band to the second image sensor 522b as the second light beam Lf2.
• the first luminous flux Lf1 in which light in the wavelength bands of the first fluorescence Lw2 and the second fluorescence Lw3 of the observation light Lf is partially, substantially, or completely suppressed, is guided to the first image sensor 522a. That is, light including at least a portion of the broadband reflected light Lw1 is guided to the first image sensor 522a as the first luminous flux Lf1.
• the second luminous flux Lf2 in which light in wavelength bands other than the wavelength band of the first fluorescence Lw2 of the observation light Lf is partially, substantially, or completely suppressed, is guided to the second image sensor 522b. That is, light including at least the first fluorescence Lw2 and the second fluorescence Lw3 is guided to the second image sensor 522b as the second luminous flux Lf2.
• the medical observation system 100 of this embodiment having the above-mentioned configuration can acquire various captured images of the observation subject S according to the following observation modes (first to third modes).
• the first mode of this embodiment is an observation mode in which broadband light and first narrowband light are irradiated onto the observation target S, and a captured image is obtained based on the broadband reflected light Lw1 and the first fluorescent light Lw2 from the observation target S.
• control device 90 controls the light source device 10 (broadband light source 11 and first narrowband light source 12 (see FIG. 1A)), and the broadband light and first narrowband light are emitted from the light source device 10 in a time-division manner, and the broadband light and first narrowband light are irradiated onto the observation object S in a time-division manner.
• the second narrowband light source 13 is kept in an off state, and the second narrowband light is not emitted from the light source device 10.
• the optical element 15 guides the first light beam Lf1, which includes the broadband reflected light Lw1 (wherein the light in the wavelength band of the first fluorescence Lw2 is partially, substantially, or completely suppressed) from the observation target S irradiated with the broadband light, to the first image sensor 522a.
• the optical element 15 also guides the second light beam Lf2, which includes the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light, to the second image sensor 522b.
• the first image capturing element 522a receives a first light beam Lf1 that includes the broadband reflected light Lw1 (wherein light in the wavelength band of the first fluorescence Lw2 is partially, substantially or completely suppressed), and the second image capturing element 522b receives a second light beam Lf2 that includes the first fluorescence Lw2. Then, under the control of the control device 90 (controller 94), the first image capturing element 522a repeatedly outputs an image signal based on the broadband reflected light Lw1 (wherein light in the wavelength band of the first fluorescence Lw2 is partially, substantially or completely suppressed), and the second image capturing element 522b repeatedly outputs an image signal based on the first fluorescence Lw2.
• the image generation unit 93 (particularly the image processing unit 931 (see FIG. 7)) generates a photographed image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 based on the image signal output from the first image sensor 522a that receives the first light beam Lf1 including the broadband reflected light Lw1 (wherein the light in the wavelength band of the first fluorescence Lw2 is partially, substantially, or completely suppressed).
• the image generation unit 93 (image processing unit 931) generates a photographed image (high-sensitivity monochrome image) of the observation target S based on the first fluorescence Lw2 based on the image signal output from the second image sensor 522b that receives the second light beam Lf2 including the first fluorescence Lw2.
• the "high-resolution color image” referred to here is a color image acquired by the high-resolution image sensor
• the "high-sensitivity monochrome image” is a monochrome image acquired by the high-sensitivity image sensor.
• a normal light image which is a reflected image of visible light (white light)
• a first fluorescent image in which the first substance in the observation target S is emphasized, i.e., an image in which fluorescence is emitted from the first substance in the observation target S excited by narrowband light
• the user can compare the normal light image and the first fluorescent image of the observation target S via the display device 70 (see FIG. 1A), and can also observe a composite image (superimposed image) created from these images.
• the broadband light and the first narrowband light are emitted in a time-division manner, but the light source device 10 may also emit the broadband light and/or the first narrowband light continuously under the control of the control device 90 (control unit 94).
• the first image sensor 522a When constant broadband light is emitted, the first image sensor 522a is continuously exposed to the broadband reflected light Lw1 (wherein the light in the wavelength band of the first fluorescent light Lw2 is partially, substantially or completely suppressed).
• the second image sensor 522b When constant first narrowband light is emitted, the second image sensor 522b is continuously exposed to the first fluorescent light Lw2. Therefore, in these cases, the amount of charge stored in the image sensor increases, making it possible to capture bright images, suppress an increase in noise caused by gain adjustment, and prevent a substantial decrease in frame rate.
• the second mode of this embodiment is an observation mode in which the object S is irradiated with broadband light and second narrowband light, and an image is captured based on the broadband reflected light Lw1 and second fluorescent light Lw3 from the object S.
• control device 90 controls the light source device 10 (broadband light source 11 and second narrowband light source 13), and the broadband light and second narrowband light are emitted from the light source device 10 in a time-division manner, and the broadband light and second narrowband light are irradiated to the observation object S in a time-division manner.
• the first narrowband light source 12 is kept in an off state, and the first narrowband light is not emitted from the light source device 10.
• the optical element 15 guides a first light beam Lf1, which includes broadband reflected light Lw1 (wherein light in the wavelength band of the first fluorescence Lw2 is partially, substantially, or completely suppressed) from the observation target S irradiated with the broadband light, to the first image sensor 522a.
• the optical element 15 also guides a second light beam Lf2, which includes second fluorescence Lw3 from the observation target S irradiated with the second narrowband light, to the second image sensor 522b.
• the first image sensor 522a receives a first light beam Lf1 that includes broadband reflected light Lw1 (wherein light in the wavelength band of the first fluorescence Lw2 is partially, substantially, or completely suppressed), and the second image sensor 522b receives a second light beam Lf2 that includes the second fluorescence Lw3.
• the first image sensor 522a Under the control of the control device 90 (control unit 94), the first image sensor 522a repeatedly outputs an image signal based on the broadband reflected light Lw1 (wherein the light in the wavelength band of the first fluorescence Lw2 is partially, substantially, or completely suppressed), and the second image sensor 522b repeatedly outputs an image signal based on the second fluorescence Lw3.
• the image generation unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 based on an image signal output from the first image sensor 522a that receives the first light beam Lf1 including the broadband reflected light Lw1 (wherein the light in the wavelength band of the first fluorescence Lw2 is partially, substantially, or completely suppressed).
• the image generation unit 93 (image processing unit 931) generates a captured image (high-sensitivity monochrome image) of the observation target S based on the second fluorescence Lw3 based on an image signal output from the second image sensor 522b that receives the second light beam Lf2 including the second fluorescence Lw3.
• the "high-resolution color image” referred to here is a color image acquired by the high-resolution image sensor
• the "high-sensitivity monochrome image” is a monochrome image acquired by the high-sensitivity image sensor.
• a normal light image which is a reflected image of visible light (white light)
• a second fluorescent image in which the second substance in the observation target S is emphasized, i.e., an image in which fluorescence is emitted from the second substance in the observation target S excited by narrowband light
• the user can compare the normal light image and the second fluorescent image of the observation target S via the display device 70, and can also observe a composite image (superimposed image) created from these images.
• the broadband light and the second narrowband light are emitted in a time-division manner, but the light source device 10 may also emit the broadband light and/or the second narrowband light continuously under the control of the control device 90 (control unit 94).
• the first image sensor 522a When constant emission of broadband light is performed, the first image sensor 522a is continuously exposed to the broadband reflected light Lw1 (wherein the light in the wavelength band of the first fluorescent light Lw2 is partially, substantially, or completely suppressed).
• the second image sensor 522b can be continuously exposed to the second fluorescent light Lw3. Therefore, in these cases, the amount of charge stored in the image sensor increases, making it possible to capture bright images, suppress an increase in noise caused by gain adjustment, and prevent a substantial decrease in the frame rate.
• the third mode of this embodiment is an observation mode in which broadband light, first narrowband light, and second narrowband light are irradiated onto the object of observation S, and a captured image is obtained based on the broadband reflected light Lw1, first fluorescence Lw2, and second fluorescence Lw3 from the object of observation S.
• control device 90 controls the light source device 10 (broadband light source 11, first narrowband light source 12, and second narrowband light source 13), and while broadband light is continuously emitted from the light source device 10, the light source device 10 emits the first narrowband light and the second narrowband light in a time-division manner.
• the broadband light is continuously irradiated onto the observation object S, and the first narrowband light and the second narrowband light are irradiated onto the observation object S in a time-division manner.
• the optical element 15 continuously guides the first light beam Lf1, which includes the broadband reflected light Lw1 (wherein the light in the wavelength band of the first fluorescence Lw2 is partially, substantially, or completely suppressed) from the observation target S irradiated with the broadband light, to the first image sensor 522a.
• the optical element 15 also sequentially guides the second light beam Lf2, which includes the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light, and the second light beam Lf2, which includes the second fluorescence Lw3 from the observation target S irradiated with the second narrowband light, to the second image sensor 522b.
• the first image sensor 522a continuously receives the first light beam Lf1, which includes the broadband reflected light Lw1 (wherein the light in the wavelength band of the first fluorescent light Lw2 is partially, substantially, or completely suppressed), and repeatedly outputs an image signal based on the broadband reflected light Lw1 under the control of the control device 90 (control unit 94).
• the second image sensor 522b sequentially receives the second light beam Lf2 containing the first fluorescence Lw2 and the second light beam Lf2 containing the second fluorescence Lw3. Then, under the control of the control device 90 (control unit 94), the second image sensor 522b repeatedly outputs an image signal based on the first fluorescence Lw2 and an image signal based on the second fluorescence Lw3.
• the image generation unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 based on an image signal output from the first image sensor 522a that receives the first light beam Lf1 including the broadband reflected light Lw1 (wherein light in the wavelength band of the first fluorescence Lw2 is partially, substantially, or completely suppressed).
• the image generation unit 93 (image processing unit 931) generates a captured image (high-sensitivity monochrome image) of the observation target S based on the first fluorescence Lw2 based on an image signal output from the second image sensor 522b that receives the second light beam Lf2 including the first fluorescence Lw2. Furthermore, under the control of the control unit 94, the image generation unit 93 (image processing unit 931) generates a captured image (high-sensitivity monochrome image) of the observation target S based on the second fluorescence Lw3, based on an image signal output from the second imaging element 522b that receives the second light flux Lf2 including the second fluorescence Lw3.
• the "high-resolution color image” referred to here is a color image acquired by the high-resolution imaging element
• the "high-sensitivity monochrome image” is a monochrome image acquired by the high-sensitivity imaging element.
• a normal light image which is a reflected image of visible light (white light)
• a first fluorescent image and a second fluorescent image which are images in which the first and second substances in the observation target S are highlighted, i.e., images in which fluorescence is emitted from the first and second substances in the observation target S excited by narrowband light
• the user can compare the normal light image, the first fluorescent image, and the second fluorescent image of the observation target S, and can also observe a composite image (superimposed image) created from these images.
• broadband light is constantly emitted, but the light source device 10 may repeatedly turn the emission of broadband light on and off under the control of the control device 90 (control unit 94).
• FIG. 20 is a diagram illustrating the types of light incident on the imaging elements (first imaging element 522a and second imaging element 522b) according to the fifth embodiment.
• the camera head 50 (particularly the imaging unit 52) of this embodiment has a two-plate type imaging module (see Figures 3 and 4) and includes a first imaging element 522a having a color filter CF and a second imaging element 522b having a color filter CF.
• the color filter CF provided on the second imaging element 522b includes a filter that can transmit the second fluorescence Lw3 in addition to the broadband reflected light Lw1 and the first fluorescence Lw2.
• the color filter CF provided on the first imaging element 522a transmits the broadband reflected light Lw1 and the first fluorescence Lw2 received by the first imaging element 522a as described below, but may or may not transmit light of a wavelength band different from the broadband reflected light Lw1 and the first fluorescence Lw2 (e.g., the second fluorescence Lw3).
• the first imaging element 522a has a relatively low sensitivity and a high resolution (e.g., 4K resolution) compared to the second imaging element 522b, while the second imaging element 522b has a relatively high sensitivity and a low resolution (e.g., HD resolution) compared to the first imaging element 522a.
• the light source device 10 (see FIG. 1A) emits light from at least one of a broadband light source 11, a first narrowband light source 12, and a second narrowband light source 13, and can irradiate the object of observation S with at least one of the broadband light, the first narrowband light, and the second narrowband light.
• the observation light Lf from the observation target S may include broadband reflected light Lw1, which is reflected light of broadband light, a first fluorescence Lw2 emitted from a first substance excited by the first narrowband light, and a second fluorescence Lw3 emitted from a second substance excited by the second narrowband light.
• the broadband reflected light Lw1 and the first fluorescence Lw2 are light contained in a first wavelength band
• the second fluorescence Lw3 is light contained in a second wavelength band outside the first wavelength band.
• the observation light Lf incident on the optical element 15 is separated into a first light beam Lf1 and a second light beam Lf2 by the optical element 15.
• the optical element 15 guides a portion of the light included in the first wavelength band to the first image sensor 522a as the first light beam Lf1, and guides a portion of the light included in the first wavelength band and the light included in the second wavelength band to the second image sensor 522b as the second light beam Lf2.
• the first luminous flux Lf1 in which the light in the first wavelength band of the observation light Lf is partially suppressed and the light in the wavelength band of the second fluorescence Lw3 is partially, substantially, or completely suppressed, is guided to the first imaging element 522a. That is, light including at least a portion of the broadband reflected light Lw1 is guided to the first imaging element 522a as the first luminous flux Lf1.
• the second luminous flux Lf2 in which the light in the first wavelength band of the observation light Lf is partially suppressed, is guided to the second imaging element 522b. That is, light including at least a portion of the broadband reflected light Lw1 and the first and second fluorescence Lw3 is guided to the second imaging element 522b as the second luminous flux Lf2.
• the medical observation system 100 of this embodiment having the above-mentioned configuration can acquire various captured images of the observation subject S according to the following observation modes (first to third modes).
• the first mode of this embodiment is an observation mode in which broadband light and first narrowband light are irradiated onto the observation target S, and a captured image is obtained based on the broadband reflected light Lw1 and the first fluorescent light Lw2 from the observation target S.
• control device 90 controls the light source device 10 (broadband light source 11 and first narrowband light source 12 (see FIG. 1A)), and the broadband light and first narrowband light are emitted from the light source device 10 in a time-division manner, and the broadband light and first narrowband light are irradiated onto the observation object S in a time-division manner.
• the second narrowband light source 13 is kept in an off state, and the second narrowband light is not emitted from the light source device 10.
• the optical element 15 then sequentially guides the first light beam Lf1 including the broadband reflected light Lw1 from the observation object S irradiated with the broadband light, and the first light beam Lf1 including the first fluorescence Lw2 from the observation object S irradiated with the first narrowband light, to the first image sensor 522a.
• the optical element 15 also sequentially guides the second light beam Lf2 including the broadband reflected light Lw1 from the observation object S irradiated with the broadband light, and the second light beam Lf2 including the first fluorescence Lw2 from the observation object S irradiated with the first narrowband light, to the second image sensor 522b.
• the first image sensor 522a sequentially receives the first light beam Lf1 containing the broadband reflected light Lw1 and the first light beam Lf1 containing the first fluorescent light Lw2.
• the second image sensor 522b sequentially receives the second light beam Lf2 containing the broadband reflected light Lw1 and the second light beam Lf2 containing the first fluorescent light Lw2.
• the first image sensor 522a sequentially and repeatedly outputs an image signal based on the broadband reflected light Lw1 and an image signal based on the first fluorescent light Lw2.
• these image signals output from the first image sensor 522a are not used to generate the captured image.
• the second image sensor 522b repeatedly outputs an image signal based on the broadband reflected light Lw1 and an image signal based on the first fluorescent light Lw2 in sequence.
• the image generation unit 93 (particularly the image processing unit 931 (see FIG. 7)) generates a captured image (high-sensitivity color image) of the observation target S based on the broadband reflected light Lw1 based on the image signal output from the second image sensor 522b that has received the second light beam Lf2 including the broadband reflected light Lw1. Also, under the control of the control unit 94, the image generation unit 93 (image processing unit 931) generates a captured image (high-sensitivity color image) of the observation target S based on the first fluorescence Lw2 based on the image signal output from the second image sensor 522b that has received the second light beam Lf2 including the first fluorescence Lw2.
• the "high-sensitivity color image” referred to here is a color image acquired by the high-sensitivity image sensor.
• a normal light image which is a reflected image of visible light (white light)
• a first fluorescent image in which the first substance in the observation target S is emphasized, i.e., an image in which fluorescence is emitted from the first substance in the observation target S excited by narrowband light
• the user can compare the normal light image and the first fluorescent image of the observation target S via the display device 70 (see FIG. 1A), and can also observe a composite image (superimposed image) created from these images.
• the image signal output from the second image sensor 522b is used to generate the captured image based on the broadband reflected light Lw1 and the first fluorescent light Lw2, but the image signal output from the first image sensor 522a may also be used.
• control unit 94 may control the imaging unit 52 (imaging element 522) and the image generation unit 93 so that an image based on the broadband reflected light Lw1 is generated based on one or both of the image signal output from the first imaging element 522a that receives the first light beam Lf1 including the broadband reflected light Lw1 and the image signal output from the second imaging element 522b that receives the second light beam Lf2 including the broadband reflected light Lw1.
• a captured image (high-resolution color image) based on the broadband reflected light Lw1 may be generated based on the image signal output from the first imaging element 522a.
• a captured image (high-sensitivity color image) based on the broadband reflected light Lw1 may be generated based on the image signal output from the second imaging element 522b.
• the "high-resolution color image” referred to here is a color image acquired by the high-resolution imaging element
• the "high-sensitivity color image” is a color image acquired by the high-sensitivity imaging element.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signal output from the first imaging element 522a.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signal output from the second imaging element 522b.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signal output from the second imaging element 522b while suppressing the amount of broadband light emitted by the light source device 10.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signal output from the first imaging element 522a and the image signal output from the second imaging element 522b.
• the image generation unit 93 image processing unit 931 may use any image synthesis technology to generate a high-quality "captured image based on the broadband reflected light Lw1" based on the image signals output from the first imaging element 522a and the second imaging element 522b.
• the control unit 94 may determine, based on instructions from a user received via the input unit 95, whether to use either or both of the image signal output from the first imaging element 522a and the image signal output from the second imaging element 522b to generate an image based on the broadband reflected light Lw1.
• the second mode of this embodiment is an observation mode in which the object S is irradiated with broadband light and second narrowband light, and an image is captured based on the broadband reflected light Lw1 and second fluorescent light Lw3 from the object S.
• control device 90 controls the light source device 10 (broadband light source 11 and second narrowband light source 13), and the broadband light and second narrowband light are emitted from the light source device 10 in a time-division manner, and the broadband light and second narrowband light are irradiated to the observation object S in a time-division manner.
• the first narrowband light source 12 is kept in an off state, and the first narrowband light is not emitted from the light source device 10.
• the optical element 15 guides the first light beam Lf1, which includes the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, to the first image sensor 522a.
• the optical element 15 also guides the second light beam Lf2, which includes the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, and the second light beam Lf2, which includes the second fluorescence Lw3 from the observation target S irradiated with the second narrowband light, to the second image sensor 522b in sequence.
• the first image sensor 522a receives the first light beam Lf1 including the broadband reflected light Lw1, and under the control of the control device 90 (controller 94), repeatedly outputs an image signal based on the broadband reflected light Lw1.
• the second image sensor 522b sequentially receives the second light beam Lf2 including the broadband reflected light Lw1 and the second light beam Lf2 including the second fluorescence Lw3.
• the second image sensor 522b sequentially and repeatedly outputs an image signal based on the broadband reflected light Lw1 and an image signal based on the second fluorescence Lw3, under the control of the control device 90 (controller 94).
• the image generating unit 93 (image processing unit 931) generates a photographed image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 based on the image signal output from the first image sensor 522a that receives the first light beam Lf1 including the broadband reflected light Lw1. Also, under the control of the control unit 94, the image generating unit 93 (image processing unit 931) generates a photographed image (high-sensitivity image) of the observation target S based on the second fluorescence Lw3 based on the image signal output from the second image sensor 522b that receives the second light beam Lf2 including the second fluorescence Lw3.
• the "high-resolution color image” referred to here is a color image acquired by the high-resolution image sensor
• the "high-sensitivity image” is an image acquired by the high-sensitivity image sensor.
• the image based on the second fluorescence Lw3 generated in this way can be called a color image because it is based on the image signal output from the second image sensor 522b having a color filter CF, but can also be called a monochrome image because it does not substantially contain color information.
• a normal light image which is a reflected image of visible light (white light)
• a second fluorescent image in which the second substance in the observation target S is emphasized, i.e., an image in which fluorescence is emitted from the second substance in the observation target S excited by narrowband light
• the user can compare the normal light image and the second fluorescent image of the observation target S via the display device 70, and can also observe a composite image (superimposed image) created from these images.
• the broadband light and the second narrowband light are emitted in a time-division manner, but the light source device 10 may also emit the second narrowband light continuously under the control of the control device 90 (control unit 94) while repeatedly turning the broadband light on and off.
• the light source device 10 (broadband light source 11) emits broadband light so that the first imaging element 522a is exposed to the broadband reflected light Lw1 in the broadband light image frame of the first imaging element 522a.
• the light source device 10 (broadband light source 11) stops emitting broadband light so that the second imaging element 522b is not exposed to the broadband reflected light Lw1 in the second fluorescence image frame of the second imaging element 522b. This allows the first imaging element 522a to properly output an image signal of the broadband light image frame, and allows the second imaging element 522b to properly output an image signal of the second fluorescence image frame.
• the second imaging element 522b also outputs an image signal of a broadband light image frame, but in the above example, the broadband light image frame from the second imaging element 522b is not used to generate a captured image.
• image data of a broadband light image frame that is not used to generate a captured image is repeatedly output as an image signal from the second imaging element 522b, but image data of a broadband light image frame that is not used to generate a captured image does not have to be output as an image signal from the second imaging element 522b.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signal of a broadband light image frame from the second imaging element 522b that is not used to generate the captured image.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signals of the broadband light image frames from both the first imaging element 522a and the second imaging element 522b.
• the captured image generated in this manner may or may not be used to generate an output image.
• Captured images that are not used to generate the output image can be used for any purpose (for example, to correct the brightness of the output image or to adjust the focus position).
• the third mode of this embodiment is an observation mode in which broadband light, first narrowband light, and second narrowband light are irradiated onto the object of observation S, and a captured image is obtained based on the broadband reflected light Lw1, first fluorescence Lw2, and second fluorescence Lw3 from the object of observation S.
• control device 90 controls the light source device 10 (broadband light source 11, first narrowband light source 12, and second narrowband light source 13), and the light source device 10 emits broadband light, first narrowband light, and second narrowband light in a time-division manner.
• the broadband light, first narrowband light, and second narrowband light are irradiated to the observation object S in a time-division manner.
• the optical element 15 sequentially guides the first light beam Lf1 including the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, and the first light beam Lf1 including the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light, to the first image sensor 522a.
• the optical element 15 also sequentially guides the second light beam Lf2 including the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, the second light beam Lf2 including the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light, and the second light beam Lf2 including the second fluorescence Lw3 from the observation target S irradiated with the second narrowband light, to the second image sensor 522b.
• the first image sensor 522a sequentially receives the first light beam Lf1 including the broadband reflected light Lw1 and the first light beam Lf1 including the first fluorescent light Lw2. Then, under the control of the control device 90 (control unit 94), the first image sensor 522a repeatedly outputs an image signal based on the broadband reflected light Lw1 and an image signal based on the first fluorescent light Lw2.
• the second image sensor 522b sequentially receives the second light beam Lf2 including the broadband reflected light Lw1, the second light beam Lf2 including the first fluorescence Lw2, and the second light beam Lf2 including the second fluorescence Lw3. Then, under the control of the control device 90 (control unit 94), the second image sensor 522b repeatedly outputs an image signal based on the broadband reflected light Lw1, an image signal based on the first fluorescence Lw2, and an image signal based on the second fluorescence Lw3.
• the image generating unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1, based on an image signal output from the first image sensor 522a that has received the first light beam Lf1 including the broadband reflected light Lw1, under the control of the control unit 94.
• the image generating unit 93 (image processing unit 931) also generates a captured image (high-sensitivity color image) of the observation target S based on the first fluorescence Lw2, based on an image signal output from the second image sensor 522b that has received the second light beam Lf2 including the first fluorescence Lw2, under the control of the control unit 94.
• the image generating unit 93 (image processing unit 931) also generates a captured image (high-sensitivity image) of the observation target S based on the second fluorescence Lw3, based on an image signal output from the second image sensor 522b that has received the second light beam Lf2 including the second fluorescence Lw3, under the control of the control unit 94.
• a "high-resolution color image” is a color image captured by a high-resolution imaging element
• a "high-sensitivity color image” is a color image captured by a high-sensitivity imaging element
• a "high-sensitivity image” is an image captured by a high-sensitivity imaging element.
• a normal light image which is a reflected image of visible light (white light)
• a first fluorescent image and a second fluorescent image which are images in which the first and second substances in the observation target S are highlighted, i.e., images in which fluorescence is emitted from the first and second substances in the observation target S excited by narrowband light
• the user can compare the normal light image, the first fluorescent image, and the second fluorescent image of the observation target S, and can also observe a composite image (superimposed image) created from these images.
• the first imaging element 522a also outputs an image signal of the first fluorescent image frame
• the second imaging element 522b also outputs an image signal of the broadband light image frame, but in the above example, these image frames are not used to generate the captured image.
• image data of these image frames that are not used to generate the captured image are also repeatedly output as image signals from the imaging elements (first imaging element 522a and second imaging element 522b), but image data of image frames that are not used to generate the captured image do not have to be output as image signals from the imaging elements.
• these image frames that are not used to generate the captured image may be used to generate the captured image.
• a captured image based on the first fluorescence Lw2 may be generated based on the image signal of the first fluorescence image frame from the first image sensor 522a.
• a captured image based on the first fluorescence Lw2 may be generated based on the image signals of the first fluorescence image frame from both the first image sensor 522a and the second image sensor 522b.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signal of the broadband light image frame from the second image sensor 522b.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signals of the broadband light image frame from both the first image sensor 522a and the second image sensor 522b.
• the captured image generated in this manner may or may not be used to generate an output image.
• Captured images that are not used to generate the output image can be used for any purpose (for example, to correct the brightness of the output image or to adjust the focus position).
• FIG. 21 is a diagram illustrating the types of light incident on the imaging elements (first imaging element 522a, second imaging element 522b, and third imaging element 522c) according to the sixth embodiment.
• the camera head 50 (particularly the imaging unit 52) of this embodiment is equipped with a three-plate type imaging module (see Figures 5 and 6), and is equipped with a first imaging element 522a having a color filter CF, and a second imaging element 522b and a third imaging element 522c that do not have a color filter CF.
• the color filter CF provided on the first imaging element 522a in this embodiment transmits the broadband reflected light Lw1 and the first fluorescence Lw2 received by the first imaging element 522a as described below, but may or may not transmit light of a wavelength band different from the broadband reflected light Lw1 and the first fluorescence Lw2 (e.g., the second fluorescence Lw3).
• the second imaging element 522b and the third imaging element 522c in this example do not have a color filter CF, but the second imaging element 522b may have a color filter CF that can transmit the broadband reflected light Lw1 and the first fluorescence Lw2, and the third imaging element 522c may have a color filter CF that can transmit the second fluorescence Lw3.
• the first imaging element 522a has a relatively low sensitivity and a high resolution (e.g., 4K resolution) compared to the second imaging element 522b and the third imaging element 522c, while the second imaging element 522b and the third imaging element 522c have a relatively high sensitivity and a low resolution (e.g., HD resolution) compared to the first imaging element 522a.
• the second imaging element 522b and the third imaging element 522c may have the same characteristics as each other, or may have different characteristics from each other.
• the resolution of the first imaging element 522a is higher than the resolution of the second imaging element 522b and the third imaging element 522c, but the sensitivity of the second imaging element 522b and the third imaging element 522c is higher than the sensitivity of the first imaging element 522a.
• the resolution and sensitivity of the first to third image pickup elements 522a to 522c are not limited to this, and the relationship between the resolution and sensitivity of the first to third image pickup elements 522a to 522c is also not limited to this.
• the light source device 10 (see FIG. 1A) emits at least one of a broadband light source 11, a first narrowband light source 12, and a second narrowband light source 13, and can irradiate at least one of broadband light, first narrowband light, and second narrowband light to the observation target S. Therefore, the observation light Lf from the observation target S can include broadband reflected light Lw1, which is reflected light of the broadband light, a first fluorescence Lw2 emitted from a first substance excited by the first narrowband light, and a second fluorescence Lw3 emitted from a second substance excited by the second narrowband light.
• the broadband reflected light Lw1 and the first fluorescence Lw2 are light included in the first wavelength band
• the second fluorescence Lw3 is light included in a second wavelength band outside the first wavelength band.
• the observation light Lf incident on the optical element 15 is separated into a first light beam Lf1, a second light beam Lf2, and a third light beam Lf3 by the optical element 15.
• the optical element 15 guides a portion of the light included in the first wavelength band to the first image sensor 522a as the first light beam Lf1, guides a portion of the light included in the first wavelength band to the second image sensor 522b as the second light beam Lf2, and guides the light included in the second wavelength band to the third image sensor 522c as the third light beam Lf3.
• the first light beam Lf1 in which the light in the first wavelength band of the observation light Lf is partially suppressed and the light in the wavelength band of the second fluorescence Lw3 is partially, substantially or completely suppressed, is guided to the first imaging element 522a by the optical element 15. That is, light including at least a part of the broadband reflected light Lw1 is guided to the first imaging element 522a as the first light beam Lf1.
• the second light beam Lf2 in which the light in the first wavelength band of the observation light Lf is partially suppressed and the light in the wavelength band of the second fluorescence Lw3 is partially, substantially or completely suppressed, is guided to the second imaging element 522b.
• the third light beam Lf3 in which the light in the first wavelength band of the observation light Lf is partially, substantially or completely suppressed, is guided to the third imaging element 522c. That is, light that includes at least the second fluorescent light Lw3 is guided to the third image sensor 522c as the third light beam Lf3.
• the medical observation system 100 of this embodiment having the above-mentioned configuration can acquire various captured images of the observation subject S according to the following observation modes (first mode and second mode).
• the first mode of this embodiment is an observation mode in which broadband light and first narrowband light are irradiated onto the observation target S, and a captured image is obtained based on the broadband reflected light Lw1 and the first fluorescent light Lw2 from the observation target S.
• control device 90 controls the light source device 10 (broadband light source 11 and first narrowband light source 12 (see FIG. 1A)), and the broadband light and first narrowband light are emitted from the light source device 10 in a time-division manner, and the broadband light and first narrowband light are irradiated onto the observation object S in a time-division manner.
• the second narrowband light source 13 is kept in an off state, and the second narrowband light is not emitted from the light source device 10.
• the optical element 15 sequentially guides the first light beam Lf1 including the broadband reflected light Lw1 from the observation object S irradiated with the broadband light, and the first light beam Lf1 including the first fluorescence Lw2 from the observation object S irradiated with the first narrowband light, to the first image sensor 522a.
• the first image sensor 522a sequentially receives the first light beam Lf1 including the broadband reflected light Lw1 and the first light beam Lf1 including the first fluorescence Lw2.
• the optical element 15 also sequentially guides the second light beam Lf2 including the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, and the second light beam Lf2 including the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light, to the second image sensor 522b.
• the second image sensor 522b sequentially receives the second light beam Lf2 including the broadband reflected light Lw1 and the second light beam Lf2 including the first fluorescence Lw2.
• each of the first image sensor 522a and the second image sensor 522b sequentially and repeatedly outputs an image signal based on the broadband reflected light Lw1 and an image signal based on the first fluorescent light Lw2 under the control of the control device 90 (control unit 94 (see FIG. 7)).
• the image generation unit 93 (particularly the image processing unit 931 (see FIG. 7)) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 from the image signal based on the broadband reflected light Lw1 output from the first imaging element 522a. Also, under the control of the control unit 94, the image generation unit 93 (image processing unit 931) generates a captured image (high-sensitivity monochrome image) of the observation target S based on the first fluorescence Lw2 from the image signal based on the first fluorescence Lw2 output from the second imaging element 522b.
• the "high-resolution color image” referred to here is a color image acquired by the high-resolution imaging element
• the "high-sensitivity monochrome image” is a monochrome image acquired by the high-sensitivity imaging element.
• a normal light image which is a reflected image of visible light (white light)
• a first fluorescent image in which the first substance in the observation target S is emphasized, i.e., an image in which fluorescence is emitted from the first substance in the observation target S excited by narrowband light
• the user can compare the normal light image and the first fluorescent image of the observation target S via the display device 70 (see FIG. 1A), and can also observe a composite image (superimposed image) created from these images.
• the first imaging element 522a also outputs an image signal of the first fluorescent image frame
• the second imaging element 522b also outputs an image signal of the broadband light image frame, but in the above example, these image frames are not used to generate the captured image.
• image data of these image frames that are not used to generate the captured image are also repeatedly output as image signals from the imaging elements (first imaging element 522a and second imaging element 522b), but image data of image frames that are not used to generate the captured image do not have to be output as image signals from the imaging elements.
• these image frames that are not used to generate the captured image may be used to generate the captured image.
• a captured image based on the first fluorescence Lw2 may be generated based on the image signal of the first fluorescence image frame from the first image sensor 522a.
• a captured image based on the first fluorescence Lw2 may be generated based on the image signals of the first fluorescence image frame from both the first image sensor 522a and the second image sensor 522b.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signal of the broadband light image frame from the second imaging element 522b.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signals of the broadband light image frame from both the first imaging element 522a and the second imaging element 522b.
• the captured image generated in this manner may or may not be used to generate an output image.
• Captured images that are not used to generate the output image can be used for any purpose (for example, to correct the brightness of the output image or to adjust the focus position).
• the second mode of this embodiment is an observation mode in which the object S is irradiated with broadband light and second narrowband light, and an image is captured based on the broadband reflected light Lw1 and second fluorescent light Lw3 from the object S.
• control device 90 controls the light source device 10 (broadband light source 11 and second narrowband light source 13), and the broadband light and second narrowband light are continuously emitted from the light source device 10, and the broadband light and second narrowband light are continuously irradiated onto the observation object S.
• the first narrowband light source 12 is kept in an off state, and the first narrowband light is not emitted from the light source device 10.
• the optical element 15 continuously guides the first light beam Lf1, which includes the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, to the first image sensor 522a, and continuously guides the second light beam Lf2, which includes the broadband reflected light Lw1, to the second image sensor 522b.
• the optical element 15 also continuously guides the third light beam Lf3, which includes the second fluorescence Lw3 from the observation target S irradiated with the second narrowband light, to the third image sensor 522c.
• the first imaging element 522a continuously receives the first light beam Lf1 including the broadband reflected light Lw1
• the second imaging element 522b continuously receives the second light beam Lf2 including the broadband reflected light Lw1
• the third imaging element 522c continuously receives the third light beam Lf3 including the second fluorescent light Lw3.
• the first image sensor 522a and the second image sensor 522b continuously and repeatedly output image signals based on the broadband reflected light Lw1, and the third image sensor 522c continuously and repeatedly output image signals based on the second fluorescent light Lw3.
• the image generation unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 from the image signal based on the broadband reflected light Lw1 output from the first imaging element 522a. Also, under the control of the control unit 94, the image generation unit 93 (image processing unit 931) generates a captured image (high-sensitivity monochrome image) of the observation target S based on the second fluorescence Lw3 from the image signal based on the second fluorescence Lw3 output from the third imaging element 522c.
• the "high-resolution color image” here refers to a color image acquired by the high-resolution imaging element
• the "high-sensitivity monochrome image” refers to a monochrome image acquired by the high-sensitivity imaging element.
• a normal light image which is a reflected image of visible light (white light)
• a second fluorescent image in which the second substance in the observation target S is emphasized, i.e., an image in which fluorescence is emitted from the second substance in the observation target S excited by narrowband light
• the user can compare the normal light image and the second fluorescent image of the observation target S via the display device 70, and can also observe a composite image (superimposed image) created from these images.
• the broadband light and the second narrowband light are constantly emitted from the light source device 10, but the light source device 10 may turn off the emission of each of the broadband light and the second narrowband light midway under the control of the control device 90 (control unit 94).
• the light source device 10 may repeatedly turn on and off the emission of the broadband light and the second narrowband light, or may emit the broadband light and the second narrowband light in a time-division manner.
• the first image sensor 522a can continuously receive the broadband reflected light Lw1
• the third image sensor 522c can continuously receive the second fluorescent light Lw3. This increases the amount of charge stored in the image sensor, making it possible to capture bright images, suppresses the increase in noise caused by gain adjustment, and prevents a decrease in the effective frame rate.
• the second imaging element 522b also outputs an image signal of a broadband light image frame, but in the above example, the broadband light image frame from the second imaging element 522b is not used to generate a captured image.
• image data of a broadband light image frame that is not used to generate a captured image is repeatedly output as an image signal from the second imaging element 522b, but image data of a broadband light image frame that is not used to generate a captured image does not have to be output as an image signal from the second imaging element 522b.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signal of a broadband light image frame from the second imaging element 522b that is not used to generate the captured image.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signals of the broadband light image frames from both the first imaging element 522a and the second imaging element 522b.
• the captured image generated in this manner may or may not be used to generate an output image.
• Captured images that are not used to generate the output image can be used for any purpose (for example, to correct the brightness of the output image or to adjust the focus position).
• the imaging element according to the seventh embodiment has the same configuration as the imaging element according to the sixth embodiment described above (see FIG. 21).
• the camera head 50 (particularly the imaging unit 52) of this embodiment is equipped with a three-plate type imaging module (see Figures 5 and 6), and is equipped with a first imaging element 522a having a color filter CF, and a second imaging element 522b and a third imaging element 522c that do not have a color filter CF.
• the color filter CF provided on the first imaging element 522a in this embodiment transmits the broadband reflected light Lw1 and the first fluorescence Lw2 received by the first imaging element 522a as described below, but may or may not transmit light of a wavelength band different from the broadband reflected light Lw1 and the first fluorescence Lw2 (e.g., the second fluorescence Lw3).
• the second imaging element 522b and the third imaging element 522c in this example do not have a color filter CF, but the second imaging element 522b may have a color filter CF that can transmit the broadband reflected light Lw1 and the first fluorescence Lw2, and the third imaging element 522c may have a color filter CF that can transmit the second fluorescence Lw3.
• the first imaging element 522a has a relatively low sensitivity and a high resolution (e.g., 4K resolution) compared to the second imaging element 522b and the third imaging element 522c, while the second imaging element 522b and the third imaging element 522c have a relatively high sensitivity and a low resolution (e.g., HD resolution) compared to the first imaging element 522a.
• the second imaging element 522b and the third imaging element 522c may have the same characteristics as each other, or may have different characteristics from each other.
• the resolution of the first imaging element 522a is higher than the resolution of the second imaging element 522b and the third imaging element 522c, but the sensitivity of the second imaging element 522b and the third imaging element 522c is higher than the sensitivity of the first imaging element 522a.
• the resolution and sensitivity of the first to third image pickup elements 522a to 522c are not limited to this, and the relationship between the resolution and sensitivity of the first to third image pickup elements 522a to 522c is also not limited to this.
• the light source device 10 (see FIG. 1A) emits at least one of a broadband light source 11, a first narrowband light source 12, and a second narrowband light source 13, and can irradiate at least one of broadband light, first narrowband light, and second narrowband light to the observation target S. Therefore, the observation light Lf from the observation target S can include broadband reflected light Lw1, which is reflected light of the broadband light, a first fluorescence Lw2 emitted from a first substance excited by the first narrowband light, and a second fluorescence Lw3 emitted from a second substance excited by the second narrowband light.
• the broadband reflected light Lw1 and the first fluorescence Lw2 are light included in the first wavelength band
• the second fluorescence Lw3 is light included in a second wavelength band outside the first wavelength band.
• the observation light Lf incident on the optical element 15 is separated into a first light beam Lf1, a second light beam Lf2, and a third light beam Lf3 by the optical element 15.
• the optical element 15 guides a portion of the light included in the first wavelength band to the first image sensor 522a as the first light beam Lf1, guides a portion of the light included in the first wavelength band to the second image sensor 522b as the second light beam Lf2, and guides the light included in the second wavelength band to the third image sensor 522c as the third light beam Lf3.
• the first light beam Lf1 in which the light in the first wavelength band of the observation light Lf is partially suppressed and the light in the wavelength band of the second fluorescence Lw3 is partially, substantially or completely suppressed, is guided to the first imaging element 522a by the optical element 15. That is, light including at least a part of the broadband reflected light Lw1 is guided to the first imaging element 522a as the first light beam Lf1.
• the second light beam Lf2 in which the light in the first wavelength band of the observation light Lf is partially suppressed and the light in the wavelength band of the second fluorescence Lw3 is partially, substantially or completely suppressed, is guided to the second imaging element 522b.
• the third light beam Lf3 in which the light in the first wavelength band of the observation light Lf is partially, substantially or completely suppressed, is guided to the third imaging element 522c. That is, light that includes at least the second fluorescent light Lw3 is guided to the third image sensor 522c as the third light beam Lf3.
• the medical observation system 100 of this embodiment having the above-mentioned configuration can acquire various captured images of the observation subject S according to the following observation modes (first to third modes).
• the light source device 10 emits broadband light and the first narrowband light in a time-division manner, and the broadband reflected light Lw1 and the first fluorescent light Lw2 are received in a time-division manner by the first and second image capture elements 522a and 522b.
• the light source device 10 continuously emits broadband light and the second narrowband light, the broadband reflected light Lw1 is continuously received by the first and second image capture elements 522a and 522b, and the second fluorescent light Lw3 is continuously received by the third image capture element 522c.
• the third mode of this embodiment is an observation mode in which broadband light, first narrowband light, and second narrowband light are irradiated onto the object of observation S, and a captured image is obtained based on the broadband reflected light Lw1, first fluorescence Lw2, and second fluorescence Lw3 from the object of observation S.
• control device 90 controls the light source device 10 (broadband light source 11, first narrowband light source 12, and second narrowband light source 13), and the light source device 10 emits broadband light and first narrowband light in a time-division manner, and emits the second narrowband light continuously.
• the broadband light and first narrowband light are irradiated to the observation object S in a time-division manner, and the second narrowband light is irradiated to the observation object S continuously.
• the optical element 15 sequentially guides the first light beam Lf1 including the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, and the first light beam Lf1 including the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light, to the first image sensor 522a.
• the optical element 15 also sequentially guides the second light beam Lf2 including the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, and the second light beam Lf2 including the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light, to the second image sensor 522b.
• the optical element 15 also continuously guides the third light beam Lf3 including the second fluorescence Lw3 from the observation target S irradiated with the second narrowband light, to the third image sensor 522c.
• the first imaging element 522a sequentially receives the first light beam Lf1 including the broadband reflected light Lw1 and the first light beam Lf1 including the first fluorescence Lw2.
• the second imaging element 522b sequentially receives the second light beam Lf2 including the broadband reflected light Lw1 and the second light beam Lf2 including the first fluorescence Lw2.
• the third imaging element 522c continuously receives the third light beam Lf3 including the second fluorescence Lw3.
• the first image sensor 522a and the second image sensor 522b each sequentially and repeatedly output an image signal based on the broadband reflected light Lw1 and an image signal based on the first fluorescence Lw2 under the control of the control device 90 (controller 94).
• the second image sensor 522b also continuously and repeatedly outputs an image signal based on the second fluorescence Lw3 under the control of the control device 90 (controller 94).
• the image generating unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 from the image signal based on the broadband reflected light Lw1 output from the first imaging element 522a under the control of the control unit 94.
• the image generating unit 93 (image processing unit 931) also generates a captured image (high-sensitivity monochrome image) of the observation target S based on the first fluorescence Lw2 from the image signal based on the first fluorescence Lw2 output from the second imaging element 522b under the control of the control unit 94.
• the image generating unit 93 (image processing unit 931) also generates a captured image (high-sensitivity monochrome image) of the observation target S based on the second fluorescence Lw3 from the image signal based on the second fluorescence Lw3 output from the third imaging element 522c under the control of the control unit 94.
• the "high-resolution color image” referred to here is a color image acquired by the high-resolution imaging element
• the "high-sensitivity monochrome image” is a monochrome image acquired by the high-sensitivity imaging element.
• FIG. 22 shows an example of a timing chart of light source emission and image sensor exposure in the third mode of the seventh embodiment.
• the vertical axis indicates the horizontal lines of the first imaging element 522a, the second imaging element 522b, and the third imaging element 522c, the horizontal axis indicates time, and the line R1 indicates the timing of starting to read pixel data for each horizontal line for each image frame.
• the "broadband light image frame" between the lines R1 is an image frame for receiving (exposing) the broadband reflected light Lw1 from the observation target S.
• the “first fluorescence image frame” between the lines R1 is an image frame for receiving (exposing) the first fluorescence Lw2 from the observation target S.
• the “second fluorescence image frame” between the lines R1 is an image frame for receiving (exposing) the second fluorescence Lw3 from the observation target S.
• FIG. 22 shows the emission timing of broadband light from the broadband light source 11
• (d) shows the emission timing of the first narrowband light from the first narrowband light source 12
• (f) shows the emission timing of the second narrowband light from the second narrowband light source 13.
• the light source device 10 emits the broadband light and the first narrowband light and irradiates the observation object S in a time-division manner.
• the light source device 10 emits the second narrowband light and irradiates the observation object S continuously.
• the control device 90 controls so that the timing of the time-division emission of the broadband light and the first narrowband light from the light source device 10 and the timing of reading out the image data from the first image sensor 522a and the second image sensor 522b are correlated to each other.
• the broadband light source 11 and the first narrowband light source 12 emit broadband light and the first narrowband light in a time-division manner, and the first image sensor 522a and the second image sensor 522b are exposed and image data is read out. More specifically, the broadband light and the first narrowband light are emitted alternately and separately in time so that the first image sensor 522a and the second image sensor 522b are not exposed to both the broadband reflected light Lw1 and the first fluorescent light Lw2 at the same time.
• image data is read out so that an image signal of a broadband light image frame exposed by the broadband reflected light Lw1 and an image signal of a first fluorescence image frame exposed by the first fluorescence Lw2 are output from the first imaging element 522a and the second imaging element 522b.
• each of the first imaging element 522a and the second imaging element 522b repeatedly outputs an image signal of the broadband light image frame and an image signal of the first fluorescence image frame alternately.
• image data can be read from the third image sensor 522c at any timing while the light source device 10 is emitting the second narrowband light.
• the third image sensor 522c continuously and repeatedly outputs the image signal of the second fluorescent light image frame.
• the image generation unit 93 (image processing unit 931) generates a normal light captured image of the observation target S, which is a reflected image of broadband light (white light), from the image signal of the broadband light image frame output from the first imaging element 522a.
• the image generation unit 93 (image processing unit 931) also generates a first fluorescent captured image, which is a captured image based on the first fluorescence Lw2 of the observation target S, from the image signal of the first fluorescent image frame output from the second imaging element 522b.
• the image generation unit 93 (image processing unit 931) also generates a second fluorescent captured image, which is a captured image based on the second fluorescence Lw3 of the observation target S, from the image signal of the second fluorescent image frame output from the third imaging element 522c.
• a normal light image which is a reflected image of visible light (white light)
• a first fluorescent image in which a first substance in the observation target S is emphasized i.e., an image in which fluorescence is emitted from a first substance in the observation target S excited by narrow band light
• a second fluorescent image in which a second substance in the observation target S is emphasized i.e., an image in which fluorescence is emitted from a second substance in the observation target S excited by narrow band light
• the user can compare and observe the normal light image, first fluorescent image, and second fluorescent image of the observation target S via the display device 70, and can also observe a composite image (superimposed image) created from these images.
• the emission of broadband light and first narrowband light is started and ended while image data is not being read out from the first imaging element 522a and the second imaging element 522b. Therefore, in the broadband light image frame of the first imaging element 522a and the second imaging element 522b, it is possible to prevent the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light from entering the first imaging element 522a. Also, in the first fluorescence image frame of the first imaging element 522a and the second imaging element 522b, it is possible to prevent the broadband reflected light Lw1 from the observation target S irradiated with the broadband light from entering the first imaging element 522a and the second imaging element 522b.
• the timings for starting and ending the emission of the broadband light and the first narrowband light are not limited to the example shown in FIG. 22 and can be set to any timing.
• the emission of one or both of the broadband light and the first narrowband light may be started or ended.
• the intensity of the fluorescence to be received in the first fluorescence image frame is weak (i.e., when the amount of fluorescence emission is small)
• the first narrowband light may be emitted from the first narrowband light source 12 while image data of the broadband light image frame is being read out.
• the exposure time of the fluorescence in the first fluorescence image frame can be lengthened, which is advantageous for obtaining bright image data of the first fluorescence image frame.
• the first narrowband light source 12 may continuously emit the first narrowband light.
• the first imaging element 522a also outputs an image signal of the first fluorescent image frame
• the second imaging element 522b also outputs an image signal of the broadband light image frame, but in the above example, these image frames are not used to generate the captured image.
• image data of these image frames that are not used to generate the captured image are also repeatedly output as image signals from the imaging elements (first imaging element 522a and second imaging element 522b), but image data of image frames that are not used to generate the captured image do not have to be output as image signals from the imaging elements.
• these image frames that are not used to generate the captured image may be used to generate the captured image.
• a captured image based on the first fluorescence Lw2 may be generated based on the image signal of the first fluorescence image frame from the first image sensor 522a.
• a captured image based on the first fluorescence Lw2 may be generated based on the image signals of the first fluorescence image frame from both the first image sensor 522a and the second image sensor 522b.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signal of the broadband light image frame from the second imaging element 522b.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signals of the broadband light image frame from both the first imaging element 522a and the second imaging element 522b.
• the captured image generated in this manner may or may not be used to generate an output image.
• Captured images that are not used to generate the output image can be used for any purpose (for example, to correct the brightness of the output image or to adjust the focus position).
• FIG. 23 is a diagram illustrating the types of light incident on the imaging elements (first imaging element 522a, second imaging element 522b, and third imaging element 522c) according to the eighth embodiment.
• the camera head 50 (particularly the imaging unit 52) of this embodiment is equipped with a three-plate type imaging module (see Figures 5 and 6), and is equipped with a first imaging element 522a having a color filter CF, and a second imaging element 522b and a third imaging element 522c that do not have a color filter CF.
• the color filter CF provided on the first imaging element 522a in this embodiment transmits the broadband reflected light Lw1 and the first fluorescence Lw2 received by the first imaging element 522a as described below, but may or may not transmit light of a wavelength band different from the broadband reflected light Lw1 and the first fluorescence Lw2 (for example, the second fluorescence Lw3 and the third fluorescence Lw4).
• the second imaging element 522b and the third imaging element 522c in this example do not have a color filter CF, but the second imaging element 522b may have a color filter CF that can transmit the broadband reflected light Lw1 and the first fluorescence Lw2, and the third imaging element 522c may have a color filter CF that can transmit the second fluorescence Lw3 and the third fluorescence Lw4.
• the first imaging element 522a has a relatively low sensitivity and a high resolution (e.g., 4K resolution) compared to the second imaging element 522b and the third imaging element 522c, while the second imaging element 522b and the third imaging element 522c have a relatively high sensitivity and a low resolution (e.g., HD resolution) compared to the first imaging element 522a.
• the second imaging element 522b and the third imaging element 522c may have the same characteristics as each other, or may have different characteristics from each other.
• the resolution of the first imaging element 522a is higher than the resolution of the second imaging element 522b and the third imaging element 522c, but the sensitivity of the second imaging element 522b and the third imaging element 522c is higher than the sensitivity of the first imaging element 522a.
• the resolution and sensitivity of the first to third image pickup elements 522a to 522c are not limited to this, and the relationship between the resolution and sensitivity of the first to third image pickup elements 522a to 522c is also not limited to this.
• the light source device 10 emits at least one of the broadband light source 11, the first narrowband light source 12, the second narrowband light source 13, and the third narrowband light source 14, and can irradiate at least one of the broadband light, the first narrowband light, the second narrowband light, and the third narrowband light to the observation target S. Therefore, the observation light Lf from the observation target S can include broadband reflected light Lw1, which is reflected light of the broadband light, a first fluorescence Lw2 emitted from a first substance excited by the first narrowband light, a second fluorescence Lw3 emitted from a second substance excited by the second narrowband light, and a third fluorescence Lw4 emitted from a third substance excited by the third narrowband light.
• the broadband reflected light Lw1 and the first fluorescence Lw2 are light included in the first wavelength band
• the second fluorescence Lw3 and the third fluorescence Lw4 are light included in a second wavelength band outside the first wavelength band.
• the observation light Lf incident on the optical element 15 is separated into a first light beam Lf1, a second light beam Lf2, and a third light beam Lf3 by the optical element 15.
• the optical element 15 guides a portion of the light included in the first wavelength band to the first image sensor 522a as the first light beam Lf1, guides a portion of the light included in the first wavelength band to the second image sensor 522b as the second light beam Lf2, and guides the light included in the second wavelength band to the third image sensor 522c as the third light beam Lf3.
• the first light beam Lf1 in which the light in the first wavelength band of the observation light Lf is partially suppressed and the light in the wavelength bands of the second fluorescence Lw3 and the third fluorescence Lw4 is partially, substantially or completely suppressed, is guided to the first imaging element 522a by the optical element 15. That is, light including at least a portion of the broadband reflected light Lw1 is guided to the first imaging element 522a as the first light beam Lf1.
• the second light beam Lf2 in which the light in the first wavelength band of the observation light Lf is partially suppressed and the light in the wavelength bands of the second fluorescence Lw3 and the third fluorescence Lw4 is partially, substantially or completely suppressed, is guided to the second imaging element 522b. That is, light including at least a portion of the broadband reflected light Lw1 and the first fluorescence Lw2 is guided to the second imaging element 522b as the second light beam Lf2.
• a third light beam Lf3, in which the light in the first wavelength band of the observation light Lf is partially, substantially, or completely suppressed, is guided to the third image sensor 522c. That is, light including at least the second fluorescence Lw3 and the third fluorescence Lw4 is guided to the third image sensor 522c as the third light beam Lf3.
• the medical observation system 100 of this embodiment which has the above-mentioned configuration, can acquire various captured images of the observation subject S according to the following observation modes (modes 1 to 5).
• the light source device 10 emits broadband light and the first narrowband light in a time-division manner, and the broadband reflected light Lw1 and the first fluorescent light Lw2 are received in a time-division manner by the first and second image capture elements 522a and 522b.
• the light source device 10 continuously emits broadband light and the second narrowband light, the broadband reflected light Lw1 is continuously received by the first and second image capture elements 522a and 522b, and the second fluorescent light Lw3 is continuously received by the third image capture element 522c.
• the light source device 10 emits broadband light and the first narrowband light in a time-division manner and continuously emits the second narrowband light, the broadband reflected light Lw1 and the first fluorescent light Lw2 are received in a time-division manner by the first and second image capture elements 522a and 522b, and the second fluorescent light Lw3 is continuously received by the third image capture element 522c.
• the third narrowband light source 14 is in an off state, and the third narrowband light is not emitted from the light source device 10.
• the fourth mode of this embodiment is an observation mode in which the object S is irradiated with broadband light and third narrowband light, and an image is captured based on the broadband reflected light Lw1 and the third fluorescent light Lw4 from the object S.
• control device 90 controls the light source device 10 (broadband light source 11 and third narrowband light source 14), and the broadband light and third narrowband light are continuously emitted from the light source device 10, and the broadband light and third narrowband light are continuously irradiated onto the observation object S.
• the first narrowband light source 12 and the second narrowband light source 13 are kept in an off state, and the first narrowband light and the second narrowband light are not emitted from the light source device 10.
• the optical element 15 continuously guides the first light beam Lf1, which includes the broadband reflected light Lw1 from the observation object S irradiated with the broadband light, to the first image sensor 522a.
• the optical element 15 also continuously guides the second light beam Lf2, which includes the broadband reflected light Lw1 from the observation object S irradiated with the broadband light, to the second image sensor 522b.
• the optical element 15 also continuously guides the third light beam Lf3, which includes the third fluorescence Lw4 from the observation object S irradiated with the third narrowband light, to the third image sensor 522c.
• the first image sensor 522a continuously receives the first light beam Lf1 including the broadband reflected light Lw1, and under the control of the control device 90 (controller 94), continuously and repeatedly outputs an image signal based on the broadband reflected light Lw1.
• the second image sensor 522b continuously receives the second light beam Lf2 including the broadband reflected light Lw1, and under the control of the control device 90 (controller 94), continuously and repeatedly outputs an image signal based on the broadband reflected light Lw1.
• the third image sensor 522c continuously receives the third light beam Lf3 including the third fluorescence Lw4, and under the control of the control device 90 (controller 94), continuously and repeatedly outputs an image signal based on the third fluorescence Lw4.
• the image generation unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 from the image signal based on the broadband reflected light Lw1 output from the first imaging element 522a. Also, under the control of the control unit 94, the image generation unit 93 (image processing unit 931) generates a captured image (high-sensitivity monochrome image) of the observation target S based on the third fluorescence Lw4 from the image signal based on the third fluorescence Lw4 output from the third imaging element 522c.
• the "high-resolution color image” here refers to a color image acquired by the high-resolution imaging element
• the "high-sensitivity monochrome image” refers to a monochrome image acquired by the high-sensitivity imaging element.
• a normal light image which is a reflected image of visible light (white light)
• a third fluorescent image in which the third substance in the observation target S is highlighted, i.e., an image in which fluorescence is emitted from the third substance in the observation target S excited by narrowband light
• the user can compare the normal light image and the third fluorescent image of the observation target S via the display device 70, and can also observe a composite image (superimposed image) created from these images.
• the second imaging element 522b also outputs an image signal of a broadband light image frame, but in the above example, the broadband light image frame from the second imaging element 522b is not used to generate the captured image.
• image data of the broadband light image frame that is not used to generate the captured image is repeatedly output as an image signal from the second imaging element 522b, but image data of the broadband light image frame that is not used to generate the captured image does not have to be output as an image signal from the second imaging element 522b.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signal of a broadband light image frame from the second imaging element 522b that is not used to generate the captured image.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signals of the broadband light image frames from both the first imaging element 522a and the second imaging element 522b.
• the captured image generated in this manner may or may not be used to generate an output image.
• Captured images that are not used to generate the output image can be used for any purpose (for example, to correct the brightness of the output image or to adjust the focus position).
• the fifth mode of this embodiment is an observation mode in which broadband light and first to third narrowband light are irradiated onto the object of observation S, and a captured image is obtained based on the broadband reflected light Lw1 and the first to third fluorescence Lw4 from the object of observation S.
• control device 90 controls the light source device 10 (broadband light source 11 and first to third narrowband light sources 12 to 14), and the light source device 10 emits broadband light and first narrowband light in a time-division manner, and also emits second narrowband light and third narrowband light in a time-division manner.
• the broadband light and first narrowband light are irradiated to the observation object S in a time-division manner
• the second narrowband light and third narrowband light are irradiated to the observation object S in a time-division manner.
• each of the broadband light and the first narrowband light may be emitted simultaneously with one of the second narrowband light and the third narrowband light, and irradiated to the observation object S at the same time.
• the optical element 15 sequentially guides the first light beam Lf1 including the broadband reflected light Lw1 from the observation target S irradiated with the broadband light and the first light beam Lf1 including the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light to the first image sensor 522a.
• the optical element 15 also sequentially guides the second light beam Lf2 including the broadband reflected light Lw1 from the observation target S irradiated with the broadband light and the second light beam Lf2 including the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light to the second image sensor 522b.
• the optical element 15 also sequentially guides the third light beam Lf3 including the second fluorescence Lw3 from the observation target S irradiated with the second narrowband light and the third light beam Lf3 including the third fluorescence Lw4 from the observation target S irradiated with the third narrowband light to the third image sensor 522c.
• the first image sensor 522a sequentially receives the first light beam Lf1 including the broadband reflected light Lw1 and the first light beam Lf1 including the first fluorescent light Lw2. Then, under the control of the control device 90 (control unit 94), the first image sensor 522a repeatedly outputs an image signal based on the broadband reflected light Lw1 and an image signal based on the first fluorescent light Lw2.
• the second image sensor 522b sequentially receives the second light beam Lf2 including the broadband reflected light Lw1 and the second light beam Lf2 including the first fluorescent light Lw2. The second image sensor 522b then sequentially and repeatedly outputs an image signal based on the broadband reflected light Lw1 and an image signal based on the first fluorescent light Lw2 under the control of the control device 90 (control unit 94).
• the third image sensor 522c sequentially receives the third light beam Lf3 containing the second fluorescence Lw3 and the third light beam Lf3 containing the third fluorescence Lw4. The third image sensor 522c then sequentially and repeatedly outputs an image signal based on the second fluorescence Lw3 and an image signal based on the third fluorescence Lw4 under the control of the control device 90 (control unit 94).
• the image generation unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 from the image signal based on the broadband reflected light Lw1 output from the first imaging element 522a. Also, under the control of the control unit 94, the image generation unit 93 (image processing unit 931) generates a captured image (high-sensitivity monochrome image) of the observation target S based on the first fluorescence Lw2 from the image signal based on the first fluorescence Lw2 output from the second imaging element 522b.
• the "high-resolution color image” here refers to a color image acquired by the high-resolution imaging element
• the "high-sensitivity monochrome image” refers to a monochrome image acquired by the high-sensitivity imaging element.
• the image generation unit 93 (image processing unit 931) generates a captured image (high-sensitivity monochrome image) of the observation target S based on the second fluorescence Lw3 from an image signal based on the second fluorescence Lw3 output from the third imaging element 522c. Further, under the control of the control unit 94, the image generation unit 93 (image processing unit 931) generates a captured image (high-sensitivity monochrome image) of the observation target S based on the third fluorescence Lw4 from an image signal based on the third fluorescence Lw4 output from the third imaging element 522c.
• the "high-sensitivity monochrome image” referred to here is a monochrome image acquired by the high-sensitivity imaging element.
• a normal light image which is a reflected image of visible light (white light)
• first to third fluorescent images which are images in which the first to third substances in the observation target S are highlighted, i.e., images in which fluorescence is emitted from the first to third substances in the observation target S excited by narrowband light
• the user can compare and observe the normal light image, first fluorescent image, second fluorescent image, and third fluorescent image of the observation target S via the display device 70, and can also observe a composite image (superimposed image) created from these images.
• the first imaging element 522a also outputs an image signal of the first fluorescent image frame
• the second imaging element 522b also outputs an image signal of the broadband light image frame, but in the above example, these image frames are not used to generate the captured image.
• image data of these image frames that are not used to generate the captured image are also repeatedly output as image signals from the imaging elements (first imaging element 522a and second imaging element 522b), but image data of image frames that are not used to generate the captured image do not have to be output as image signals from the imaging elements.
• these image frames that are not used to generate the captured image may be used to generate the captured image.
• a captured image based on the first fluorescence Lw2 may be generated based on the image signal of the first fluorescence image frame from the first image sensor 522a.
• a captured image based on the first fluorescence Lw2 may be generated based on the image signals of the first fluorescence image frame from both the first image sensor 522a and the second image sensor 522b.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signal of the broadband light image frame from the second imaging element 522b.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signals of the broadband light image frame from both the first imaging element 522a and the second imaging element 522b.
• the captured image generated in this manner may or may not be used to generate an output image.
• Captured images that are not used to generate the output image can be used for any purpose (for example, to correct the brightness of the output image or to adjust the focus position).
• FIG. 24 is a diagram illustrating the types of light incident on the imaging elements (first imaging element 522a, second imaging element 522b, and third imaging element 522c) according to the ninth embodiment.
• the camera head 50 (particularly the imaging unit 52) of this embodiment is equipped with a three-plate type imaging module (see Figures 5 and 6), and is equipped with a first imaging element 522a having a color filter CF, and a second imaging element 522b and a third imaging element 522c that do not have a color filter CF.
• the color filter CF provided on the first imaging element 522a in this embodiment transmits the broadband reflected light Lw1 and the first fluorescence Lw2 received by the first imaging element 522a as described below, but may or may not transmit light of a wavelength band different from the broadband reflected light Lw1 and the first fluorescence Lw2 (e.g., the second fluorescence Lw3 and the third fluorescence Lw4).
• the second imaging element 522b and the third imaging element 522c in this example do not have a color filter CF, but each of the second imaging element 522b and the third imaging element 522c may have a color filter CF that can transmit the second fluorescence Lw3 and the third fluorescence Lw4.
• the first imaging element 522a has a relatively low sensitivity and a high resolution (e.g., 4K resolution) compared to the second imaging element 522b and the third imaging element 522c, while the second imaging element 522b and the third imaging element 522c have a relatively high sensitivity and a low resolution (e.g., HD resolution) compared to the first imaging element 522a.
• the second imaging element 522b and the third imaging element 522c may have the same characteristics as each other, or may have different characteristics from each other.
• the resolution and sensitivity of the first imaging element 522a to the third imaging element 522c are not limited to this, and the relationship between the resolution and sensitivity of the first imaging element 522a to the third imaging element 522c is not limited to this either.
• the light source device 10 emits at least one of the broadband light source 11, the first narrowband light source 12, the second narrowband light source 13, and the third narrowband light source 14, and can irradiate at least one of the broadband light, the first narrowband light, the second narrowband light, and the third narrowband light to the observation target S. Therefore, the observation light Lf from the observation target S can include broadband reflected light Lw1, which is reflected light of the broadband light, a first fluorescence Lw2 emitted from a first substance excited by the first narrowband light, a second fluorescence Lw3 emitted from a second substance excited by the second narrowband light, and a third fluorescence Lw4 emitted from a third substance excited by the third narrowband light.
• the broadband reflected light Lw1 and the first fluorescence Lw2 are light included in the first wavelength band
• the second fluorescence Lw3 and the third fluorescence Lw4 are light included in a second wavelength band outside the first wavelength band.
• the observation light Lf incident on the optical element 15 is separated by the optical element 15 into a first light beam Lf1, a second light beam Lf2, and a third light beam Lf3.
• the optical element 15 guides light included in the first wavelength band to the first image sensor 522a as the first light beam Lf1, guides light including the second fluorescence Lw3 to the second image sensor 522b as the second light beam Lf2, and guides light including the third fluorescence Lw4 to the third image sensor 522c as the third light beam Lf3.
• the first luminous flux Lf1 in which the light in the wavelength bands of the second fluorescence Lw3 and the third fluorescence Lw4 of the observation light Lf is partially, substantially or completely suppressed, is guided to the first imaging element 522a. That is, light including the broadband reflected light Lw1 and the first fluorescence Lw2 is guided to the first imaging element 522a as the first luminous flux Lf1.
• the second luminous flux Lf2 in which the light in the first wavelength band of the observation light Lf is partially, substantially or completely suppressed and the light in the wavelength band of the third fluorescence Lw4 is partially, substantially or completely suppressed, is guided to the second imaging element 522b.
• the third light beam Lf3 in which the light in the first wavelength band of the observation light Lf is partially, substantially, or completely suppressed and the light in the wavelength band of the second fluorescence Lw3 is partially, substantially, or completely suppressed, is guided to the third image sensor 522c. That is, light including at least the third fluorescence Lw4 is guided to the third image sensor 522c as the third light beam Lf3.
• the medical observation system 100 of this embodiment having the above-mentioned configuration can acquire various captured images of the observation subject S according to the following observation modes (modes 1 to 6).
• the first mode of this embodiment is an observation mode in which broadband light and first narrowband light are irradiated onto the observation target S, and a captured image is obtained based on the broadband reflected light Lw1 and the first fluorescent light Lw2 from the observation target S.
• control device 90 controls the light source device 10 (broadband light source 11 and first narrowband light source 12 (see FIG. 1A)), the light source device 10 emits broadband light and first narrowband light in a time-division manner, and the broadband light and first narrowband light are irradiated to the observation object S in a time-division manner.
• the second narrowband light source 13 and the third narrowband light source 14 are kept in an off state, and the light source device 10 does not emit the second narrowband light and the third narrowband light.
• the optical element 15 sequentially guides the first light beam Lf1 including the broadband reflected light Lw1 from the observation object S irradiated with the broadband light, and the first light beam Lf1 including the first fluorescence Lw2 from the observation object S irradiated with the first narrowband light, to the first image sensor 522a.
• the first image sensor 522a sequentially receives the first light beam Lf1 including the broadband reflected light Lw1 and the first light beam Lf1 including the first fluorescence Lw2.
• the first image sensor 522a repeatedly outputs an image signal based on the broadband reflected light Lw1 and an image signal based on the first fluorescent light Lw2.
• the image generation unit 93 (particularly the image processing unit 931 (see FIG. 7)) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 from the image signal based on the broadband reflected light Lw1 output from the first imaging element 522a. Also, under the control of the control unit 94, the image generation unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the first fluorescence Lw2 from the image signal based on the first fluorescence Lw2 output from the first imaging element 522a.
• the "high-resolution color image” referred to here is a color image acquired by the high-resolution imaging element.
• a normal light image which is a reflected image of visible light (white light)
• a first fluorescent image in which the first substance in the observation target S is emphasized, i.e., an image in which fluorescence is emitted from the first substance in the observation target S excited by narrowband light
• the user can compare the normal light image and the first fluorescent image of the observation target S via the display device 70 (see FIG. 1A), and can also observe a composite image (superimposed image) created from these images.
• the second mode of this embodiment is an observation mode in which the object S is irradiated with broadband light and second narrowband light, and an image is captured based on the broadband reflected light Lw1 and second fluorescent light Lw3 from the object S.
• control device 90 controls the light source device 10 (broadband light source 11 and second narrowband light source 13), and the broadband light and second narrowband light are continuously emitted from the light source device 10, and the broadband light and second narrowband light are continuously irradiated onto the observation object S.
• the first narrowband light source 12 and the third narrowband light source 14 are kept in an off state, and the first narrowband light and the third narrowband light are not emitted from the light source device 10.
• the optical element 15 continuously guides the first light beam Lf1, which includes the broadband reflected light Lw1 from the observation object S irradiated with the broadband light, to the first image sensor 522a.
• the optical element 15 also continuously guides the second light beam Lf2, which includes the second fluorescence Lw3 from the observation object S irradiated with the second narrowband light, to the second image sensor 522b.
• the optical element 15 also continuously guides the third light beam Lf3, which includes the second fluorescence Lw3 from the observation object S irradiated with the second narrowband light, to the third image sensor 522c.
• the first imaging element 522a continuously receives the first light beam Lf1 that includes the broadband reflected light Lw1.
• the second imaging element 522b continuously receives the second light beam Lf2 that includes the second fluorescent light Lw3
• the third imaging element 522c continuously receives the third light beam Lf3 that includes the second fluorescent light Lw3.
• the first image sensor 522a continuously and repeatedly outputs an image signal based on the broadband reflected light Lw1
• the second image sensor 522b and the third image sensor 522c continuously and repeatedly output an image signal based on the second fluorescent light Lw3.
• the image generation unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 from the image signal based on the broadband reflected light Lw1 output from the first imaging element 522a. Also, under the control of the control unit 94, the image generation unit 93 (image processing unit 931) generates a captured image (high-sensitivity monochrome image) of the observation target S based on the second fluorescence Lw3 from the image signal based on the second fluorescence Lw3 output from the second imaging element 522b.
• the "high-resolution color image” here refers to a color image acquired by the high-resolution imaging element
• the "high-sensitivity monochrome image” refers to a monochrome image acquired by the high-sensitivity imaging element.
• a normal light image which is a reflected image of visible light (white light)
• a second fluorescent image in which the second substance in the observation target S is emphasized, i.e., an image in which fluorescence is emitted from the second substance in the observation target S excited by narrowband light
• the user can compare the normal light image and the second fluorescent image of the observation target S via the display device 70, and can also observe a composite image (superimposed image) created from these images.
• the broadband light and the second narrowband light are constantly emitted from the light source device 10, but the light source device 10 may turn off the emission of each of the broadband light and the second narrowband light midway under the control of the control device 90 (control unit 94).
• control device 90 control unit 94
• the light source device 10 constantly emit the broadband light and the second narrowband light, it becomes possible to obtain a bright captured image, it is possible to suppress an increase in noise caused by gain adjustment, and it is possible to prevent a decrease in the effective frame rate.
• the third imaging element 522c also outputs an image signal of the second fluorescent image frame, but in the above example, the second fluorescent image frame from the third imaging element 522c is not used to generate the captured image.
• image data of the second fluorescent image frame that is not used to generate the captured image is repeatedly output as an image signal from the third imaging element 522c, but image data of the second fluorescent image frame that is not used to generate the captured image does not have to be output as an image signal from the third imaging element 522c.
• a captured image based on the second fluorescence Lw3 may be generated based on the image signal of the second fluorescence image frame from the third image sensor 522c that is not used to generate the captured image.
• a captured image based on the second fluorescence Lw3 may be generated based on the image signals of the second fluorescence image frame from both the second image sensor 522b and the third image sensor 522c.
• the captured image generated in this manner may or may not be used to generate an output image.
• Captured images that are not used to generate the output image can be used for any purpose (for example, to correct the brightness of the output image or to adjust the focus position).
• the third mode of this embodiment is an observation mode in which broadband light, first narrowband light, and second narrowband light are irradiated onto the object of observation S, and a captured image is obtained based on the broadband reflected light Lw1, first fluorescence Lw2, and second fluorescence Lw3 from the object of observation S.
• control device 90 controls the light source device 10 (broadband light source 11, first narrowband light source 12, and second narrowband light source 13), and the light source device 10 emits broadband light and first narrowband light in a time-division manner, and emits the second narrowband light continuously.
• the broadband light and first narrowband light are irradiated to the observation object S in a time-division manner, and the second narrowband light is irradiated to the observation object S continuously.
• the third narrowband light source 14 is kept in an off state, and the third narrowband light is not emitted from the light source device 10.
• the optical element 15 sequentially guides the first light beam Lf1 including the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, and the first light beam Lf1 including the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light, to the first imaging element 522a.
• the optical element 15 also continuously guides the second light beam Lf2 including the second fluorescence Lw3 from the observation target S irradiated with the second narrowband light, to the second imaging element 522b.
• the optical element 15 also continuously guides the third light beam Lf3 including the second fluorescence Lw3 from the observation target S irradiated with the second narrowband light, to the third imaging element 522c.
• the first imaging element 522a sequentially receives the first light beam Lf1 including the broadband reflected light Lw1 and the first light beam Lf1 including the first fluorescent light Lw2.
• the second imaging element 522b continuously receives the second light beam Lf2 including the second fluorescent light Lw3.
• the third imaging element 522c continuously receives the third light beam Lf3 including the second fluorescent light Lw3.
• the first image sensor 522a under the control of the control device 90 (controller 94), sequentially and repeatedly outputs an image signal based on the broadband reflected light Lw1 and an image signal based on the first fluorescence Lw2. Also, each of the second image sensor 522b and the third image sensor 522c, under the control of the control device 90 (controller 94), continuously and repeatedly outputs an image signal based on the second fluorescence Lw3.
• the image generating unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 from an image signal based on the broadband reflected light Lw1 output from the first imaging element 522a under the control of the control unit 94.
• the image generating unit 93 (image processing unit 931) also generates a captured image (high-resolution color image) of the observation target S based on the first fluorescence Lw2 from an image signal based on the first fluorescence Lw2 output from the first imaging element 522a under the control of the control unit 94.
• the image generating unit 93 (image processing unit 931) also generates a captured image (high-sensitivity monochrome image) of the observation target S based on the second fluorescence Lw3 from an image signal based on the second fluorescence Lw3 output from the second imaging element 522b under the control of the control unit 94.
• the "high-resolution color image” referred to here is a color image acquired by the high-resolution imaging element
• the "high-sensitivity monochrome image” is a monochrome image acquired by the high-sensitivity imaging element.
• a normal light image which is a reflected image of visible light (white light)
• a first fluorescent image in which a first substance in the observation target S is emphasized i.e., an image in which fluorescence is emitted from a first substance in the observation target S excited by narrow band light
• a second fluorescent image in which a second substance in the observation target S is emphasized i.e., an image in which fluorescence is emitted from a second substance in the observation target S excited by narrow band light
• the user can compare and observe the normal light image, first fluorescent image, and second fluorescent image of the observation target S via the display device 70, and can also observe a composite image (superimposed image) created from these images.
• the third imaging element 522c also outputs an image signal of the second fluorescent image frame, but in the above example, the second fluorescent image frame from the third imaging element 522c is not used to generate the captured image.
• image data of the second fluorescent image frame that is not used to generate the captured image is repeatedly output as an image signal from the third imaging element 522c, but image data of the second fluorescent image frame that is not used to generate the captured image does not have to be output as an image signal from the third imaging element 522c.
• a captured image based on the second fluorescence Lw3 may be generated based on the image signal of the second fluorescence image frame from the third image sensor 522c that is not used to generate the captured image.
• a captured image based on the second fluorescence Lw3 may be generated based on the image signals of the second fluorescence image frame from both the second image sensor 522b and the third image sensor 522c.
• the captured image generated in this manner may or may not be used to generate an output image.
• Captured images that are not used to generate the output image can be used for any purpose (for example, to correct the brightness of the output image or to adjust the focus position).
• the fourth mode of this embodiment is an observation mode in which the object S is irradiated with broadband light and third narrowband light, and an image is captured based on the broadband reflected light Lw1 and the third fluorescent light Lw4 from the object S.
• control device 90 controls the light source device 10 (broadband light source 11 and third narrowband light source 14), and the broadband light and third narrowband light are continuously emitted from the light source device 10, and the broadband light and third narrowband light are continuously irradiated onto the observation object S.
• the first narrowband light source 12 and the second narrowband light source 13 are kept in an off state, and the first narrowband light and the second narrowband light are not emitted from the light source device 10.
• the optical element 15 continuously guides the first light beam Lf1, which includes the broadband reflected light Lw1 from the observation object S irradiated with the broadband light, to the first image sensor 522a.
• the optical element 15 also continuously guides the second light beam Lf2, which includes the third fluorescence Lw4 from the observation object S irradiated with the third narrowband light, to the second image sensor 522b.
• the optical element 15 also continuously guides the third light beam Lf3, which includes the third fluorescence Lw4 from the observation object S irradiated with the third narrowband light, to the third image sensor 522c.
• the first image sensor 522a continuously receives the first light beam Lf1 including the broadband reflected light Lw1, and under the control of the control device 90 (controller 94), continuously and repeatedly outputs an image signal based on the broadband reflected light Lw1.
• the second image sensor 522b continuously receives the second light beam Lf2 including the third fluorescence Lw4, and under the control of the control device 90 (controller 94), continuously and repeatedly outputs an image signal based on the third fluorescence Lw4.
• the third image sensor 522c continuously receives the third light beam Lf3 including the third fluorescence Lw4, and under the control of the control device 90 (controller 94), continuously and repeatedly outputs an image signal based on the third fluorescence Lw4.
• the image generation unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 from the image signal based on the broadband reflected light Lw1 output from the first imaging element 522a. Also, under the control of the control unit 94, the image generation unit 93 (image processing unit 931) generates a captured image (high-sensitivity monochrome image) of the observation target S based on the third fluorescence Lw4 from the image signal based on the third fluorescence Lw4 output from the third imaging element 522c.
• the "high-resolution color image” here refers to a color image acquired by the high-resolution imaging element
• the "high-sensitivity monochrome image” refers to a monochrome image acquired by the high-sensitivity imaging element.
• a normal light image which is a reflected image of visible light (white light)
• a third fluorescent image in which the third substance in the observation target S is highlighted, i.e., an image in which fluorescence is emitted from the third substance in the observation target S excited by narrowband light
• the user can compare the normal light image and the third fluorescent image of the observation target S via the display device 70, and can also observe a composite image (superimposed image) created from these images.
• the second imaging element 522b also outputs an image signal of the third fluorescent image frame, but in the above example, the third fluorescent image frame from the second imaging element 522b is not used to generate the captured image.
• image data of the third fluorescent image frame that is not used to generate the captured image is repeatedly output as an image signal from the second imaging element 522b, but image data of the third fluorescent image frame that is not used to generate the captured image does not have to be output as an image signal from the second imaging element 522b.
• a captured image based on the third fluorescence Lw4 may be generated based on the image signal of the third fluorescence image frame from the second imaging element 522b that is not used to generate the captured image.
• a captured image based on the third fluorescence Lw4 may be generated based on the image signals of the third fluorescence image frame from both the second imaging element 522b and the third imaging element 522c.
• the captured image generated in this manner may or may not be used to generate an output image.
• Captured images that are not used to generate the output image can be used for any purpose (for example, to correct the brightness of the output image or to adjust the focus position).
• the fifth mode of this embodiment is an observation mode in which broadband light, second narrowband light, and third narrowband light are irradiated onto the object of observation S, and a captured image is obtained based on the broadband reflected light Lw1, the second fluorescence Lw3, and the third fluorescence Lw4 from the object of observation S.
• control device 90 controls the light source device 10 (broadband light source 11, second narrowband light source 13, and third narrowband light source 14), and broadband light is continuously emitted from the light source device 10, and the second narrowband light and the third narrowband light are emitted in a time-division manner.
• the broadband light is continuously irradiated onto the observation object S, and the second narrowband light and the third narrowband light are irradiated onto the observation object S in a time-division manner.
• the first narrowband light source 12 is kept in an off state, and the first narrowband light is not emitted from the light source device 10.
• the optical element 15 continuously guides the first light beam Lf1, which includes the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, to the first imaging element 522a.
• the optical element 15 also sequentially guides the second light beam Lf2, which includes the second fluorescence Lw3 from the observation target S irradiated with the second narrowband light, and the second light beam Lf2, which includes the third fluorescence Lw4 from the observation target S irradiated with the third narrowband light, to the second imaging element 522b.
• the optical element 15 also guides the third light beam Lf3, which includes the second fluorescence Lw3 from the observation target S irradiated with the second narrowband light, and the third light beam Lf3, which includes the third fluorescence Lw4 from the observation target S irradiated with the third narrowband light, to the third imaging element 522c.
• the first imaging element 522a continuously receives the first light beam Lf1 including the broadband reflected light Lw1.
• the second imaging element 522b sequentially receives the second light beam Lf2 including the second fluorescence Lw3 and the second light beam Lf2 including the third fluorescence Lw4.
• the third imaging element 522c sequentially receives the third light beam Lf3 including the second fluorescence Lw3 and the third light beam Lf3 including the third fluorescence Lw4.
• the first image sensor 522a repeatedly outputs an image signal based on the broadband reflected light Lw1 under the control of the control device 90 (controller 94).
• the second image sensor 522b and the third image sensor 522c each repeatedly outputs an image signal based on the second fluorescence Lw3 and an image signal based on the third fluorescence Lw4 under the control of the control device 90 (controller 94).
• the image generating unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 from the image signal based on the broadband reflected light Lw1 output from the first imaging element 522a under the control of the control unit 94.
• the image generating unit 93 (image processing unit 931) also generates a captured image (high-sensitivity monochrome image) of the observation target S based on the second fluorescence Lw3 from the image signal based on the second fluorescence Lw3 output from the second imaging element 522b under the control of the control unit 94.
• the image generating unit 93 (image processing unit 931) also generates a captured image (high-sensitivity monochrome image) of the observation target S based on the third fluorescence Lw4 from the image signal based on the third fluorescence Lw4 output from the third imaging element 522c under the control of the control unit 94.
• the "high-resolution color image” referred to here is a color image acquired by the high-resolution imaging element
• the "high-sensitivity monochrome image” is a monochrome image acquired by the high-sensitivity imaging element.
• FIG. 25 shows an example of a timing chart of light source emission and image sensor exposure in the fifth mode of the ninth embodiment.
• the vertical axis indicates the horizontal lines of the first imaging element 522a, the second imaging element 522b, and the third imaging element 522c, the horizontal axis indicates time, and line R1 indicates the timing of starting to read pixel data for each horizontal line for each image frame.
• the "broadband light image frame" between the lines R1 is an image frame for receiving (exposing) the broadband reflected light Lw1 from the observation target S.
• the “second fluorescence image frame” between the lines R1 is an image frame for receiving (exposing) the second fluorescence Lw3 from the observation target S.
• the “third fluorescence image frame” between the lines R1 is an image frame for receiving (exposing) the third fluorescence Lw4 from the observation target S.
• (b) shows the emission timing of broadband light from the broadband light source 11
• (d) shows the emission timing of the second narrowband light from the second narrowband light source 13
• (f) shows the emission timing of the third narrowband light from the third narrowband light source 14.
• the light source device 10 continuously emits broadband light and irradiates the observation object S.
• the light source device 10 emits the second narrowband light and the third narrowband light and irradiates the observation object S in a time-division manner.
• control device 90 controls so that the timing of the time-division emission of the second and third narrowband light from the light source device 10 and the timing of reading out the image data from the second and third imaging elements 522b and 522c are correlated with each other.
• the second narrowband light source 13 and the third narrowband light source 14 emit the second narrowband light and the third narrowband light in a time-division manner based on a common synchronization signal, and exposure and image data reading are performed at the second image sensor 522b and the third image sensor 522c. More specifically, the second narrowband light and the third narrowband light are emitted alternately and separately in time so that the second image sensor 522b and the third image sensor 522c are not simultaneously exposed to the second fluorescence Lw3 and the third fluorescence Lw4.
• image data is read out so that an image signal of the second fluorescence image frame in which exposure is performed using the second fluorescence Lw3 and an image signal of the third fluorescence image frame in which exposure is performed using the third fluorescence Lw4 are output from the second imaging element 522b and the third imaging element 522c.
• each of the second imaging element 522b and the third imaging element 522c repeatedly outputs an image signal of the second fluorescence image frame and an image signal of the third fluorescence image frame alternately.
• image data i.e., broadband light image frames
• the first image sensor 522a continuously and repeatedly outputs image signals of the broadband light image frames.
• the image generation unit 93 (image processing unit 931) generates a normal light captured image of the observation target S, which is a reflected image of broadband light (white light), from the image signal of the broadband light image frame output from the first imaging element 522a.
• the image generation unit 93 (image processing unit 931) also generates a second fluorescent captured image, which is a captured image based on the second fluorescence Lw3 of the observation target S, from the image signal of the second fluorescent image frame from the second imaging element 522b.
• the image generation unit 93 (image processing unit 931) also generates a third fluorescent captured image, which is an image of the third fluorescence Lw4 of the observation target S, from the image signal of the third fluorescent image frame from the third imaging element 522c.
• a normal light image of the observation target S in a common time frame, a normal light image of the observation target S, a second fluorescent image in which the second substance in the observation target S is emphasized, i.e., an image in which fluorescence is emitted from the second substance in the observation target S excited by the narrow band light, and a third fluorescent image in which the third substance in the observation target S is emphasized, i.e., an image in which fluorescence is emitted from the third substance in the observation target S excited by the narrow band light, are acquired. Therefore, the user can compare the normal light image, the second fluorescent image, and the third fluorescent image of the observation target S via the display device 70, and can also observe a composite image (superimposed image) created from these images.
• the emission of the second narrowband light and the third narrowband light is started and ended while image data is not being read out from the second imaging element 522b and the third imaging element 522c. Therefore, in the second fluorescence image frame of the second imaging element 522b and the third imaging element 522c, the third fluorescence Lw4 from the observation object S irradiated with the third narrowband light can be prevented from entering the second imaging element 522b. Also, in the third fluorescence image frame of the second imaging element 522b and the third imaging element 522c, the second fluorescence Lw3 from the observation object S irradiated with the second narrowband light can be prevented from entering the second imaging element 522b and the third imaging element 522c.
• timings for starting and ending the emission of the second narrowband light and the third narrowband light are not limited to the example shown in FIG. 25, and can be set to any timing.
• the emission of one or both of the second narrowband light and the third narrowband light may be started or ended.
• the second imaging element 522b also outputs an image signal of the third fluorescent image frame
• the third imaging element 522c also outputs an image signal of the second fluorescent image frame
• these image frames are not used to generate the captured image.
• image data of these image frames that are not used to generate the captured image are also repeatedly output as image signals from the imaging elements (the second imaging element 522b and the third imaging element 522c), but image data of image frames that are not used to generate the captured image do not have to be output as image signals from the imaging elements.
• these image frames that are not used to generate the captured image may be used to generate the captured image.
• a captured image based on the third fluorescence Lw4 may be generated based on the image signal of the third fluorescence image frame from the second imaging element 522b.
• a captured image based on the third fluorescence Lw4 may be generated based on the image signals of the third fluorescence image frame from both the second imaging element 522b and the third imaging element 522c.
• a captured image based on the second fluorescence Lw3 may be generated based on the image signal of the second fluorescence image frame from the third image sensor 522c.
• a captured image based on the second fluorescence Lw3 may be generated based on the image signal of the second fluorescence image frame from both the second image sensor 522b and the third image sensor 522c.
• the captured image generated in this manner may or may not be used to generate an output image.
• Captured images that are not used to generate the output image can be used for any purpose (for example, to correct the brightness of the output image or to adjust the focus position).
• the sixth mode of this embodiment is an observation mode in which broadband light and first to third narrowband light are irradiated onto the object of observation S, and a captured image is obtained based on the broadband reflected light Lw1 and the first to third fluorescence Lw4 from the object of observation S.
• control device 90 controls the light source device 10 (broadband light source 11 and first to third narrowband light sources 12 to 14), and the light source device 10 emits broadband light and first narrowband light in a time-division manner, and also emits second narrowband light and third narrowband light in a time-division manner.
• the broadband light and first narrowband light are irradiated to the observation object S in a time-division manner
• the second narrowband light and third narrowband light are irradiated to the observation object S in a time-division manner.
• each of the broadband light and the first narrowband light may be emitted simultaneously with one of the second narrowband light and the third narrowband light, and irradiated to the observation object S at the same time.
• the optical element 15 sequentially guides the first light beam Lf1 including the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, and the first light beam Lf1 including the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light, to the first imaging element 522a.
• the optical element 15 also sequentially guides the second light beam Lf2 including the second fluorescence Lw3 from the observation target S irradiated with the second narrowband light, and the second light beam Lf2 including the third fluorescence Lw4 from the observation target S irradiated with the third narrowband light, to the second imaging element 522b.
• the optical element 15 also sequentially guides the third light beam Lf3 including the second fluorescence Lw3 from the observation target S irradiated with the second narrowband light, and the third light beam Lf3 including the third fluorescence Lw4 from the observation target S irradiated with the third narrowband light, to the third imaging element 522c.
• the first image sensor 522a sequentially receives the first light beam Lf1 including the broadband reflected light Lw1 and the first light beam Lf1 including the first fluorescent light Lw2. Then, under the control of the control device 90 (control unit 94), the first image sensor 522a repeatedly outputs an image signal based on the broadband reflected light Lw1 and an image signal based on the first fluorescent light Lw2.
• the second imaging element 522b sequentially receives the second light beam Lf2 containing the second fluorescence Lw3 and the second light beam Lf2 containing the third fluorescence Lw4.
• the third imaging element 522c sequentially receives the third light beam Lf3 containing the second fluorescence Lw3 and the third light beam Lf3 containing the third fluorescence Lw4.
• Each of the second imaging element 522b and the third imaging element 522c sequentially and repeatedly outputs an image signal based on the second fluorescence Lw3 and an image signal based on the third fluorescence Lw4 under the control of the control device 90 (control unit 94).
• the image generation unit 93 (image processing unit 931), under the control of the control unit 94, generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 from the image signal based on the broadband reflected light Lw1 output from the first imaging element 522a.
• the image generation unit 93 (image processing unit 931), under the control of the control unit 94, generates a captured image (high-resolution color image) of the observation target S based on the first fluorescence Lw2 from the image signal based on the first fluorescence Lw2 output from the first imaging element 522a.
• the image generation unit 93 (image processing unit 931), under the control of the control unit 94, generates a captured image (high-sensitivity monochrome image) of the observation target S based on the second fluorescence Lw3 from the image signal based on the second fluorescence Lw3 output from the second imaging element 522b. Furthermore, under the control of the control unit 94, the image generation unit 93 (image processing unit 931) generates a captured image (high-sensitivity monochrome image) of the observation target S based on the third fluorescence Lw4 from an image signal based on the third fluorescence Lw4 output from the third imaging element 522c.
• the "high-resolution color image” here refers to a color image acquired by the high-resolution imaging element
• the "high-sensitivity monochrome image” refers to a monochrome image acquired by the high-sensitivity imaging element.
• a normal light image which is a reflected image of visible light (white light)
• first to third fluorescent images which are images in which the first to third substances in the observation target S are highlighted, i.e., images in which fluorescence is emitted from the first to third substances in the observation target S excited by narrowband light
• the user can compare and observe the normal light image, first fluorescent image, second fluorescent image, and third fluorescent image of the observation target S via the display device 70, and can also observe a composite image (superimposed image) created from these images.
• the second imaging element 522b also outputs an image signal of the third fluorescent image frame
• the third imaging element 522c also outputs an image signal of the second fluorescent image frame
• these image frames are not used to generate the captured image.
• image data of these image frames that are not used to generate the captured image are also repeatedly output as image signals from the imaging elements (the second imaging element 522b and the third imaging element 522c), but image data of image frames that are not used to generate the captured image do not have to be output as image signals from the imaging elements.
• these image frames that are not used to generate the captured image may be used to generate the captured image.
• a captured image based on the third fluorescence Lw4 may be generated based on the image signal of the third fluorescence image frame from the second imaging element 522b.
• a captured image based on the third fluorescence Lw4 may be generated based on the image signals of the third fluorescence image frame from both the second imaging element 522b and the third imaging element 522c.
• a captured image based on the second fluorescence Lw3 may be generated based on the image signal of the second fluorescence image frame from the third image sensor 522c.
• a captured image based on the second fluorescence Lw3 may be generated based on the image signal of the second fluorescence image frame from both the second image sensor 522b and the third image sensor 522c.
• the captured image generated in this manner may or may not be used to generate an output image.
• Captured images that are not used to generate the output image can be used for any purpose (for example, to correct the brightness of the output image or to adjust the focus position).
• FIG. 26 is a diagram explaining the types of light incident on the imaging elements (first imaging element 522a, second imaging element 522b, and third imaging element 522c) according to the tenth embodiment.
• the camera head 50 (particularly the imaging unit 52) of this embodiment is equipped with a three-plate type imaging module (see Figures 5 and 6), and is equipped with a first imaging element 522a and a second imaging element 522b that have a color filter CF, and a third imaging element 522c that does not have a color filter CF.
• the color filters CF provided on each of the first and second imaging elements 522a and 522b in this embodiment transmit the broadband reflected light Lw1 and the first fluorescence Lw2 received by each of the first and second imaging elements 522a and 522b as described below, but may or may not transmit light of a wavelength band different from the broadband reflected light Lw1 and the first fluorescence Lw2 (e.g., the second fluorescence Lw3).
• the third imaging element 522c in this example does not have a color filter CF, but the third imaging element 522c may have a color filter CF that can transmit the second fluorescence Lw3.
• the first imaging element 522a has a relatively low sensitivity and a high resolution (e.g., 4K resolution) compared to the second imaging element 522b and the third imaging element 522c, while the second imaging element 522b and the third imaging element 522c have a relatively high sensitivity and a low resolution (e.g., HD resolution) compared to the first imaging element 522a.
• the second imaging element 522b and the third imaging element 522c may have the same characteristics as each other, or may have different characteristics from each other.
• the resolution and sensitivity of the first imaging element 522a to the third imaging element 522c are not limited to this, and the relationship between the resolution and sensitivity of the first imaging element 522a to the third imaging element 522c is not limited to this either.
• the light source device 10 (see FIG. 1A) emits light from at least one of a broadband light source 11, a first narrowband light source 12, and a second narrowband light source 13, and can irradiate the object of observation S with at least one of the broadband light, the first narrowband light, and the second narrowband light.
• the observation light Lf from the observation target S may include broadband reflected light Lw1, which is reflected light of broadband light, a first fluorescence Lw2 emitted from a first substance excited by the first narrowband light, and a second fluorescence Lw3 emitted from a second substance excited by the second narrowband light.
• the broadband reflected light Lw1 and the first fluorescence Lw2 are light contained in a first wavelength band
• the second fluorescence Lw3 is light contained in a second wavelength band outside the first wavelength band.
• the observation light Lf incident on the optical element 15 is separated into a first light beam Lf1, a second light beam Lf2, and a third light beam Lf3 by the optical element 15.
• the optical element 15 guides a portion of the light included in the first wavelength band to the first image sensor 522a as the first light beam Lf1, guides a portion of the light included in the first wavelength band to the second image sensor 522b as the second light beam Lf2, and guides the light included in the second wavelength band to the third image sensor 522c as the third light beam Lf3.
• the first light beam Lf1 in which the light in the first wavelength band of the observation light Lf is partially suppressed and the light in the wavelength band of the second fluorescence Lw3 is partially, substantially or completely suppressed, is guided to the first imaging element 522a by the optical element 15. That is, light including at least a part of the broadband reflected light Lw1 is guided to the first imaging element 522a as the first light beam Lf1.
• the second light beam Lf2 in which the light in the first wavelength band of the observation light Lf is partially suppressed and the light in the wavelength band of the second fluorescence Lw3 is partially, substantially or completely suppressed, is guided to the second imaging element 522b.
• the third light beam Lf3 in which the light in the first wavelength band of the observation light Lf is partially, substantially or completely suppressed, is guided to the third imaging element 522c. That is, light that includes at least the second fluorescent light Lw3 is guided to the third image sensor 522c as the third light beam Lf3.
• the medical observation system 100 of this embodiment having the above-mentioned configuration can acquire various captured images of the observation subject S according to the following observation modes (first to third modes).
• the first mode of this embodiment is an observation mode in which broadband light and first narrowband light are irradiated onto the observation target S, and a captured image is obtained based on the broadband reflected light Lw1 and the first fluorescent light Lw2 from the observation target S.
• control device 90 controls the light source device 10 (broadband light source 11 and first narrowband light source 12 (see FIG. 1A)), and the broadband light and first narrowband light are emitted from the light source device 10 in a time-division manner, and the broadband light and first narrowband light are irradiated onto the observation object S in a time-division manner.
• the second narrowband light source 13 is kept in an off state, and the second narrowband light is not emitted from the light source device 10.
• the optical element 15 sequentially guides the first light beam Lf1 including the broadband reflected light Lw1 from the observation object S irradiated with the broadband light, and the first light beam Lf1 including the first fluorescence Lw2 from the observation object S irradiated with the first narrowband light, to the first image sensor 522a.
• the first image sensor 522a sequentially receives the first light beam Lf1 including the broadband reflected light Lw1 and the first light beam Lf1 including the first fluorescence Lw2.
• the optical element 15 also sequentially guides the second light beam Lf2 including the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, and the second light beam Lf2 including the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light, to the second image sensor 522b.
• the second image sensor 522b sequentially receives the second light beam Lf2 including the broadband reflected light Lw1 and the second light beam Lf2 including the first fluorescence Lw2.
• each of the first image sensor 522a and the second image sensor 522b sequentially and repeatedly outputs an image signal based on the broadband reflected light Lw1 and an image signal based on the first fluorescent light Lw2 under the control of the control device 90 (control unit 94 (see FIG. 7)).
• the image generation unit 93 (particularly the image processing unit 931 (see FIG. 7)) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 from the image signal based on the broadband reflected light Lw1 output from the first imaging element 522a. Also, under the control of the control unit 94, the image generation unit 93 (image processing unit 931) generates a captured image (high-sensitivity color image) of the observation target S based on the first fluorescence Lw2 from the image signal based on the first fluorescence Lw2 output from the second imaging element 522b.
• the "high-resolution color image” referred to here is a color image acquired by the high-resolution imaging element
• the "high-sensitivity color image” is a color image acquired by the high-sensitivity imaging element.
• a normal light image which is a reflected image of visible light (white light)
• a first fluorescent image in which the first substance in the observation target S is emphasized, i.e., an image in which fluorescence is emitted from the first substance in the observation target S excited by narrowband light
• the user can compare the normal light image and the first fluorescent image of the observation target S via the display device 70 (see FIG. 1A), and can also observe a composite image (superimposed image) created from these images.
• the image signal output from the first image sensor 522a is used to generate the captured image based on the broadband reflected light Lw1, but the image signal output from the second image sensor 522b may also be used.
• control unit 94 may control the imaging unit 52 (imaging element 522) and the image generation unit 93 so that an image based on the broadband reflected light Lw1 is generated based on one or both of the image signal output from the first imaging element 522a that receives the first light beam Lf1 including the broadband reflected light Lw1 and the image signal output from the second imaging element 522b that receives the second light beam Lf2 including the broadband reflected light Lw1.
• a captured image (high-resolution color image) based on the broadband reflected light Lw1 may be generated based on the image signal output from the first imaging element 522a.
• a captured image (high-sensitivity color image) based on the broadband reflected light Lw1 may be generated based on the image signal output from the second imaging element 522b.
• a captured image based on the broadband reflected light Lw1 may be generated based on the image signal output from the first imaging element 522a and the image signal output from the second imaging element 522b.
• the "high-resolution color image” referred to here is a color image acquired by the high-resolution imaging element
• the "high-sensitivity color image” is a color image acquired by the high-sensitivity imaging element.
• the control unit 94 may determine, based on instructions from a user received via the input unit 95, whether to use either or both of the image signal output from the first imaging element 522a and the image signal output from the second imaging element 522b to generate an image based on the broadband reflected light Lw1.
• the image signal output from the second image sensor 522b is used to generate the captured image based on the first fluorescence Lw2, but the image signal output from the first image sensor 522a may also be used. That is, the control unit 94 may control the image sensor 52 (image sensor 522) and the image generator 93 so that an image based on the first fluorescence Lw2 is generated based on one or both of the image signal output from the first image sensor 522a that receives the first light beam Lf1 including the first fluorescence Lw2 and the image signal output from the second image sensor 522b that receives the second light beam Lf2 including the first fluorescence Lw2.
• the control unit 94 may determine, based on an instruction from a user received via the input unit 95, whether to use either or both of the image signal output from the first imaging element 522a and the image signal output from the second imaging element 522b to generate an image based on the first fluorescence Lw2.
• the second mode of this embodiment is an observation mode in which the object S is irradiated with broadband light and second narrowband light, and an image is captured based on the broadband reflected light Lw1 and second fluorescent light Lw3 from the object S.
• control device 90 controls the light source device 10 (broadband light source 11 and second narrowband light source 13), and the broadband light and second narrowband light are continuously emitted from the light source device 10, and the broadband light and second narrowband light are continuously irradiated onto the observation object S.
• the first narrowband light source 12 is kept in an off state, and the first narrowband light is not emitted from the light source device 10.
• the optical element 15 continuously guides the first light beam Lf1, which includes the broadband reflected light Lw1 from the observation object S irradiated with the broadband light, to the first image sensor 522a, and continuously guides the second light beam Lf2, which includes the broadband reflected light Lw1 from the observation object S irradiated with the broadband light, to the second image sensor 522b.
• the optical element 15 also continuously guides the third light beam Lf3, which includes the second fluorescence Lw3 from the observation object S irradiated with the second narrowband light, to the third image sensor 522c.
• the first imaging element 522a continuously receives the first light beam Lf1 including the broadband reflected light Lw1
• the second imaging element 522b continuously receives the second light beam Lf2 including the broadband reflected light Lw1
• the third imaging element 522c continuously receives the third light beam Lf3 including the second fluorescent light Lw3.
• the first image sensor 522a and the second image sensor 522b continuously and repeatedly output image signals based on the broadband reflected light Lw1, and the third image sensor 522c continuously and repeatedly output image signals based on the second fluorescent light Lw3.
• the image generation unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1 from the image signal based on the broadband reflected light Lw1 output from the first imaging element 522a. Also, under the control of the control unit 94, the image generation unit 93 (image processing unit 931) generates a captured image (high-sensitivity monochrome image) of the observation target S based on the second fluorescence Lw3 from the image signal based on the second fluorescence Lw3 output from the third imaging element 522c.
• the "high-resolution color image” here refers to a color image acquired by the high-resolution imaging element
• the "high-sensitivity monochrome image” refers to a monochrome image acquired by the high-sensitivity imaging element.
• a normal light image which is a reflected image of visible light (white light)
• a second fluorescent image in which the second substance in the observation target S is emphasized, i.e., an image in which fluorescence is emitted from the second substance in the observation target S excited by narrowband light
• the user can compare the normal light image and the second fluorescent image of the observation target S via the display device 70, and can also observe a composite image (superimposed image) created from these images.
• the broadband light and the second narrowband light are constantly emitted from the light source device 10, but the light source device 10 may turn off the emission of each of the broadband light and the second narrowband light midway under the control of the control device 90 (control unit 94).
• the light source device 10 may repeatedly turn on and off the emission of the broadband light and the second narrowband light, or may emit the broadband light and the second narrowband light in a time-division manner.
• the image signal output from the first imaging element 522a is used to generate the captured image based on the broadband reflected light Lw1, but the image signal output from the second imaging element 522b may also be used. That is, the control unit 94 may control the imaging unit 52 (imaging element 522) and the image generation unit 93 so that an image based on the broadband reflected light Lw1 is generated based on one or both of the image signal output from the first imaging element 522a that receives the first light beam Lf1 including the broadband reflected light Lw1 and the image signal output from the second imaging element 522b that receives the second light beam Lf2 including the broadband reflected light Lw1.
• the control unit 94 may determine, based on instructions from a user received via the input unit 95, whether to use either or both of the image signal output from the first imaging element 522a and the image signal output from the second imaging element 522b to generate an image based on the broadband reflected light Lw1.
• the third mode of this embodiment is an observation mode in which broadband light, first narrowband light, and second narrowband light are irradiated onto the object of observation S, and a captured image is obtained based on the broadband reflected light Lw1, first fluorescence Lw2, and second fluorescence Lw3 from the object of observation S.
• control device 90 controls the light source device 10 (broadband light source 11, first narrowband light source 12, and second narrowband light source 13), and the light source device 10 emits broadband light and first narrowband light in a time-division manner, and emits the second narrowband light continuously.
• the broadband light and first narrowband light are irradiated to the observation object S in a time-division manner, and the second narrowband light is irradiated to the observation object S continuously.
• the optical element 15 sequentially guides the first light beam Lf1 including the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, and the first light beam Lf1 including the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light, to the first image sensor 522a.
• the optical element 15 also sequentially guides the second light beam Lf2 including the broadband reflected light Lw1 from the observation target S irradiated with the broadband light, and the second light beam Lf2 including the first fluorescence Lw2 from the observation target S irradiated with the first narrowband light, to the second image sensor 522b.
• the optical element 15 also continuously guides the third light beam Lf3 including the second fluorescence Lw3 from the observation target S irradiated with the second narrowband light, to the third image sensor 522c.
• the first imaging element 522a sequentially receives the first light beam Lf1 including the broadband reflected light Lw1 and the first light beam Lf1 including the first fluorescence Lw2.
• the second imaging element 522b sequentially receives the second light beam Lf2 including the broadband reflected light Lw1 and the second light beam Lf2 including the first fluorescence Lw2.
• the third imaging element 522c continuously receives the third light beam Lf3 including the second fluorescence Lw3.
• the first and second image capturing elements 522a and 522b under the control of the control device 90 (controller 94), sequentially and repeatedly output an image signal based on the broadband reflected light Lw1 and an image signal based on the first fluorescence Lw2.
• the third image capturing element 522c under the control of the control device 90 (controller 94), continuously and repeatedly outputs an image signal based on the second fluorescence Lw3.
• the image generating unit 93 (image processing unit 931) generates a captured image (high-resolution color image) of the observation target S based on the broadband reflected light Lw1, based on an image signal output from the first image sensor 522a that has received the first light beam Lf1 including the broadband reflected light Lw1, under the control of the control unit 94.
• the image generating unit 93 (image processing unit 931) also generates a captured image (high-sensitivity color image) of the observation target S based on the first fluorescence Lw2, based on an image signal output from the second image sensor 522b that has received the second light beam Lf2 including the first fluorescence Lw2, under the control of the control unit 94.
• the image generating unit 93 (image processing unit 931) also generates a captured image (high-sensitivity image) of the observation target S based on the second fluorescence Lw3, based on an image signal output from the third image sensor 522c that has received the third light beam Lf3 including the second fluorescence Lw3, under the control of the control unit 94.
• a "high-resolution color image” is a color image captured by a high-resolution imaging element
• a "high-sensitivity color image” is a color image captured by a high-sensitivity imaging element
• a "high-sensitivity image” is an image captured by a high-sensitivity imaging element.
• a normal light image which is a reflected image of visible light (white light)
• a first fluorescent image and a second fluorescent image which are images in which the first and second substances in the observation target S are highlighted, i.e., images in which fluorescence is emitted from the first and second substances in the observation target S excited by narrowband light
• the user can compare the normal light image, the first fluorescent image, and the second fluorescent image of the observation target S, and can also observe a composite image (superimposed image) created from these images.
• the image signal output from the first imaging element 522a is used to generate the captured image based on the broadband reflected light Lw1, but the image signal output from the second imaging element 522b may also be used. That is, the control unit 94 may control the imaging unit 52 (imaging element 522) and the image generation unit 93 so that an image based on the broadband reflected light Lw1 is generated based on one or both of the image signal output from the first imaging element 522a that receives the first light beam Lf1 including the broadband reflected light Lw1 and the image signal output from the second imaging element 522b that receives the second light beam Lf2 including the broadband reflected light Lw1.
• the control unit 94 may determine, based on instructions from a user received via the input unit 95, whether to use either or both of the image signal output from the first imaging element 522a and the image signal output from the second imaging element 522b to generate an image based on the broadband reflected light Lw1.
• the image signal output from the second image sensor 522b is used to generate the captured image based on the first fluorescence Lw2, but the image signal output from the first image sensor 522a may also be used. That is, the control unit 94 may control the image sensor 52 (image sensor 522) and the image generator 93 so that an image based on the first fluorescence Lw2 is generated based on one or both of the image signal output from the first image sensor 522a that receives the first light beam Lf1 including the first fluorescence Lw2 and the image signal output from the second image sensor 522b that receives the second light beam Lf2 including the first fluorescence Lw2.
• the control unit 94 may determine, based on an instruction from a user received via the input unit 95, whether to use either or both of the image signal output from the first imaging element 522a and the image signal output from the second imaging element 522b to generate an image based on the first fluorescence Lw2.
• the medical observation method described below can be implemented by any medical observation system 100 and control device 90 (image generating device). Therefore, the medical observation method described below may be implemented by a medical observation system 100 and control device 90 based on each of the above-mentioned embodiments and modifications, or may be implemented by a medical observation system 100 and control device 90 different from each of the above-mentioned embodiments and modifications.
• FIG. 27 is a flowchart showing an example of a medical observation method performed by the medical observation system 100.
• the light irradiated to the object of observation may include visible light (e.g., broadband light such as white light) and excitation light (narrowband light in the visible light wavelength band or narrowband light in the invisible light wavelength band).
• the multiple captured images may also include images based on visible light reflected by the object of observation and images based on fluorescence from the object of observation.
• the light source device 10 may emit a plurality of types of light
• the imaging element 522 of the camera head 50 may receive (expose) a plurality of types of light intermittently, continuously (successively), in a time-division manner, or simultaneously.
• the image generation unit 93 generates an output image based on the multiple captured images (S3), and the output image is displayed on the display device 70 (S4).
• a user such as a doctor can view the output image displayed on the display device 70 and confirm the state and characteristics of the tissue of the subject being observed.
• the above-mentioned series of processes continues repeatedly as long as the user does not instruct the process to end (N in S5), and ends when the user instructs the process to end (Y in S5).
• the control unit 94 can receive such an instruction to end the process from the user via the input unit 95 (see FIG. 7).
• high-resolution imaging may be required to provide a highly detailed image.
• high-sensitivity imaging since the intensity of the fluorescence from the object being observed is not necessarily strong, high-sensitivity imaging may be required to capture an image based on the fluorescence. Both high-resolution imaging and high-sensitivity imaging can be achieved by using a high-resolution, high-sensitivity imaging element, but such high-performance imaging elements are expensive, and it may be difficult to prepare such high-performance imaging elements.
• the observation light from the subject may be directed to a high-sensitivity imaging element or a high-resolution imaging element based on the wavelength band. That is, by directing observation light of a certain wavelength band (e.g., a light beam including reflected white light) to a high-resolution imaging element and directing observation light of another wavelength band (e.g., a light beam including fluorescence) to a high-sensitivity imaging element, the desired high-resolution imaging and high-sensitivity imaging can be performed.
• a certain wavelength band e.g., a light beam including reflected white light
• another wavelength band e.g., a light beam including fluorescence
• the sensor sizes of the imaging elements may differ.
• the range (angle of view) of the observation object captured in the captured image may differ between the multiple imaging elements, and the number of pixels (size) of the captured image may differ between the imaging elements.
• the user When such captured images are displayed while being switched between on a single display device, the user must mentally align the field of view between the displayed images. Observing the observation object while matching or associating the position and range of the observation object (for example, an identification object such as a lesion) between the displayed images by aligning the field of view between the displayed images is a cumbersome task that places a burden on the user and may hinder accurate recognition of the identification object.
• multiple captured images with different angles of view may contain not only an area of the object of observation that is included in common to the multiple captured images, but also an area of the object of observation that is included in one captured image but not in the other captured images.
• an area of the object of observation that is included in the normal light captured image may not be included in the fluorescent image.
• a superimposed image generated by superimposing and synthesizing such a normal light captured image and a fluorescent image an area of the object of observation that is included only in the normal light captured image is represented as the normal light captured image, but not as the fluorescent image.
• an area of the object of observation that is included only in the normal light captured image can be recognized as the normal light captured image by a user such as a doctor in the superimposed image.
• an area of the object of observation that is included only in the normal light captured image is not included in the fluorescent image, so even if the area contains a specific substance and emits fluorescence, the fluorescent image of the area is not included in the superimposed image. Therefore, when a user looks at the superimposed image, even if the area of the object of observation, which is only included in the normal light image, emits fluorescence, the user may mistakenly recognize the area as "tissue that does not emit fluorescence (i.e. tissue that does not contain a specific substance)," which may result in a misdiagnosis.
• the superimposed image may contain not only the area of the object of observation that is included in all captured images in common (common image area), but also areas of the object of observation that are included in some captured images but not in others (non-common image area). It is not necessarily easy for users, such as doctors, to accurately distinguish between such common image areas and non-common image areas in the superimposed images.
• the output image generated based on multiple captured images be generated in a way that is advantageous for the user to accurately recognize the image of the area to be observed that is commonly captured in the multiple captured images.
• image generation method Next, a specific example of an image generating method for generating an output image will be described.
• the image generating method described below can be implemented in any medical observation system 100, control device 90 (image generating device), and medical observation method. Therefore, the image generating method described below may be implemented in the medical observation system 100, control device 90, and medical observation method based on each of the above-mentioned embodiments and modifications, or the image generating method described below may be implemented in a medical observation system 100, control device 90, and medical observation method different from each of the above-mentioned embodiments and modifications.
• endoscopic devices are used to observe subjects to assist in surgery. Endoscopic devices can capture images of subjects under illumination with white light to obtain photographed images of the subject.
• fluorescent observation is effective. For example, by irradiating excitation light onto a subject administered a fluorescent drug and observing the fluorescence emitted from the subject, it is possible to visually grasp the characteristics and condition of the subject tissue.
• the superimposed image obtained by combining the white light image and the fluorescent image makes it possible to simultaneously visualize various characteristics and conditions of the subject.
• FIG. 28 shows an example of the physical size relationship between the imaging area 410 (effective pixel area) of the first imaging element 522a and the imaging area 420 (effective pixel area) of the second imaging element 522b.
• the rectangle shown by the two-dot chain line superimposed on the imaging area 410 (solid line) of the first imaging element 522a indicates the range corresponding to the imaging area 420 of the second imaging element 522b shown in FIG. 28.
• a first image is generated based on the image signal (pixel data) output from the imaging area 410 of the first imaging element 522a
• a second image is generated based on the image signal output from the imaging area 420 of the second imaging element 522b.
• the first imaging element 522a and the second imaging element 522b are different types of imaging elements, and the imaging area 410 of the first imaging element 522a is physically larger in size than the imaging area 420 of the second imaging element 522b.
• the angle of view of the first imaging element 522a is wider than the angle of view of the second imaging element 522b, and the range of the observation object imaged by the first imaging element 522a is wider than the range of the observation object imaged by the second imaging element 522b.
• the entire range of the observation object imaged by the second imaging element 522b is included in the range of the observation object imaged by the first imaging element 522a. Therefore, the observation object captured in the first image includes the entire range of the observation object captured in the second image.
• the first image is an image of a first range of the observation target captured by the first imaging element 522a.
• the second image is an image of a second range of the observation target that is different from the first range, captured by the second imaging element 522b of a different type from the first imaging element 522a.
• one of the first range and the second range is wider than the other, and in the example shown in FIG. 28, the second range is included in the first range.
• the "type" referred to here is, for example, the physical size, image size, or model number of the imaging element, but is not limited to these.
• an output image is generated based on the first image and the second image by an image generation method performed by the control device 90 (image generation device) as described later.
• the output image generated in this way is, for example, an image in which one or more pixel signals of the first image and the second image are replaced or deleted based on the first range and the second range of the observation target.
• the imaging area 410 of the first imaging element 522a includes a common light receiving area 410A that receives light from the common portion of the observation object, and a non-common light receiving area 410B that receives light from the non-common object portion of the observation object.
• the imaging area 420 of the second imaging element 522b includes only the common light receiving area 410A that receives light from the common portion of the observation object.
• the common light receiving area 410A of the first imaging element 522a is an area that optically corresponds to the entire imaging area 420 of the second imaging element 522b, and the same part of the observation object imaged by the second imaging element 522b is imaged in the common light receiving area 410A.
• the non-common light receiving area 410B of the first imaging element 522a is an area that does not optically correspond to the imaging area 420 of the second imaging element 522b, and the part of the observation object not imaged by the second imaging element 522b is imaged in the non-common light receiving area 410B.
• the first imaging element 522a and the second imaging element 522b respectively acquire a first image and a second image in which the same observation object is captured but the range of the captured observation object is different from each other.
• the number of pixels P1 in the first imaging element 522a is greater than the number of pixels P2 in the second imaging element 522b (imaging region 420).
• the number of pixels in the first imaging element 522a may be 3840 x 2160, which corresponds to 4K resolution
• the number of pixels in the second imaging element 522b may be 1920 x 1080, which corresponds to full HD (High Definition).
• the number of pixels of the first imaging element 522a and the number of pixels of the second imaging element 522b are not limited.
• the number of pixels of an imaging element is related not only to the physical size of the imaging area but also to the size of each pixel (i.e., pixel size). Therefore, the number of pixels of the first imaging element 522a may be the same as the number of pixels of the second imaging element 522b, or may be less than the number of pixels of the second imaging element 522b. In this way, the relationship in the number of pixels between the first imaging element 522a and the second imaging element 522b is not limited.
• FIG. 29 is a schematic diagram showing an example of a first image 411 generated based on an image signal from the first image sensor 522a shown in FIG. 28.
• the first image 411 shown in FIG. 29 has a common image area 411A and a non-common image area 411B adjacent to the common image area 411A.
• An image based on an image signal output from the common light receiving area 410A (see FIG. 28) of the first imaging element 522a is projected onto the common image area 411A, and an image based on an image signal output from the non-common light receiving area 410B (see FIG. 28) is projected onto the non-common image area 411B.
• the common image area 411A shows an image of the part of the object being observed that is captured by both the first imaging element 522a and the second imaging element 522b, and that is captured in the entire second image captured and acquired by the second imaging element 522b.
• the non-common image area 411B shows an image of the part of the object being observed that is not captured in the second image.
• the non-common image area 411B surrounds the entire common image area 411A (top, bottom, left, right), but the range of the non-common image area 411B is not limited, and the non-common image area 411B does not necessarily surround the entire common image area 411A. Furthermore, it is not necessarily required to identify the specific ranges of the common image area 411A and the non-common image area 411B in the first image 411. Therefore, the image generation unit 93 (e.g., the superimposed image generation unit 934) may or may not perform a process to identify the ranges of the common image area 411A and the non-common image area 411B in the first image 411.
• the first image sensor 522a captures an image (normal light captured image; first image) by capturing an image of an object being observed irradiated with visible light (particularly white light; first light) in a first wavelength band from a broadband light source (first light source) 11.
• the second image sensor 522b captures an image (fluorescence captured image; second image) by capturing an image of an object being observed irradiated with excitation light (second light) from a first narrowband light source (second light source) 12, which excites a specific substance to emit visible fluorescence or invisible fluorescence.
• the excitation light is light of a wavelength band at least partially different from the first wavelength band.
• the first image captured by the first image sensor 522a includes an image based on the reflected light of visible light (white light) from the object being observed.
• the second image acquired by the second image sensor 522b includes a captured image based on fluorescence from the object of observation irradiated with excitation light (i.e., visible fluorescence having at least some wavelengths different from the first wavelength band, or fluorescence in the invisible light wavelength band).
• the disclosed technology is not limited to the image generation examples shown below, and can be applied as appropriate to other image generation examples not shown below.
• the imaging unit 52 does not necessarily have to be equipped with the two imaging elements 522a and 522b shown in FIG. 28.
• the technology shown in each of the image generation examples below can be effectively applied even when the imaging unit 52 has three or more imaging elements 522.
• the wavelength characteristics of the light received by the imaging element 522 are not limited.
• the technology shown in each of the image generation examples below can be effectively applied even when two or more types of fluorescence having different wavelength bands are emitted from an observation object irradiated with excitation light and are received by one or more imaging elements 522.
• FIG. 30 is a diagram for explaining an example of a generation process of the first images 411 and 412, the second images 421 and 422, and the superimposed image 431 in the first image generation example.
• 30(a) and (c) show first images 411 and 412 generated based on image signals from the first imaging element 522a, where (a) shows the first image 411 before the size adjustment processing step, and (c) shows the first image 412 after the size adjustment processing step.
• 30(b) and (d) show second images 421 and 422 generated based on image signals from the second imaging element 522b, where (b) shows the second image 421 before the size adjustment processing step, and (d) shows the second image 422 after the size adjustment processing step.
• 30(e) shows a superimposed image 431 generated based on the first image 412 and the second image 422 after the size adjustment processing step.
• the process of generating the various images shown in FIG. 30 is appropriately performed by the image generating unit 93 under the control of the control unit 94 in the control device 90 (see FIG. 7).
• an organ having fluorescent coloring sites B1 and B2 as identification targets is imaged as an observation target by the first imaging element 522a and the second imaging element 522b.
• the fluorescent coloring sites B1 and B2 are shown surrounded by a two-dot chain line, but this figure is merely a reference diagram showing the corresponding range and position of the fluorescent coloring sites B1 and B2.
• the first images 411 and 412 in (a) and (c) of FIG. 30 do not include the fluorescent coloring sites B1 and B2 as identification images.
• the fluorescent coloring site B2 is shown surrounded by a two-dot chain line, but this figure is merely a reference diagram showing the corresponding range and position of the fluorescent coloring site B2.
• the superimposed image 431 in (e) of FIG. 30 does not include the fluorescent coloring site B2 as an identification image.
• the superimposed image 431 in FIG. 30(e) includes the fluorescent color-producing region B1 as an identifiable image.
• captured images 412, 422 (see (c) and (d) in Figures 30) of the desired size are generated from captured images 411, 421 (see (a) and (b) in Figures 30) based on the image signal output from the image sensor 522. Then, an output image is generated based on the first image 412 and the second image 422 after the size adjustment processing step.
• a scaling process is performed on the first image 411 and the second image 421, and the size (number of pixels) of both or one of the first image 411 and the second image 421 is adjusted to provide the first image 412 and the second image 422 after the size adjustment process step.
• pixel interpolation is performed, and the number of pixels constituting the image is increased, and the overall size of the image is increased.
• pixel thinning is performed, and the number of pixels constituting the image is reduced, and the overall size of the image is reduced.
• the zoom ratio in the zoom process can be determined taking into consideration the generation of the superimposed image 431 described below. Specifically, the zoom ratio is determined to a ratio that makes the size of the area (common image area) of the observation target (including the identification target) that is commonly captured in the first image 411 and the second image 421 consistent between the first image 411 and the second image 421. As a result, the size of one or both of the first image 411 and the second image 421 is adjusted so that the number of pixels in the common image area is consistent between the first image 412 and the second image 422.
• the size (number of pixels) of the second image 421 based on the image signal output from the second imaging element 522b is smaller than the size of the first image 411 based on the image signal output from the first imaging element 522a. Therefore, the enlargement ratio of the enlargement processing for the second image 421 is determined so that after the size adjustment processing step, the entire second image 422 has the same size (number of pixels) as the common image area of the first image 412 (see reference symbol "411A" in FIG. 29).
• the common image area of the first image 412 has the same angle of view as the entire second image 422, and the observed object is shown in the common image area of the first image 412 and the second image 422 with the same range and size.
• the first image 411 may be enlarged, reduced, or neither enlarged nor reduced in the size adjustment processing step. Therefore, the first images 411, 412 (see (a) and (c) in FIG. 30) based on the image signal from the first imaging element 522a may have the same number of pixels before and after the size adjustment processing step. In this case, enlargement/reduction processing is not necessary for the first image 411, and size adjustment processing of the first image 411 is not actually performed in the size adjustment processing step. Note that even in this case, the first image before the size adjustment processing step is given the symbol "411", and the first image after the size adjustment processing step is given the symbol "412".
• the range of the object of observation captured in the second images 421 and 422 is narrower than the range of the object of observation captured in the first images 411 and 412, and the area of the object of observation that is not captured in the second images 421 and 422 (non-common image area 411B in FIG. 29) is included in the first images 411 and 412. Therefore, after the size adjustment processing step, the total number of pixels does not match between the first image 412 and the second image 422, and the number of pixels in the second image 422 is less than the number of pixels in the first image 412.
• a superimposed image 431 is generated based on the first image 412 and the second image 422 after the size adjustment processing step (see (e) in Figure 30).
• first image 412 and the second image 422 are superimposed so that the size and position of the area of the observation subject that is commonly captured in the first image 412 and the second image 422 match.
• first image 412 and the second image 422 may be superimposed so that the center of the second image 422 is aligned with the center of the common image area of the first image 412.
• the number of pixels in the common image area of the first image 412 matches the total number of pixels in the second image 422.
• the areas of the object of observation that are commonly captured between the first image 412 (particularly the common image area) and the second image 422 correspond to each other on a pixel-by-pixel basis, so that the superimposition process of the first image 412 and the second image 422 can be appropriately performed on a pixel-by-pixel basis.
• the superimposed image 431 is generated based on the first image 412 and the second image 422, which have been adjusted so that the image sizes (number of pixels) of the area to be observed (particularly the fluorescent color-producing area B1 to be identified) match, and the identification target is appropriately displayed in the superimposed image 431.
• the superimposed image 431 includes an overlapping area 431A to which an image based on the first image 412 and the second image 422 is assigned, and a non-overlapping area 431B to which an image based on the first image 412 but not the second image 422 is assigned.
• the overlapping area 431A of the overlapping image 431 is assigned an overlapping composite image of the area to be observed that is commonly captured in the first image 412 and the second image 422.
• the non-overlapping area 431B adjacent to the overlapping area 431A is assigned an image of the area to be observed that is captured only in the first image 412 and not in the second image 422 (i.e., the non-common image area of the first image 412 (see reference symbol "411B" in FIG. 29)).
• the non-overlapping area 431B surrounds the entire overlapping area 431A, but the scope of the non-overlapping area 431B is not limited, and the non-overlapping area 431B does not necessarily have to surround the entire overlapping area 431A.
• the second image 422 is reflected in the image assigned to the overlapping area 431A of the overlapping image 431, but is not reflected in the image assigned to the non-overlapping area 431B.
• the reflected light of white light irradiated onto the fluorescent sites B1 and B2 has wavelength characteristics similar to those of the reflected light of white light irradiated onto the areas surrounding the fluorescent sites B1 and B2. Therefore, the first images 411 and 412, which are images taken under normal light, include the observed organ A as a identifiable image, but do not include the fluorescent sites B1 and B2 as identifiable images. Therefore, it is difficult for the user to visually separate and identify the fluorescent sites B1 and B2 from the surrounding areas in the first images 411 and 412 (images taken under normal light) acquired by the first image sensor 522a (see FIG. 30(a)).
• the fluorescent color-producing sites B1 and B2 in this example are irradiated with excitation light (second light) from the first narrowband light source (second light source) 12, they emit fluorescence that can be imaged by the second imager 522b. Therefore, in the second image 421 (see FIG. 30(b)), which is a fluorescent photographic image, the observed organ A is not included as an identifiable image.
• the fluorescent color-producing sites particularly the fluorescent color-producing site B1 located in the region of the observed object, which is commonly captured in the first image 412 and the second image 422 are included as identifiable images in the second image 421. Therefore, the user can visually distinguish the fluorescent color-producing site B1 from the surrounding sites in the second image 421 (fluorescent photographic image) acquired by the second imager 522b.
• fluorescent coloring site B2 in the organ area which is shown as an image captured under normal light in non-overlapping region 431B, is not shown as a fluorescent image in superimposed image 431. Therefore, a user who views superimposed image 431 may mistakenly believe that a fluorescent coloring site does not exist, even if such a fluorescent coloring site exists in the organ area shown in non-overlapping region 431B.
• boundary F between overlapping area 431A and non-overlapping area 431B in superimposed image 431 is not necessarily visually clear.
• boundary F between overlapping area 431A and non-overlapping area 431B is indicated by a two-dot chain line, but this boundary F is a virtual line added for convenience, and is not a line that is explicitly displayed from the beginning in superimposed image 431. Therefore, it is not necessarily easy for the user to clearly identify the image range related to second image 422 in superimposed image 431, and there is a possibility that the user will mistakenly recognize at least a part of the range of non-overlapping area 431B as overlapping area 431A.
• Figures 31 to 33 are diagrams that explain an example of the output image generation process in the first image generation example.
• FIG. 31 shows an example of a first output image 413 generated from a first image 412 (normal light captured image), which is an image captured mainly by the first imaging element 522a.
• (a) of FIG. 31 shows the first image 412, which is the original image of the first output image 413 (see (b) to (d) of FIG. 31).
• the process of generating the various images shown in FIG. 31 is appropriately performed by the image generating unit 93 under the control of the control unit 94 in the control device 90 (see FIG. 7).
• a first output image 413 is generated based on a first image 412, and can be displayed on the display device 70 simultaneously with or switchably displayed with other images.
• the first output image 413 in this image generation example includes a first image area 451 and a second image area 452 that is different from the first image area 451.
• the first image area 451 is generated based on the first image 412 and the second image 422, and is assigned an image of the area to be observed that is commonly captured in the first image 412 and the second image 422.
• the fact that the first image area 451 is "generated based on the first image 412 and the second image 422" may mean that the range and size of the image of the area to be observed that is assigned to the first image area 451 is determined based on the first image 412 and the second image 422.
• the first output image 413 based on the first image 412 may be, for example, the same image as the first image 412 (see FIG. 31(a)), which is the original image (see FIG. 31(b)).
• the boundary between the first image area 451 and the second image area 452 is not shown in the first output image 413, and the user can view the first output image 413 via the display device 70 without being aware of the boundary.
• pixel replacement processing may be performed so that a boundary enhancement image 446 indicating the boundary 445 of the first image region 451 is included in the first output image 413.
• the pixel replacement process referred to here includes all processes that change the value of one or more pixels.
• the pixel values before and after the pixel replacement process may be unrelated to each other, or the pixel value after the pixel replacement process may be determined based on the pixel value before the pixel replacement process. Therefore, pixel synthesis, in which a pixel value after pixel replacement process is derived based on multiple pixel values before the pixel replacement process, is also included in pixel replacement process.
• a linear (e.g., dotted) border-enhanced image 446 extends along the border 445 between the first image region 451 and the second image region 452 to indicate the border 445.
• a pixel replacement process is performed on corresponding pixels in the first output image 413 so that the corresponding pixels have a specific color indicating the border-enhanced image 446, thereby making it possible to include the border-enhanced image 446 in the first output image 413.
• the border-enhanced image 446 thus extending along the border 445 of the first image region 451 may be located only in the first image region 451, may be located only in the second image region 452, or may be located in both the first image region 451 and the second image region 452. Note that reducing the area (e.g., line width) occupied by the border-enhanced image 446 is preferable from the viewpoint of preventing a reduction in the amount of information in the first output image 413.
• a unique image is assigned to the second image area 452 as the boundary enhancement image 446, and the boundary 445 of the first image area 451 is shown in the first output image 413.
• the boundary-enhanced image 446 assigned to the second image area 452 is an image that is visually distinct and identifiable from the image assigned to the first image area 451 (i.e., the first image 412), and can have any color, pattern, and design. Therefore, the boundary-enhanced image 446 may be, for example, a monochrome mask image, an image with a diagonal line pattern, or an image with a checkerboard pattern.
• the size of the observation target does not change between the displayed images, allowing users such as doctors to smoothly proceed with medical procedures such as surgery.
• an image based on the first image 412 may be displayed in the second image area 452 together with the boundary-emphasized image 446.
• an image (semi-transparent image) of the non-common image area of the first image 412 combined with the boundary-emphasized image 446 by alpha blending may be assigned to the second image area 452 of the first output image 413.
• organ A is included as an identifiable image, but fluorescent color-producing areas B1 and B2 are not included as identifiable images. Therefore, the user can identify organ A from the first output image 413, but has difficulty identifying fluorescent color-producing areas B1 and B2.
• FIG. 32 shows an example of a second output image 423 generated from a second image 422 (fluorescence image) that is an image captured mainly by the second imaging element 522b.
• FIG. 32(a) shows the second image 422 that is the original image of the second output image 423 (see FIG. 32(b)).
• the process of generating the various images shown in FIG. 32 is appropriately performed by the image generating unit 93 under the control of the control unit 94 in the control device 90 (see FIG. 7).
• a second output image 423 based on the second image 422 is generated and can be displayed on the display device 70 simultaneously with or switchably displayed with other images.
• a second output image 423 based on the second image 422 may have a first image region 451 to which the second image 422 is assigned, and a second image region 452 to which a unique image serving as the boundary enhancement image 446 is assigned.
• the boundary-enhanced image 446 assigned to the second image area 452 is an image that is visually distinct and identifiable from the image assigned to the first image area 451 (i.e., the second image 422), and can have any color, pattern, and design.
• an image unrelated to the first image 412 and the second image 422 e.g., a black mask image
• the boundary-enhanced image 446 may be used as the boundary-enhanced image 446.
• the image processing method for generating such second output image 423 is not limited.
• the second output image 423 may be generated by the image generation unit 93 performing pixel addition processing that adds a boundary enhancement image 446 to the periphery of the second image 422.
• the observed organ A is not included as an identifiable image, but the fluorescent coloring site (fluorescent coloring site B1 located in the first image area 451) is included as an identifiable image. Therefore, although it is difficult for the user to identify organ A from the second output image 423, he or she can identify fluorescent coloring site B1.
• FIG. 33 shows an example of a superimposed output image 432 generated from a first image 412 (normal light image) and a second image 422 (fluorescence image) captured by the first and second image capture elements 522a and 522b.
• (a) of FIG. 33 shows a superimposed image 431, which is the original image of the superimposed output image 432 (see (b) and (d) of FIG. 33).
• the process of generating the various images shown in FIG. 33 is appropriately performed by the image generation unit 93 under the control of the control unit 94 in the control device 90 (see FIG. 7).
• a superimposed output image 432 is generated based on the superimposed image 431, and can be displayed on the display device 70 simultaneously with or switchably to other images.
• a superimposed output image 432 based on a superimposed image 431 may have a first image area 451 to which the superimposed image is assigned, and a second image area 452 to which an image different from the first image area 451 is assigned.
• a pixel replacement process may be performed so that a boundary enhancement image 446 showing a boundary 445 of the first image area 451 is included in the superimposed output image 432.
• the superimposed image assigned to the first image area 451 of the superimposed output image 432 is generated based on the image of the common image area in the first image 412 and the image of the common image area in the second image 422, which have the same number of pixels, as described above.
• a linear (e.g., dotted) border-enhanced image 446 extends along the border 445 between the first image region 451 and the second image region 452.
• the border-enhanced image 446 can be included in the superimposed output image 432 by performing pixel replacement processing on the corresponding pixel in the superimposed output image 432 so that the corresponding pixel exhibits a specific color indicating the border-enhanced image 446.
• the border-enhanced image 446 extending along the border 445 of the first image region 451 may be located only in the first image region 451, may be located only in the second image region 452, or may be located in both the first image region 451 and the second image region 452.
• a unique image is assigned to the second image area 452 as the boundary-enhanced image 446, thereby showing the boundary 445 of the first image area 451 in the superimposed output image 432.
• the boundary-enhanced image 446 assigned to the second image area 452 is an image that is visually distinct and identifiable from the image assigned to the first image area 451 (i.e., the superimposed image of the first image 412 and the second image 422), and may have any color, pattern, or design.
• using an image unrelated to the first image 412 and the second image 422 e.g., a black mask image
• the boundary-enhanced image 446 is advantageous in preventing erroneous recognition of the superimposed output image 432.
• the second image area 452 may display an image based on the first image 412 (e.g., an image of the non-common image area 411B (see FIG. 29)) along with the boundary-enhanced image 446.
• superimposed output image 432 may be generated by performing pixel replacement processing to replace data of a plurality of pixels of superimposed image 431 (particularly, a plurality of pixels constituting non-overlapping region 431B) with data of boundary emphasis image 446.
• superimposed output image 432 may be generated by performing pixel replacement processing to combine superimposed image 431 (particularly, the image of non-overlapping region 431B) with boundary emphasis image 446.
• the image generating unit 93 of the control device 90 outputs the second output image 423 and the superimposed output image 432 to the display device 70.
• the superimposed output image 432 is an image in which one or more pixel signals of the first image 412 and the second image 422 are replaced or deleted based on a first range, which is the range of the observation object imaged by the first image sensor 522a to obtain the first images 411 and 412, and a second range, which is the range of the observation object imaged by the second image sensor 522b to obtain the second images 421 and 422.
• the second output image 423 is an image generated by adding one or more pixel signals (i.e., pixel data (pixel values)) to an image created from at least a portion of the pixel signals of the second image 422.
• FIGS. 34A to 34C are flowcharts showing an example of the process for generating the superimposed output image 432.
• Fig. 34A is a flowchart of an example of the generation process corresponding to the example shown in Figs. 30 to 33 described above. That is, after the size adjustment process step is performed on the first image 411 and the second image 421 (S11 in Fig. 34A; (a) to (d) in Fig. 30), a superimposed image 431 is generated based on the first image 412 and the second image 422 after the size adjustment process (S12; (e) in Fig. 30).
• a boundary enhancement image 446 indicating the boundary of the superimposed area 431A (and thus the boundary 445 of the first image area 451 of the superimposed output image 432) is added to the superimposed image 431, thereby generating a superimposed output image 432 (S13; Fig. 33).
• the superimposed output image 432 can also be generated by other processing flows.
• a size adjustment process step may be performed on the first image 411 and the second image 421 (S22).
• a superimposed image 431 (S23) generated based on the first image 412 and the second image 422 after the size adjustment process step can be used as a superimposed output image 432 (see Fig. 33).
• the "image that emphasizes the boundary between the common image area 411A and the non-common image area 411B" that is applied to the first image 411 by the boundary clarification process (S21) for the first image 411 ultimately constitutes the boundary-emphasized image 446 in the superimposed output image 432.
• the "image that emphasizes the boundary between the common image area 411A and the non-common image area 411B" referred to here is not limited, and may be a line-shaped image that extends along the boundary between the common image area 411A and the non-common image area 411B.
• a unique image that is visually distinct and identifiable from the image assigned to the common image area 411A may be assigned to the non-common image area 411B as the "image that emphasizes the boundary between the common image area 411A and the non-common image area 411B".
• a boundary clarification process may be executed for the first image 412 to clarify the boundary of the common image area 411A.
• a superimposed image 431 (S33) generated based on the first image 412 and the second image 422 after the boundary clarification process can be used as a superimposed output image 432 (see FIG. 33).
• data of a new pixel may be created based on data of multiple pixels in the original image.
• the boundary emphasis image 446 in the superimposed output image 432 may become unclear.
• the first image area 451 of the output images 413, 423, 432 is assigned an image of the area to be observed that is commonly captured in the first images 411, 412 and the second images 421, 422, and that is based on both or one of the first images 411, 412 and the second images 421, 422.
• the output images 413, 423, 432 can then include a boundary-enhanced image 446 that indicates the boundary 445 of the first image area 451.
• the user can clearly grasp the boundary 445 of the first image area 451 in the output images 413, 423, 432 based on the boundary-enhanced image 446 in the output images 413, 423, 432. Therefore, the user can accurately recognize the image of the area to be observed that is commonly captured in the first images 411, 412 and the second images 421, 422 in the output images 413, 423, 432.
• the first output image 413, the second output image 423, and the superimposed output image 432 can be generated to have the same number of pixels and resolution overall.
• the number of pixels and resolution of the first image area 451 to which the image of the area to be observed, which is commonly captured in the first images 411, 412 and the second images 421, 422, is assigned can be made to match between the first output image 413, the second output image 423, and the superimposed output image 432.
• the same area of the observation object is displayed at the same size in the first image area 451 of each output image. Therefore, even when multiple output images 413, 423, and 432 are switched and displayed on a common display device 70, the user can seamlessly identify the observation object in each output image, and can easily visually compare the observation objects between the output images.
• the image generating unit 93 (display control unit 935 (see FIG. 7)) of the control device 90 may change the display angle of view of the output images 413, 423, 432 on the display device 70 based on a digital zoom instruction from the user via the input unit 95.
• a digital zoom instruction from the user via the input unit 95.
• the boundary emphasis image 446 is displayed in the second image region 452 and the second image region 452 is used as a mask region (see FIG. 31(d), FIG. 32(b), and FIG.
• the output images 413, 423, 432 may be generated so that the display size of the mask region (second image region 452) is not changed on the display device 70, and only the image of the first image region 451 inside the mask region is displayed at the angle of view based on the user's instruction.
• the output images 413, 423, 432 may be generated such that the overall display size of the output images 413, 423, 432 including the mask region (second image region 452) is changed on the display device 70, and the output images 413, 423, 432 are displayed at an angle of view based on a user instruction.
• FIG. 35 is a diagram illustrating an example of the generation process of the first images 411, 412, the second images 421, 422, and the superimposed image 431 in the second image generation example.
• 35(a) and (c) show first images 411 and 412 generated based on image signals from the first imaging element 522a, where (a) shows the first image 411 before the size adjustment processing step, and (c) shows the first image 412 after the size adjustment processing step.
• 35(b) and (d) show second images 421 and 422 generated based on image signals from the second imaging element 522b, where (b) shows the second image 421 before the size adjustment processing step, and (d) shows the second image 422 after the size adjustment processing step.
• 35(e) shows a superimposed image 431 generated based on the first image 412 and the second image 422 after the size adjustment processing step.
• the process of generating the various images shown in FIG. 35 is appropriately performed by the image generating unit 93 under the control of the control unit 94 in the control device 90 (see FIG. 7).
• an organ having fluorescent color-producing sites B1 and B2 as identification targets is imaged as an observation target by the first image sensor 522a and the second image sensor 522b.
• FIG. 35 shows the fluorescent color-producing sites B1 and B2 as surrounded by a two-dot chain line, this is merely a reference diagram showing the corresponding range and position of the fluorescent color-producing sites B1 and B2.
• the first images 411 and 412 in (a) and (c) of FIG. 35 do not include the fluorescent color-producing sites B1 and B2 as identification targets.
• captured images 412, 422 (see (c) and (d) in Figures 35) of the desired size are generated from captured images 411, 412 (see (a) and (b) in Figures 35) based on the image signal output from the image sensor 522. Then, an output image is generated based on the first image 412 and second image 422 after the size adjustment processing step.
• an image of the area to be observed (common object area) that is commonly captured in the first image 411 and the second image 421 (i.e., an image of the common image area 411A (see FIG. 29)) is extracted from the first image 411, and a size adjustment process is performed on the extracted portion.
• the process of extracting the image of the common image area 411A here corresponds to a pixel deletion process that deletes pixels of the non-common image area 411B from the first image 411.
• a scaling process is performed on the extracted portions of the first image 411 (image of the common image area 411A) and/or the second image 421, and the size (number of pixels) of the first image 411 and/or the second image 421 is adjusted (size adjustment process step).
• the scaling factor in the scaling process is determined to be a factor that makes the size of the common image area consistent between the first image 412 and the second image 422. Therefore, the size of the first image 411 and/or the second image 421 is adjusted so that the number of pixels of the common image area is consistent between the first image 412 and the second image 422.
• the overall size (number of pixels) of the second image 421 is smaller than the size of the partial image of the first image 411 based on the image signal output from the common image area 411A of the first imaging element 522a.
• the extracted partial image of the first image 411 (the image of the common image area 411A) is enlarged to form the entire first image 412. Meanwhile, the entire second image 421 is enlarged.
• the first image 412 and the second image 422 contain only the image of the area to be observed (the common image area) that is commonly captured, and have the same number of pixels and resolution.
• a superimposed image 431 is generated based on the first image 412 and the second image 422 after the size adjustment processing step (see FIG. 35(e)). That is, the first image 412 and the second image 422 are superimposed so that the size and position of the area of the observation target that is commonly captured in the first image 412 and the second image 422 match.
• the superimposed image 431 includes only the superimposed area 431A to which images based on the first image 412 and the second image 422 are assigned, and does not include the non-superimposed area 431B (see FIG. 30(e)).
• FIG. 36 is a diagram illustrating an example of the output image generation process in the second image generation example.
• FIG. 36 shows a first image 412 which is the original image of a first output image 413 (see (d) of FIG. 36).
• (b) of FIG. 36 shows a second image 422 which is the original image of a second output image 423 (see (e) of FIG. 36).
• (c) of FIG. 36 shows a superimposed image 431 which is the original image of a superimposed output image 432 (see (f) of FIG. 36).
• the process of generating the various images shown in FIG. 36 is appropriately performed by an image generating unit 93 under the control of a control unit 94 in the control device 90 (see FIG. 7).
• a first output image 413 based on a first image 412, a second output image 423 based on a second image 422, and a superimposed output image 432 based on a superimposed image 431 are generated, and can be displayed simultaneously or switchably on a display device 70.
• the first output image 413, the second output image 423, and the superimposed output image 432 may be the same images as the first image 412, the second image 422, and the superimposed image 431, respectively, which are the original images.
• the output images 413, 423, 432 shown in (d) to (f) of FIG. 36 include only the first image area 451 to which the image of the common image area is assigned, and do not include a second image area 452 that is different from the first image area 451.
• the outline portions of the output images 413, 423, 432 indicate the boundary of the first image area 451 to which the image of the common image area is assigned. Therefore, the output images 413, 423, 432 of this image generation example do not require an "additional image to indicate the boundary 445 of the first image area 451" like the boundary emphasis image 446 in the first image generation example described above (see (c) and (d) of FIG. 31, etc.).
• the output image is generated based on the entire second image 421, but the output image may be generated based on a partial range of the second image 421.
• the image generating unit 93 may extract a partial area of the second image 421 including the area in which the observation target is captured, and perform a size adjustment process on the image of the extracted area to generate the second image 421.
• surgery using a medical observation system may generally be performed while switching between several observation modes (for example, "normal light image observation mode based on normal light images” and “superimposed image observation mode based on superimposed images of normal light images and fluorescent images”).
• the image generation unit 93 may generate a superimposed image by superimposing a fluorescent image on a normal light image captured using parameters of the normal light image observation mode under the control of the control unit 94.
• the parameters include image processing parameters such as image WB (white balance), color tone, color mode, and color matrix adjustment.
• image WB white balance
• color tone color mode
• color matrix adjustment color matrix adjustment
• the parameters of the normal light image observation mode may be used as is, or parameters set based on the parameters of the normal light image observation mode may be used.
• the parameters used when acquiring an image captured under fluorescent light in the superimposed image observation mode may be determined based on such parameters related to the image captured under normal light, or may be set independently of the parameters related to the image captured under normal light.
• the light source device 10 is connected to the insertion device 20 configured as the endoscope body, but the light source device 10 may be connected to any other light emitting device.
• Figure 37 is a conceptual diagram showing an example of a medical observation system 100 configured as a surgical field illumination observation device in which a ring light 60 (an open field illumination device for biological observation) is connected to a light source device 10.
• a medical observation system 100 configured as a surgical field illumination observation device in which a ring light 60 (an open field illumination device for biological observation) is connected to a light source device 10.
• the light source device 10 and the camera head 50 are connected to a ring light 60.
• Light emitted by the light source device 10 is sent to the ring light 60 via the light guide 30, and is emitted from the ring light 60 toward the observation target S.
• the ring light 60 is used to emit light outside the observation target, and is different from the above-mentioned insertion device 20 (see FIGS. 1A and 1B) which is intended to emit light inside the subject and irradiate the light to the observation target.
• the ring light 60 shown in FIG. 37 is also connected to the camera head 50, and observation light from the observation subject S passes through the ring light 60 and enters the camera head 50.
• a light emitting device such as the ring light 60 connected to the light source device 10 does not have to be connected to the camera head 50, and may be provided separately from the camera head 50 (particularly the imaging unit).
• FIG. 38 is a diagram showing an example of a medical observation system 100 configured as a microscope system.
• the medical observation system 100 shown in FIG. 38 is a surgical microscope system that has the function of magnifying and capturing an image of the field of view of the observation subject, and displaying an output image that is generated based on the captured image.
• the medical observation system 100 (surgical microscope system) of this example includes a microscope device 110 that captures an image of an object to be observed, and a display device 70 that displays the image captured by the microscope device 110.
• the display device 70 shown in FIG. 38 is provided separately from the microscope device 110, but may also be provided integrally with the microscope device 110.
• the microscope device 110 has a microscope section 125 that magnifies and captures a minute area of the observation target, a support section 124 having an arm that rotatably supports the microscope section 125, and a base section 126 that rotatably supports the support section 124 and is movable on the floor.
• the base section 126 has a control device 90 that controls the operation of the medical observation system 100.
• the control device 90 is connected to the microscope section 125 via a transmission cable 123.
• the base section 126 may be configured to be fixed to a ceiling, a wall, etc.
• the microscope unit 125 has the above-mentioned imaging unit (see FIG. 7; not shown in FIG. 38), an operation unit (not shown) such as a switch that accepts input of operational instructions for the microscope device 110, and a cover glass (not shown) for protecting the inside.
• the user can move the microscope unit 125 while operating the operation unit of the microscope unit 125.
• a captured image of the object to be observed is acquired by the microscope unit 125. Then, as in each of the above-described embodiments, the control device 90 generates an output image from the captured image, and the output image is displayed on the display device 70.
• a light emission device that is connected to a light source device (not shown) and emits light from the light source device may be provided integrally with the microscope section 125 (imaging section) or may be provided separately from the microscope section 125.
• the light emission device may be provided on the base section 126, on the support section 124, or may be provided separately from the microscope device 110.
• the technical category that embodies the above-mentioned technical idea is not limited.
• the above-mentioned technical idea may be embodied by a computer program that causes a computer to execute one or more procedures (steps) included in a method of manufacturing or using the above-mentioned device.
• the above-mentioned technical idea may also be embodied by a computer-readable non-transitory recording medium on which such a computer program is recorded.
• the present disclosure may also have the following configuration.
• a light source device that emits broadband light in a first wavelength band, first narrowband light that excites a first substance to emit a first fluorescence in a wavelength band included in the first wavelength band, and second narrowband light that excites a second substance to emit a second fluorescence in a wavelength band not included in the first wavelength band;
• a control unit that controls the light source device, The control unit is In a first mode, the light source device is controlled so that the broadband light and the first narrowband light are irradiated onto an object to be observed in a time-division manner; controlling the light source device so that the broadband light and the second narrowband light are irradiated onto the observation target in a second mode different from the first mode; Medical observation system.
• An imaging unit including a first imaging element and a second imaging element; an optical element that separates light from the object to be observed into a plurality of light beams including a first light beam and a second light beam, guides the first light beam to the first image sensor, and guides the second light beam to the second image sensor; 2.
• the medical observation system according to item 1.
• the second imaging element has a higher sensitivity than the first imaging element. 3.
• the first imaging element has a color filter.
• Item 4 The medical observation system according to item 2 or 3.
• the second imaging element does not have a color filter. 5.
• the medical observation system according to any one of items 2 to 4.
• the optical element is In the first mode, the first light flux including reflected light from the observation object irradiated with the broadband light and the first light flux including the first fluorescence from the observation object irradiated with the first narrowband light are sequentially guided to the first image sensor; In the second mode, the first light flux including reflected light from the observation object irradiated with the broadband light is guided to the first image sensor, and the second light flux including the second fluorescence from the observation object irradiated with the second narrowband light is guided to the second image sensor. 7.
• the medical observation system according to item 6.
• the control unit controls the light source device so that the broadband light and the first narrowband light are irradiated onto the observation object in a time-division manner, and the second narrowband light is irradiated onto the observation object; the optical element sequentially guides the first light flux including reflected light from the observation object irradiated with the broadband light and the first light flux including the first fluorescence from the observation object irradiated with the first narrowband light to the first image sensor, and guides the second light flux including the second fluorescence from the observation object irradiated with the second narrowband light to the second image sensor.
• the light source device emits third narrowband light that excites a third substance that emits third fluorescence in a wavelength band that is not included in the first wavelength band and is at least partially different from a wavelength band of the second fluorescence;
• the control unit controls the light source device so that the broadband light and the third narrowband light are irradiated onto the observation target;
• the optical element guides the first light flux including reflected light from the observation object irradiated with the broadband light to the first image sensor, and guides the second light flux including the third fluorescence from the observation object irradiated with the third narrowband light to the second image sensor.
• the light source device emits third narrowband light that excites a third substance that emits third fluorescence in a wavelength band that is not included in the first wavelength band and is at least partially different from a wavelength band of the second fluorescence;
• the control unit controls the light source device so that the broadband light and the first narrowband light are irradiated to the observation object in a time-division manner, and the second narrowband light and the third narrowband light are irradiated to the observation object in a time-division manner;
• the optical element sequentially guides, to the first imaging element, the first light flux including reflected light from the observation object irradiated with the broadband light and the first light flux including the first fluorescence from the observation object irradiated with the first narrowband light, and sequentially guides, to the second imaging element, the second light flux including the second fluorescence from the observation object irradiated with the second narrowband light and the second light flux including the third fluorescence
• the optical element is Guide the first light flux in which the light in the first wavelength band is partially suppressed to the first image sensor;
• the second light flux in which the light in the first wavelength band is partially suppressed is guided to the second image sensor.
• Item 12 The medical observation system according to item 11.
• an image generating unit that generates an image based on an image signal from the imaging unit
• the control unit controls the imaging unit and the image generating unit
• the optical element sequentially guides, to the first imaging element, the first light flux including reflected light from the observation object irradiated with the broadband light and the first light flux including the first fluorescence from the observation object irradiated with the first narrowband light
• the control unit controls the imaging unit and the image generation unit so that an image based on reflected light is generated based on an image signal output from the first imaging element that receives the first light flux including reflected light, and an image based on the first fluorescence is generated based on an image signal output from the second imaging element that receives the second light flux including the first fluorescence,
• the second imaging element controls the imaging unit
• an image generating unit that generates an image based on an image signal from the imaging unit
• the control unit controls the imaging unit and the image generating unit
• the control unit controls the light source device so that the broadband light, the first narrowband light, and the second narrowband light are irradiated onto the object of observation in a time-division manner;
• the optical element sequentially guides, to the first imaging element, the first light flux including reflected light from the observation object irradiated with the broadband light and the first light flux including the first fluorescence from the observation object irradiated with the first narrowband light, and sequentially guides, to the second imaging element, the second light flux including reflected light from the observation object irradiated with the broadband light, the second light flux including the first fluorescence from the observation object irradiated with the first narrowband light, and the second light flux including the second fluorescence from the observation object irradiated with the second narrowband light, the control unit controls the imaging
• the optical element is guiding the first light flux in which the light in the wavelength band of the first fluorescence is partially, substantially, or completely suppressed to the first image sensor; guide the second light flux, in which light in a wavelength band other than the wavelength band of the first fluorescence in the first wavelength band is partially, substantially, or completely suppressed, to the second image sensor; Item 12.
• the optical element is In the first mode, the first light flux including reflected light from the observation object irradiated with the broadband light is guided to the first image sensor, and the second light flux including the first fluorescence from the observation object irradiated with the first narrowband light is guided to the second image sensor. In the second mode, the first light flux including reflected light from the observation object irradiated with the broadband light is guided to the first image sensor, and the second light flux including the second fluorescence from the observation object irradiated with the second narrowband light is guided to the second image sensor.
• the medical observation system according to item 15.
• the control unit controls the light source device so that the broadband light is irradiated onto the observation target, and the first narrowband light and the second narrowband light are irradiated onto the observation target in a time-division manner;
• the optical element guides the first light beam including reflected light from the observation object irradiated with the broadband light to the first image sensor, and sequentially guides the second light beam including the first fluorescence from the observation object irradiated with the first narrowband light and the second light beam including the second fluorescence from the observation object irradiated with the second narrowband light to the second image sensor.
• the medical observation system according to any of items 16 to 17.
• the second imaging element has a color filter. 5.
• the optical element is Guide the first light flux in which the light in the first wavelength band is partially suppressed to the first image sensor;
• the second light flux in which the light in the first wavelength band is partially suppressed is guided to the second image sensor.
• an image generating unit that generates an image based on an image signal from the imaging unit
• the control unit controls the imaging unit and the image generating unit
• the optical element sequentially guides, to the first imaging element, the first light flux including reflected light from the observation object irradiated with the broadband light and the first light flux including the first fluorescence from the observation object irradiated with the first narrowband light
• the control unit is controlling the imaging unit and the image generating unit so that an image based on reflected light is generated based on one or both of an image signal output from the first imaging element receiving the first light flux including reflected light and an image signal output from the second imaging element receiving the second light flux including reflected light; controlling the imaging section and the image generating section so that an image based on the first
• an image generating unit that generates an image based on an image signal from the imaging unit
• the control unit controls the imaging unit and the image generating unit
• the control unit controls the light source device so that the broadband light, the first narrowband light, and the second narrowband light are irradiated onto the object of observation in a time-division manner;
• the optical element sequentially guides, to the first imaging element, the first light flux including reflected light from the observation object irradiated with the broadband light and the first light flux including the first fluorescence from the observation object irradiated with the first narrowband light, and sequentially guides, to the second imaging element, the second light flux including reflected light from the observation object irradiated with the broadband light, the second light flux including the first fluorescence from the observation object irradiated with the first narrowband light, and the second light flux including the second fluorescence from the observation object irradiated with the second narrowband light, the control unit controls the imaging
• an imaging unit including a first imaging element, a second imaging element, and a third imaging element; an optical element that separates light from the object to be observed into a plurality of light beams including a first light beam, a second light beam, and a third light beam, and guides the first light beam to the first image sensor, the second light beam to the second image sensor, and the third light beam to a third image sensor; 2.
• the medical observation system according to item 1.
• the second imaging element and the third imaging element have a higher sensitivity than the first imaging element. 24.
• the first imaging element has a color filter. 25.
• the second imaging element and the third imaging element do not have a color filter. 26.
• the medical observation system according to any one of items 23 to 25.
• the optical element is In the first mode, the first light flux including reflected light from the observation object irradiated with the broadband light is guided to the first image sensor, and the second light flux including the first fluorescence from the observation object irradiated with the first narrowband light is guided to the second image sensor. In the second mode, the first light flux including reflected light from the observation object irradiated with the broadband light is guided to the first image sensor, and the third light flux including the second fluorescence from the observation object irradiated with the second narrowband light is guided to the third image sensor.
• the control unit controls the light source device so that the broadband light and the first narrowband light are irradiated onto the observation object in a time-division manner, and the second narrowband light is irradiated onto the observation object;
• the optical element guides the first light flux including reflected light from the observation object irradiated with the broadband light to the first imaging element, guides the second light flux including the first fluorescence from the observation object irradiated with the first narrowband light to the second imaging element, and guides the third light flux including the second fluorescence from the observation object irradiated with the second narrowband light to the third imaging element.
• the light source device emits third narrowband light that excites a third substance that emits third fluorescence in a wavelength band that is not included in the first wavelength band and is at least partially different from a wavelength band of the second fluorescence;
• the control unit controls the light source device so that the broadband light and the third narrowband light are irradiated onto the observation target;
• the optical element guides the first light flux including reflected light from the observation object irradiated with the broadband light to the first image sensor, and guides the third light flux including the third fluorescence from the observation object irradiated with the third narrowband light to the third image sensor.
• the light source device emits third narrowband light that excites a third substance that emits third fluorescence in a wavelength band that is not included in the first wavelength band and is at least partially different from a wavelength band of the second fluorescence;
• the control unit controls the light source device so that the broadband light and the first narrowband light are irradiated to the observation object in a time-division manner, and the second narrowband light and the third narrowband light are irradiated to the observation object in a time-division manner;
• the optical element guides the first light flux including reflected light from the observation object irradiated with the broadband light to the first imaging element, guides the second light flux including the first fluorescence from the observation object irradiated with the first narrowband light to the second imaging element, and sequentially guides a third light flux including the second fluorescence from the observation object irradiated with the second narrowband light and a third light flux including the third fluorescence from the observation
• the optical element is In the first mode, the first light flux including reflected light from the observation object irradiated with the broadband light and the first light flux including the first fluorescence from the observation object irradiated with the first narrowband light are sequentially guided to the first image sensor; In the second mode, the first light flux including reflected light from the observation object irradiated with the broadband light is guided to the first image sensor, and the second light flux including the second fluorescence from the observation object irradiated with the second narrowband light is guided to the second image sensor.
• the medical observation system according to item 32.
• the control unit controls the light source device so that the broadband light and the first narrowband light are irradiated onto the observation object in a time-division manner, and the second narrowband light is irradiated onto the observation object; the optical element sequentially guides the first light flux including reflected light from the observation object irradiated with the broadband light and the first light flux including the first fluorescence from the observation object irradiated with the first narrowband light to the first image sensor, and guides the second light flux including the second fluorescence from the observation object irradiated with the second narrowband light to the second image sensor.
• the control unit controls the light source device so that the broadband light and the first narrowband light are irradiated onto the observation object in a time-division manner, and the second narrowband light is irradiated onto the observation object; the optical element sequentially guides the first light flux including reflected light from the observation object irradiated with the broadband light and the first light flux including the first fluorescence from the observation object irradiated with
• the control unit controls the light source device so that the broadband light and the third narrowband light are irradiated onto the observation object; the optical element guides the first light flux including reflected light from the observation object irradiated with the broadband light to the first image sensor, and guides the second light flux including third fluorescence from the observation object irradiated with the third narrowband light to the second image sensor.
• the control unit controls the light source device so that the broadband light and the third narrowband light are irradiated onto the observation object; the optical element guides the first light flux including reflected light from the observation object irradiated with the broadband light to the first image sensor, and guides the second light flux including third fluorescence from the observation object irradiated with the third narrowband light to the second image sensor.
• the light source device emits third narrowband light that excites a third substance that emits third fluorescence in a wavelength band that is not included in the first wavelength band and is at least partially different from a wavelength band of the second fluorescence;
• the control unit controls the light source device so that the broadband light and the third narrowband light are irradiated onto the observation target;
• the optical element guides the first light flux including reflected light from the observation object irradiated with the broadband light to the first image sensor, and guides the third light flux including the third fluorescence from the observation object irradiated with the third narrowband light to the third image sensor.
• Item 35 The medical observation system according to item 33 or 34.
• an image generating unit that generates an image based on an image signal from the imaging unit,
• the control unit controls the imaging unit and the image generating unit, the light source device emits third narrowband light that excites a third substance that emits third fluorescence having a wavelength band that is not included in the first wavelength band and that at least partially coincides with a wavelength band of the second fluorescence;
• the control unit controls the light source device so that the broadband light is irradiated onto the observation target, and the second narrowband light and the third narrowband light are irradiated onto the observation target in a time-division manner;
• the optical element guides the first light flux including reflected light from the object of observation irradiated with the broadband light to the first imaging element, sequentially guides the second light flux including the second fluorescence from the object of observation irradiated with the second narrowband light and the second light flux including the third fluorescence from the object of observation irradiated with the third narrowband
• an image generating unit that generates an image based on an image signal from the imaging unit,
• the control unit controls the imaging unit and the image generating unit, the light source device emits third narrowband light that excites a third substance that emits third fluorescence having a wavelength band that is not included in the first wavelength band and that at least partially coincides with a wavelength band of the second fluorescence;
• the control unit controls the light source device so that the broadband light and the first narrowband light are irradiated to the observation object in a time-division manner, and the second narrowband light and the third narrowband light are irradiated to the observation object in a time-division manner;
• the optical element sequentially guides the first light flux including reflected light from the observation object irradiated with the broadband light and the first light flux including the first fluorescence from the observation object irradiated with the first narrowband light to the first imaging element; sequentially guides the second light flux including the second fluorescence
• the first imaging element and the second imaging element have color filters; 25.
• the third imaging element does not have a color filter. 40.
• an image generating unit that generates an image based on an image signal from the imaging unit
• the control unit controls the imaging unit and the image generating unit
• the optical element sequentially guides, to the first imaging element, the first light flux including reflected light from the observation object irradiated with the broadband light and the first light flux including the first fluorescence from the observation object irradiated with the first narrowband light
• the control unit is controlling the imaging unit and the image generating unit so that an image based on reflected light is generated based on one or both of an image signal output from the first imaging element receiving the first light flux including reflected light and an image signal output from the second imaging element receiving the second light flux including reflected light; controlling the imaging unit and the image generating unit so that an image based on the first
• an image generating unit that generates an image based on an image signal from the imaging unit,
• the control unit controls the imaging unit and the image generating unit,
• the control unit controls the light source device so that the broadband light and the first narrowband light are irradiated onto the observation object in a time-division manner, and the second narrowband light is irradiated onto the observation object;
• the optical element sequentially guides the first light flux including reflected light from the object of observation irradiated with the broadband light and the first light flux including the first fluorescence from the object of observation irradiated with the first narrowband light to the first imaging element, sequentially guides the second light flux including reflected light from the object of observation irradiated with the broadband light and the second light flux including the first fluorescence from the object of observation irradiated with the first narrowband light to the second imaging element, and guides a third light flux including the second fluorescence from the object of observation irradiated with the
• Item 44 The medical observation system according to any one of items 1 to 43, wherein the first wavelength band is included in a visible light wavelength band.
• Item 45 The medical observation system according to any one of items 1 to 44, wherein the wavelength of the second fluorescent light is included in the invisible light wavelength band.
• Item 47 The medical observation system of any of items 1 to 46, comprising a filter element that partially, substantially or completely suppresses light in the wavelength band of the first narrowband light.
• Item 48 The medical observation system according to any one of items 1 to 47, comprising a filter element for partially, substantially or completely suppressing light in the wavelength band of the second narrowband light.
• An instruction receiving unit that receives an instruction from a user, The control unit determines, based on an instruction from a user received by an instruction receiving unit, whether to use one or both of the image signal output from the first imaging element and the image signal output from the second imaging element to generate an image based on reflected light.
• Item 21 or 43 The medical observation system according to item 21 or 43.
• the method includes a step of emitting at least one of broadband light in a first wavelength band, first narrowband light that excites a first substance that emits a first fluorescence in a wavelength band included in the first wavelength band, or second narrowband light that excites a second substance that emits a second fluorescence in a wavelength band not included in the first wavelength band, from a light source device;
• a first mode the broadband light and the first narrowband light are emitted from the light source device so that the broadband light and the first narrowband light are irradiated onto an object to be observed in a time-division manner;
• a second mode different from the first mode the broadband light and the second narrowband light are emitted from the light source device so that the broadband light and the second narrowband light are irradiated onto the observation object.
• Medical observation methods are used to observe a first wavelength band, first narrowband light that excites a first substance that emits a first fluorescence in a wavelength band included in the first wavelength band, or second narrowband
• An image generating device that generates an output image based on a first image, which is an image of a first range of an observation object captured by a first imaging element, and a second image, which is an image of a second range of the observation object different from the first range, captured by a second imaging element of a different type from the first imaging element, the output image is an image that has been subjected to a process of replacing or deleting one or more pixel signals among the pixel signals of the first image and the second image based on the first range and the second range.
• Image generating device that generates an output image based on a first image, which is an image of a first range of an observation object captured by a first imaging element, and a second image, which is an image of a second range of the observation object different from the first range, captured by a second imaging element of a different type from the first imaging element, the output image is an image that has been subjected to a process of replacing or deleting one or more pixel signals among the pixel signals of the first image
• the type is a physical size, an image size, or a model number of an imaging element; Item 52.
• One of the first range and the second range is wider than the other. 54.
• An image generating device according to any one of items 51 to 53.
• a first image region of the output image is generated based on the first image and the second image; 55.
• An image generating device according to any one of items 51 to 54.
• the output image includes the first image area and a second image area different from the first image area; Item 56.
• An image generating device according to item 55.
• the output image consists of only the first image region; Item 56.
• An image generating device according to item 55.
• the output image includes a boundary enhancement image showing a boundary between the first image region and the second image region; 58.
• An image generating device according to any one of items 56 and 57.
• the second range is included in the first range. 60.
• An image generating device according to any one of items 51 to 59.
• Item 61 An image generating apparatus according to any one of items 51 to 60, wherein the output image is generated based on the entirety of the second image.
• [Item 62] Enlarging or reducing one or both of the first image and the second image so that a number of pixels of a common region, which is an image region of a portion of the observation target that is commonly captured in the first image and the second image, is matched between the first image and the second image; generating an image to be assigned to the first image area of the output image based on an image of the common area in the first image and an image of the common area in the second image, the numbers of pixels of which are the same; 60.
• An image generating device according to any one of items 55 to 59.
• [Item 63] a display device or devices that can be connected to the display device or devices that can display the output image; generating and outputting the output image corresponding to the characteristics of each of the one or more display devices; Item 63.
• An image generating device according to any one of items 51 to 62.
• the first image is acquired by photographing the object to be observed irradiated with a first light of a first wavelength band
• the second image is acquired by photographing the object to be observed irradiated with second light having a wavelength band at least partially different from the first wavelength band.
• An image generating device according to any one of items 51 to 63.
• the first light is visible light; the second light is an excitation light that excites a specific substance to emit fluorescence,
• the first image includes a captured image based on reflected light of the visible light from the observation target,
• the second image includes a captured image based on the fluorescence from the observation target irradiated with the excitation light.
• An image generating device according to item 64.
• Item 50 A medical observation system according to any one of items 1 to 49, an imaging device having a first imaging element and a second imaging element of a different type from the first imaging element; an image generating device for generating an output image, The image generating device generating the output image based on a first image, which is an image of a first range of an observation object captured by using the first imaging element, and a second image, which is an image of a second range of the observation object different from the first range, captured by using the second imaging element; the output image is an image that has been subjected to a process of replacing or deleting one or more pixel signals among the pixel signals of the first image and the second image based on the first range and the second range. Medical observation system.
• the type is a physical size, an image size, or a model number of an imaging element; Item 68.
• the medical observation system according to item 66 or 67.
• One of the first range and the second range is wider than the other. 70.
• the medical observation system according to any one of items 66 to 69.
• a first image region of the output image is generated based on the first image and the second image; 71.
• the medical observation system according to any one of items 66 to 70.
• the output image includes the first image area and a second image area different from the first image area; 72.
• the medical observation system according to item 71.
• the output image consists of only the first image region; 72.
• the medical observation system according to item 71.
• the output image includes a boundary enhancement image showing a boundary between the first image region and the second image region; Item 74.
• the medical observation system according to any one of items 72 or 73.
• the second range is included in the first range.
• Item 76 A medical observation system according to any one of items 66 to 75.
• Item 77 The medical observation system according to any of items 66 to 76, wherein the output image is generated based on the entirety of the second image.
• [Item 78] Enlarging or reducing one or both of the first image and the second image so that the number of pixels of a common region, which is a portion of the observation target that is commonly captured in the first image and the second image, matches between the first image and the second image; generating an image to be assigned to the first image area of the output image based on an image of the common area in the first image and an image of the common area in the second image, the numbers of pixels of which are the same; 76.
• the medical observation system according to any one of items 71 to 75.
• [Item 79] a display device or devices that can be connected to the display device or devices that can display the output image; generating and outputting the output image corresponding to the characteristics of each of the one or more display devices; Item 79.
• the medical observation system according to any one of items 66 to 78.
• the first image is acquired by photographing the object to be observed irradiated with a first light of a first wavelength band
• the second image is acquired by photographing the object to be observed irradiated with second light having a wavelength band at least partially different from the first wavelength band.
• the medical observation system according to any one of items 66 to 79.
• the first light is visible light; the second light is an excitation light that excites a specific substance to emit fluorescence,
• the first image includes a captured image based on reflected light of the visible light from the observation target,
• the second image includes a captured image based on the fluorescence from the observation target irradiated with the excitation light.
• Item 81. The medical observation system of item 80.
• the medical observation method generating an output image based on a first image, which is an image of a first range of an observation target captured by a first imaging element, and a second image, which is an image of a second range of the observation target different from the first range, captured by a second imaging element of a different type from the first imaging element; the output image is an image that has been subjected to a process of replacing or deleting one or more pixel signals among the pixel signals of the first image and the second image based on the first range and the second range. Medical observation methods.
• the medical observation method according to item 82 comprising:
• the type is a physical size, an image size, or a model number of an imaging element; 85.
• the medical observation method according to any one of items 82 to 84.
• One of the first range and the second range is wider than the other. 87.
• the medical observation method according to any one of items 82 to 86.
• a first image region of the output image is generated based on the first image and the second image; 88.
• the medical observation method according to any one of items 82 to 87.
• the output image includes the first image area and a second image area different from the first image area; Item 89.
• the output image consists of only the first image region; Item 89.
• the output image includes a boundary enhancement image showing a boundary between the first image region and the second image region; 91.
• the medical observation method according to any of items 89 or 90.
• the second range is included in the first range. 93.
• the medical observation method according to any one of items 82 to 92.
• Item 94 A method for medical observation according to any of items 82 to 93, wherein the output image is generated based on the entirety of the second image.
• [Item 96] a display device or devices that can be connected to the display device or devices that can display the output image; generating and outputting the output image corresponding to the characteristics of each of the one or more display devices; Item 96.
• the medical observation method according to any one of items 82 to 95.
• the first image is acquired by photographing the object to be observed irradiated with a first light of a first wavelength band
• the second image is acquired by photographing the object to be observed irradiated with second light having a wavelength band at least partially different from the first wavelength band.
• the medical observation method according to any one of items 82 to 96.
• the first light is visible light; the second light is an excitation light that excites a specific substance to emit fluorescence,
• the first image includes a captured image based on reflected light of the visible light from the observation target,
• the second image includes a captured image based on the fluorescence from the observation target irradiated with the excitation light.
• Item 98. The medical observation method according to item 97.
• an image generating unit that generates an image based on an image signal from an imaging unit that images the observation target, When the image generating unit is switched from one of the first mode and the second mode to the other mode under the control of the control unit, the image generating unit uses at least a part of parameters used in the one mode while maintaining the parameters in the other mode.
• Item 82. The medical observation system according to any one of items 1 to 49 and 66 to 81.
• the parameter is a parameter related to at least one of white balance, color tone, and color mode. 99.
• the parameter is a parameter related to at least one of white balance, color tone, and color mode. Item 102.
• An image generating unit that generates an image based on an image signal from an imaging unit that images the observation target uses at least a part of parameters used in one mode while maintaining the parameters in the other mode when the mode is switched from one of the first mode and the second mode to the other mode under the control of the control unit.
• Item 50 The medical observation method according to any one of items 50 and 82 to 98.
• the parameter is a parameter related to at least one of white balance, color tone, and color mode.
• Item 104 The medical observation method according to item 103.

Landscapes

• Health & Medical Sciences (AREA)
• Life Sciences & Earth Sciences (AREA)
• Surgery (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Biomedical Technology (AREA)
• Optics & Photonics (AREA)
• Pathology (AREA)
• Radiology & Medical Imaging (AREA)
• Biophysics (AREA)
• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Heart & Thoracic Surgery (AREA)
• Medical Informatics (AREA)
• Molecular Biology (AREA)
• Animal Behavior & Ethology (AREA)
• General Health & Medical Sciences (AREA)
• Public Health (AREA)
• Veterinary Medicine (AREA)
• Endoscopes (AREA)

Priority Applications (3)

Applications Claiming Priority (4)

Publications (1)

Family

ID=92674593

Family Applications (1)

Country Status (4)

Citations (6)

• 2024
  • 2024-02-15 JP JP2025505174A patent/JPWO2024185435A1/ja active Pending
  • 2024-02-15 CN CN202480015333.3A patent/CN120916679A/zh active Pending
  • 2024-02-15 WO PCT/JP2024/005183 patent/WO2024185435A1/ja not_active Ceased
  • 2024-02-15 EP EP24766807.2A patent/EP4678087A1/en active Pending

Patent Citations (6)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events