WO2020261741A1 - 投影システム - Google Patents

投影システム Download PDF

Info

Publication number
WO2020261741A1
WO2020261741A1 PCT/JP2020/017625 JP2020017625W WO2020261741A1 WO 2020261741 A1 WO2020261741 A1 WO 2020261741A1 JP 2020017625 W JP2020017625 W JP 2020017625W WO 2020261741 A1 WO2020261741 A1 WO 2020261741A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
marker
projection
captured image
unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/017625
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
朋之 齊藤
木庭　優治
平野　哲司
雄飛 佐々木
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Panasonic Intellectual Property Management Co Ltd
Original Assignee
Panasonic Intellectual Property Management Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Panasonic Intellectual Property Management Co Ltd filed Critical Panasonic Intellectual Property Management Co Ltd
Priority to JP2021527419A priority Critical patent/JP7281632B2/ja
Priority to EP20830848.6A priority patent/EP3968628B1/en
Publication of WO2020261741A1 publication Critical patent/WO2020261741A1/ja
Priority to US17/554,501 priority patent/US20220109814A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3191—Testing thereof
    • H04N9/3194—Testing thereof including sensor feedback
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
    • A61B5/0071—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence by measuring fluorescence emission
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/361—Image-producing devices, e.g. surgical cameras
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B15/00—Special procedures for taking photographs; Apparatus therefor
    • G03B15/14—Special procedures for taking photographs; Apparatus therefor for taking photographs during medical operations
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B17/00—Details of cameras or camera bodies; Accessories therefor
    • G03B17/48—Details of cameras or camera bodies; Accessories therefor adapted for combination with other photographic or optical apparatus
    • G03B17/54—Details of cameras or camera bodies; Accessories therefor adapted for combination with other photographic or optical apparatus with projector
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B21/00—Projectors or projection-type viewers; Accessories therefor
    • G03B21/14—Details
    • G03B21/28—Reflectors in projection beam
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G3/00—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes
    • G09G3/001—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background
    • G09G3/002—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background to project the image of a two-dimensional display, such as an array of light emitting or modulating elements or a CRT
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • H04N9/3179—Video signal processing therefor
    • H04N9/3185—Geometric adjustment, e.g. keystone or convergence
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B2090/364—Correlation of different images or relation of image positions in respect to the body
    • A61B2090/366—Correlation of different images or relation of image positions in respect to the body using projection of images directly onto the body
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37—Surgical systems with images on a monitor during operation
    • A61B2090/373—Surgical systems with images on a monitor during operation using light, e.g. by using optical scanners
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10141—Special mode during image acquisition
    • G06T2207/10152—Varying illumination
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30204—Marker
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2380/00—Specific applications
    • G09G2380/08—Biomedical applications

Definitions

  • the present disclosure relates to a projection system that projects a projected image based on a captured image of a subject.
  • Patent Document 1 discloses an optical imaging system used in the medical field.
  • the optical imaging system of Patent Document 1 has an electronic imaging device that images the surgical field, a projector that projects a visible light image of the imaging result of the surgical field during surgery, and the same optical axis as the electronic imaging device and the projector. It is equipped with an optical element that aligns with.
  • a test sample is arranged in advance before surgery, and the correspondence between the captured image and the projected image on the same optical axis is adjusted while acquiring the captured image of the test sample and generating the corresponding projected image. By doing so, calibration is performed to accurately project the projected image at the time of surgery.
  • An object of the present disclosure is to provide a projection system that can easily adjust the positional relationship between a captured image and a projected image in a projection system that projects a projected image based on the captured image.
  • the projection system in the present disclosure includes an imaging unit, a projection unit, and a control unit.
  • the imaging unit captures the subject and generates a first captured image.
  • the projection unit projects a projected image corresponding to the first captured image onto the subject.
  • the control unit adjusts the first operation mode in which the projected image is generated according to the first captured image and projected onto the projection unit, and the positional relationship in which the position in the first captured image and the position in the projected image are associated with each other. It has two operation modes. In the second operation mode, the control unit projects a marker image including a marker indicating a reference in the positional relationship onto the projection unit, and the imaging unit captures the projected marker image to generate a second captured image. The positional relationship is adjusted based on the marker in the acquired marker image and the marker in the second captured image.
  • the projection system in the present disclosure it is possible to easily adjust the positional relationship between the captured image and the projected image in the projection system that projects the projected image based on the captured image.
  • FIG. 1 Schematic diagram showing the configuration of the surgery support system according to the first embodiment
  • Block diagram showing the configuration of the head device in the surgical support system
  • Functional block diagram illustrating the normal mode of the projection control device in a surgical support system
  • Flowchart to explain normal mode operation in surgical support system Diagram illustrating the state of the surgical field before projection in the surgical support system
  • Diagram illustrating the state of the surgical field after projection in a surgical support system
  • Functional block diagram illustrating the position adjustment mode of the projection control device in the surgical support system
  • Flow chart exemplifying the operation of the position adjustment mode in the surgical support system The figure which illustrates the image data for the projection of the marker image in the position adjustment mode.
  • the figure which illustrates the captured image before correction at the time of operation of a position adjustment mode The figure which illustrates the captured image after correction at the time of operation of a position adjustment mode
  • the figure which shows the display example of the monitor in the position adjustment mode A flowchart illustrating automatic correction processing in the position adjustment mode
  • the figure which illustrates the image data for projection of the marker image for automatic correction The figure which shows the display example in the operation of the 1st position adjustment in the operation support system
  • the figure which shows the display example in the operation of the 2nd position adjustment in the operation support system The figure which shows the display example in the operation of the 3rd position adjustment in the operation support system
  • the figure which shows the display example in the operation of the 4th position adjustment in the operation support system The figure explaining the interpolation method of the projected image in the normal mode of the operation support system
  • Functional block diagram showing modification 1 of the position adjustment mode in the surgical support system The figure which illustrates the image data for projection of the marker image which concerns on modification 2.
  • FIG. 1 is a schematic view showing the configuration of the surgery support system 100 according to the first embodiment.
  • the surgery support system 100 includes a camera 210, a projector 220, and an excitation light source 230.
  • the surgery support system 100 is a system that visually supports surgery performed on a patient by a doctor or the like in an operating room or the like by using a projected image.
  • the patient 120 undergoing surgery is pre-administered with a photosensitizer.
  • a light-sensitive substance is a substance that fluoresces in response to excitation light.
  • the photosensitive substance for example, ICG (indocyanine green) or the like is used.
  • ICG indocyanine green
  • the ICG emits fluorescence in the infrared region having a wavelength of 800 to 860 nm by being irradiated with excitation light in the infrared region having a wavelength of 760 to 780 nm.
  • the light-sensitive substance When the light-sensitive substance is administered to the patient 120, it accumulates in the affected area 130 where the flow of blood or lymph is stagnant. Therefore, it is possible to specify the region of the affected portion 130 by detecting the region that emits fluorescence in response to the irradiation of the excitation light 300.
  • the fluorescence emitted by the affected area 130 is weak or the wavelength band of the fluorescence is in the invisible region or in the vicinity of the invisible region, doctors and the like can see the area of the affected area 130 even by visually observing the surgical field. Difficult to identify. Therefore, in the surgery support system 100, the area of the affected area 130 that emits the fluorescence 310 is specified by using the camera 210. Further, the projected light 320 of visible light is irradiated from the projector 220 to the affected area 130 so that the identified affected area 130 can be visually recognized by humans. As a result, a projected image that visualizes the area of the identified affected area 130 is projected, and it is possible to assist a doctor or the like performing surgery to identify the area of the affected area 130.
  • the configuration of the surgical support system 100 will be described below with reference to FIG.
  • the surgery support system 100 includes a head device 200, a memory 240, and a projection control device 250.
  • the surgery support system 100 includes a mechanism for changing the arrangement of the head device 200, for example, a drive arm mechanically connected to the head device 200, and a set of the surgery support system 100. Equipped with casters for the pedestal to be placed.
  • the head device 200 is arranged vertically above the operating table 110 on which the patient 120 is placed or above at an angle from the vertical direction.
  • the operating table 110 may be provided with a drive mechanism whose height and orientation can be changed.
  • the head device 200 is an example of a projection device in which a camera 210, a projector 220, and an excitation light source 230 are integrally assembled together with an optical system such as a dichroic mirror 201. Details of the configuration of the head device 200 will be described later.
  • the memory 240 is a storage medium that is appropriately accessed when the projection control device 250 executes various operations.
  • the memory 240 is composed of, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory).
  • the memory 240 is an example of a storage unit.
  • the projection control device 250 controls each part constituting the surgery support system 100 in an integrated manner.
  • the projection control device 250 is an example of a control unit.
  • the projection control device 250 is electrically connected to the camera 210, the projector 220, the excitation light source 230, and the memory 240, and outputs control signals for controlling each unit.
  • the projection control device 250 is composed of, for example, a CPU (Central Processing Unit), and realizes its function by executing a predetermined program.
  • the function of the projection control device 250 may be realized by an electronic circuit specially designed or a reconfigurable electronic circuit (FPGA (Field Programmable Gate Array), ASIC (Application Specific IC), etc.).
  • FPGA Field Programmable Gate Array
  • ASIC Application Specific IC
  • the surgery support system 100 includes a display control device 150, a monitor 160, and an operation unit 170.
  • the display control device 150 is composed of, for example, a PC (personal computer) and is connected to the projection control device 250.
  • the display control device 150 includes, for example, a CPU and performs image processing for controlling an image displayed on the monitor 160.
  • the control unit of the system 100 may include a display control device 150 and a projection control device 250. Further, the display control device 150 includes an internal memory (ROM, RAM, etc.) which is an example of a storage unit.
  • the monitor 160 is composed of, for example, a liquid crystal display or an organic EL display, and has a display surface for displaying an image.
  • the monitor 160 is an example of a display unit.
  • the operation unit 170 is an input interface that accepts various user operations input from the user 140.
  • the operation unit 170 includes various operation members such as direction instruction keys, buttons, switches, a keyboard, a mouse, a touch pad, and a touch panel.
  • the user 140 can check the captured image of the camera 210 on the monitor 160, for example, during the operation. In addition, the user 140 can adjust various settings of the projected image.
  • FIG. 2 is a block diagram showing the configuration of the head device 200 in the surgery support system 100.
  • the head device 200 includes an excitation light source 230, a camera 210, a zoom lens 215, an optical filter 216, a projector 220, a projection lens 221 and a dichroic mirror 201, and a mirror 202.
  • the head device 200 is arranged at a position where the distance (height) to the subject such as the surgical field 135 is, for example, 1 m.
  • the excitation light source 230 is a light source device that irradiates an excitation light 300 for making a light-sensitive substance fluorescently emit light.
  • the excitation light source 230 irradiates the excitation light 300 having a wavelength band (for example, in the vicinity of 760 to 780 nm) including the excitation wavelength of the ICG.
  • the excitation light source 230 is an example of an illumination unit.
  • the excitation light source 230 switches the irradiation of the excitation light 300 ON / OFF according to the control signal from the projection control device 250.
  • the excitation light source 230 may be configured separately from the head device 200.
  • the camera 210 captures a subject including the surgical field 135 of the patient 120 and generates an captured image.
  • the camera 210 transmits image data indicating the generated captured image to the projection control device 250.
  • the camera 210 includes an image sensor 211 for infrared light, an image sensor 212 for visible light, a prism 213, and an optical filter 214.
  • Each image sensor 211,212 is composed of, for example, a CMOS image sensor or a CCD image sensor, respectively.
  • Prism 213 has an optical characteristic of reflecting an optical component in the infrared region and reflecting an optical component in the visible region (or outside the infrared region), for example, in incident light.
  • An image sensor 211 for infrared light is arranged on the reflecting surface side of the prism 213. Further, the prism 213 is arranged between the image sensor 212 for visible light and the zoom lens 215.
  • the optical filter 214 is composed of, for example, a bandpass filter or a lowpass filter, transmits light in the visible region, and blocks light outside the visible region (for example, ultraviolet region).
  • the optical filter 214 is arranged between the image sensor 212 for visible light and the prism 213.
  • the image pickup sensor 211 for infrared light performs imaging with infrared light (an example of invisible light) including a wavelength band of 800 nm to 860 nm, which is fluorescence of ICG, and generates an invisible light image as an image.
  • the image sensor 211 for infrared light may include a filter or the like that blocks light other than infrared light.
  • the image sensor 211 for infrared light is an example of an invisible image pickup unit.
  • the image sensor 212 for visible light performs imaging in visible light including a part or the whole of the visible region, and generates, for example, a monochrome visible light image as an image.
  • the image sensor 212 for visible light is an example of a visible image pickup unit.
  • the visible imaging unit is not limited to monochrome, and may be configured to be capable of capturing an RGB captured image, for example.
  • the visible imaging unit may be composed of one CMOS sensor or the like in which an RGB color filter is provided for each pixel, or may be composed of three CMOS sensors or the like for capturing each color light of RGB. You may.
  • the prism 213 and the optical filter 214 are examples of an internal optical system provided inside the camera 210.
  • the internal optical system of the camera 210 is not limited to the above.
  • the optical path length adjusting member may be arranged between the imaging sensors 211 and 212 and the prism 213.
  • the prism 213 may have an optical characteristic that reflects invisible light such as infrared light and mainly transmits visible light, instead of the above-mentioned optical characteristics, or an optical that reflects both lights. It may have characteristics.
  • the arrangement of the imaging sensors 211 and 212 is appropriately changed according to the optical characteristics of the prism 213.
  • the zoom lens 215 is attached to the camera 210 and collects light from the outside inside the camera 210.
  • the zoom lens 215 adjusts the angle of view (zoom value), depth of field, focus, and the like of the camera 210.
  • the zoom lens 215 includes various lens elements and an aperture.
  • the zoom value by the zoom lens 215, the F value of the aperture, and the like can be set from the outside, for example.
  • the zoom lens 215 is an example of an imaging optical system.
  • the imaging optical system is not limited to the zoom lens 215, and may include, for example, the internal optical system of the camera 210 and various external optical elements, or may be incorporated into the camera 210 as an internal optical system.
  • the optical filter 216 is arranged on the incident surface of the zoom lens 215, for example, as shown in FIG.
  • the optical filter 216 is composed of a band cut filter that blocks wavelength band components of 680 to 825 nm including a wavelength of 760 to 780 nm of excitation light in incident light.
  • the projector 220 is, for example, a projector such as a DLP system, a 3LCD system, or an LCOS system.
  • the projector 220 emits projected light 315 so as to project a projected image based on a video signal input from the projection control device 250 with visible light.
  • the projector 220 is an example of a projection unit.
  • the projector 220 includes, for example, a light source, an image forming unit, an internal optical system, and the like.
  • the light source of the projector 220 is composed of, for example, an LD (semiconductor laser) or an LED (Light Emitting Diode).
  • the image forming unit of the projector 220 includes a spatial light modulation element such as a DMD (Digital Micromirror Device) or an LCD (Liquid Crystal Display), and the image forming surface of the spatial light modulation element is based on a video signal from the projection control device 250. Form an image.
  • the projector 220 spatially modulates the light from the light source according to the formed image to generate the projected light 315, which is emitted from the internal optical system.
  • the projector 220 may have a projection control circuit that realizes functions unique to the projector 220, such as keystone correction and lens shift functions. Further, each of the above functions may be realized in the projection control device 250. Further, the projector 220 may be a laser scanning type, or may be configured to include a MEMS mirror or a galvano mirror that can be driven in the scanning direction.
  • the projection lens 221 is arranged so as to be optically coupled to the internal optical system of the projector 220.
  • the projection lens 221 is composed of, for example, a teleconversion, and extends the focal length of the projector 220 to the telephoto side.
  • the mirror 202 is arranged between the projection lens 221 and the dichroic mirror 201.
  • various optical elements may be arranged in the optical path from the projector 220 to the dichroic mirror 201.
  • the dichroic mirror 201 is an example of a light guide unit having optical characteristics of transmitting or reflecting light selectively incident on the wavelength band of light.
  • the transmittance of infrared light is set to 100% and the reflectance is set to 0% within the tolerance range.
  • the dichroic mirror 201 is set with the reflectance and transmittance of visible light so as to transmit visible light within a range smaller than the reflectance of visible light.
  • the visible light transmittance of the dichroic mirror 201 is preferably 5% or less.
  • the visible light reflectance of the dichroic mirror 201 is 99% and the visible light transmittance is 1%.
  • the visible light that can be imaged by the image sensor 212 for visible light is 5% or less by the dichroic mirror 201.
  • the dichroic mirror 201 In order to reduce the content to 5% or less, not only the dichroic mirror 201 but also members on the optical path of imaging may be combined to realize the dichroic mirror 201.
  • an optical filter 216, a prism 213, etc. A filter may be provided in the above to reduce the total content to 5% or less.
  • the dichroic mirror 201 transmits the fluorescence 310 or the like toward the camera 210 through the zoom lens 215 or the like due to the above optical characteristics, while the majority (half) of the projected light 315 emitted from the projector 220 or the like is transmitted. Above) is reflected.
  • the reflected projected light 320 is applied onto the surgical field 135.
  • the dichroic mirror 201 has an optical axis of incident light incident on the camera 210 such as a fluorescence 310 from the surgical field 135 and an optical axis of the projected light 320 that projects a projected image on the surgical field 135. , Guide the light so that it coincides with the optical axis J1. This makes it possible to reduce the positional deviation of the projected image based on the captured image of the camera 210.
  • a tolerance may be set as appropriate for the alignment of the optical axes in the present disclosure.
  • the optical axes may match with a margin of error within a range of ⁇ 5 degrees in angle or within a range of 1 cm between optical axes.
  • the optical characteristics of the dichroic mirror 201 can be appropriately set according to the fluorescence characteristics of the light-sensitive substance used.
  • FIG. 3 is a functional block diagram illustrating a normal mode of the projection control device 250.
  • the surgery support system 100 and the projection control device 250 have a normal mode which is an example of the first operation mode and a position adjustment mode which is an example of the second operation mode.
  • the projection control device 250 has, as a functional configuration, a position correction unit 251 for an invisible light image, an image generation unit 252, a position correction unit 253 for a visible light image, an image superimposition unit 254, a marker generation unit 255, and a correction calculation.
  • the unit 256 is provided.
  • Each position correction unit 251,253 performs a process of correcting the position of the entire image according to the correction information set in advance on the image captured by the camera 210.
  • the correction information is an example of information indicating the positional relationship between the captured image and the projected image, and includes various parameters that define various coordinate transformations.
  • the image generation unit 252 performs various image processing on the captured image corrected by the position correction unit 251 for the invisible light image to generate image data (video signal) indicating the projected image.
  • the image processing includes binarization or multi-value conversion, color conversion, and the like.
  • the image superimposing unit 254 performs image composition for superimposing the captured image and the projected image, and outputs the image to the display control device 150 or the monitor 160.
  • the marker generation unit 255 and the correction calculation unit 256 operate in the position adjustment mode. Various operation modes and various functions of the projection control device 250 will be described later.
  • the normal mode is an operation mode in which the operation support system 100 performs a basic projection operation for supporting the operation.
  • FIG. 4 is a flowchart for explaining the operation of the normal mode in the surgery support system 100.
  • FIG. 5A shows the state of the surgical field 135 in the surgical support system 100 before performing the projection operation in the normal mode.
  • FIG. 5B shows a state in which a projection operation is performed on the surgical field 135 of FIG. 5A.
  • Each process shown in the flowchart of FIG. 4 is executed by the projection control device 250.
  • the projection control device 250 drives the excitation light source 230 and irradiates the surgical field 135 with the excitation light 300 as shown in FIG. 5A (S1).
  • the affected portion 130 in the surgical field 135 emits fluorescence, and the fluorescence 310 from the affected portion 130 is incident on the head device 200.
  • the fluorescence 310 transmits the dichroic mirror 201 and the optical filter 216 of the camera 210.
  • the camera 210 receives the fluorescence 310 at the image sensor 211 for infrared light.
  • the reflected light of the excitation light 300 is blocked by the optical filter 216.
  • the projection control device 250 controls, for example, the camera 210 to image the surgical field 135, and acquires the captured image from the camera 210 (S2).
  • the captured image acquired in step S2 includes a fluorescence image generated by the light reception of the fluorescence 310 emitted by the affected portion 130.
  • the projection control device 250 functions as a position correction unit 251 and an image generation unit 252, and performs image processing for generating a projection image based on the acquired captured image (S3).
  • the projection control device 250 generates an image corresponding to the fluorescence image in the captured image and outputs it to the projector 220 as a video signal.
  • the projection control device 250 first, as the position correction unit 251, refers to the correction information stored in the memory 240 and set in advance, shifts, rotates, and enlarges / reduces the acquired image. Perform coordinate conversion such as. As a result, the position of the image is corrected.
  • the projection control device 250 may further correct image distortion and the like.
  • the projection control device 250 binarizes the distribution of the light receiving intensity in the corrected captured image based on a predetermined threshold value, and considers it as a region of the fluorescent image in the captured image. Identify the area to be.
  • the image generation unit 252 sets different colors inside and outside the specified region to generate an image representing the specific region according to the fluorescence image in the captured image (S3).
  • the inside of the specified area is set to a chromatic color such as blue
  • the outside of the specified area is set to an achromatic color such as white.
  • the projection control device 250 controls the projector 220 so as to project a projected image based on the generated video signal (S4).
  • the projector 220 Under the control of the projection control device 250, the projector 220 generates projected light 315 representing a projected image corresponding to the image signal from the projection control device 250, and emits the projected light 315 to the dichroic mirror 201 via the projection lens 221. (See Fig. 2).
  • the dichroic mirror 201 reflects (most of) the projected light 315, which is visible light, and emits the projected light 320 along the optical axis J1.
  • the head device 200 irradiates the surgical field 135 with the projected light 320, and the projected image G 320 is projected onto the affected portion 130 in the surgical field 135.
  • the projected image G320 is, for example, a monochromatic image.
  • the above processing is repeatedly executed in a predetermined cycle (for example, 1/60 to 1/30 seconds).
  • the projection control device 250 identifies the region of the affected portion 130 that fluoresces based on the image captured by the camera 210, and the projected image G320 of visible light is projected onto the affected portion 130 from the projector 220.
  • the surgical support system 100 can visualize the affected area 130, which is difficult to visually recognize.
  • the surgery support system 100 allows doctors and the like to visually recognize the real-time state of the affected area 130.
  • the projection control device 250 may generate a multi-gradation projection image by, for example, determining the region of the fluorescence image in the captured image in multiple stages using a plurality of threshold values. Further, the projection control device 250 may generate a projection image so as to continuously reproduce the distribution of the light receiving intensity in the captured image. Further, the projected image may be generated in a plurality of colors or full colors.
  • the surgical support system 100 captures images of the surgical field and the like in visible light in addition to imaging a fluorescent image (S2 in FIG. 4) for generating a projected image as described above. Take an image.
  • the visible imaging function in the surgery support system 100 will be described with reference to FIG.
  • the visible light 330 incident on the head device 200 of the surgery support system 100 includes light reflected by external light on a subject such as a surgical field, reflected light of projected light 320, and the like.
  • the visible light 330 is incident on the dichroic mirror 201 in the head device 200.
  • the dichroic mirror 201 of the present embodiment transmits a part of the incident visible light 330 and causes the dichroic mirror 201 to enter the zoom lens 215 via the optical filter 216.
  • the optical filter 216 of the present embodiment transmits the incident visible light 330 at a predetermined transmittance.
  • the zoom lens 215 adjusts the luminous flux of the incident visible light 330 according to the set zoom value and aperture value, and causes the incident light to enter the camera 210.
  • the prism 213 transmits the incident visible light 330.
  • the image sensor 212 for visible light receives the visible light 330 transmitted through the prism 213.
  • the image sensor 212 for visible light captures an image of visible light 330 from a subject or the like.
  • the camera 210 outputs a visible light image of the image capture result of the image sensor 212 for visible light to, for example, a display control device 150 and / or a projection control device 250 (see FIG. 1).
  • the prism 213 reflects the incident infrared light and guides it to the image pickup sensor 211 for infrared light. According to the camera 210, the invisible light image of the infrared light image sensor 211 and the visible light image of the visible light image sensor 212 can be simultaneously captured.
  • the above visible imaging function is used, for example, to display or record the state of the surgical field during surgery.
  • the display control device 150 (FIG. 1) displays a visible light image on the monitor 160 or records it in a memory 240 or the like.
  • various display modes can be set in the surgery support system 100 by performing image processing such as superimposing a non-visible light image on a visible light image.
  • the visible imaging function can also be used to correct the misalignment of the projected image G320 (FIG. 5B).
  • FIG. 6 is a functional block diagram illustrating a position adjustment mode of the projection control device 250 in the system 100.
  • the surgery support system 100 visualizes the affected area 130, which is difficult to see visually during surgery, by projecting the projected image G320 from the projector 220 based on the image captured by the camera 210 in the normal mode (FIGS. 5A and 5B). reference).
  • the position adjustment is performed in advance by aligning the captured image of the camera 210 with the projected image of the projector 220.
  • a mode is provided. Positioning in the position adjustment mode is performed sequentially, for example, every time an operation is scheduled.
  • a projected image is once generated based on the captured image of the sample, and the projected image is projected onto the sample, and the sample is projected.
  • the deviation of the projected image regenerated after feedback with respect to the sample causes a situation in which it becomes difficult to converge, and it is difficult to easily adjust the positional relationship between the captured image and the projected image. is there.
  • the operation order of the camera 210 and the projector 220 is reversed so as to change the roles of the camera 210 and the projector 220 from the above-mentioned normal mode. That is, first, as shown in FIG. 6, the projector 220 projects a predetermined marker image generated by the marker generation unit 255 onto, for example, a white chart 400 composed of a white plate. Next, the camera 210 takes an image of the projected marker image, and the correction calculation unit 256 calculates the position shift of the marker image in the captured image with reference to the marker image of the projection source.
  • FIG. 7 is a flowchart illustrating the operation of the position adjustment mode in the surgery support system 100. This flowchart starts when, for example, the operation unit 170 inputs a user operation for starting the position adjustment mode. Each process shown in this flowchart is executed by, for example, the projection control device 250.
  • the projection control device 250 performs an automatic correction process (S10).
  • the automatic correction process automatically adjusts the positional relationship between the projected image and the captured image using the marker image, and the position correction unit 253 and the position correction so that the positional deviation of the captured image is within a predetermined margin of error. This is a process for initializing the unit 251. The details of the automatic correction process will be described later.
  • the projection control device 250 functions as a marker generation unit 255 and controls the projector 220 so as to project a marker image (S11).
  • the projection control device 250 reads out the image data for projection corresponding to the marker image from the memory 240, and outputs the read image data to the projector 220.
  • FIG. 8 illustrates the marker image G1 shown by the image data D1 for projection in step S11.
  • the image data for projection such as the image data D1 has projected coordinates (Xp, Yp) which are two-dimensional coordinates that define the position in each projected image.
  • the marker image G1 defined by the image data D1 includes a projection marker G10 and a region other than the projection marker G10.
  • the projection marker G10 is an example of a marker set at a reference position in the projection coordinates (Xp, Yp).
  • the projection control device 250 acquires an captured image obtained by capturing the projected marker image G1 from the camera 210, and performs processing as the position correction unit 253 (S12).
  • the projection control device 250 acquires the captured image corrected by, for example, the automatic correction process (S10) by the process as the position correction unit 253.
  • 9A and 9B show the captured images Im1 before and after the correction in step S12.
  • FIG. 9A exemplifies the captured image Im1 captured by the camera 210 when the marker image G1 of FIG. 8 is projected.
  • FIG. 9B illustrates a state in which the position correction unit 253 initially set in step S10 has processed the captured image Im1 of FIG. 9A.
  • the projection marker G10 reflected in the captured image Im1 is referred to as an “imaging marker”.
  • step S12 the process of the position correction unit 253 changes the image captured from the imaged coordinates (Xi, Yi) at the time of imaging of the camera 210 to the corrected imaged coordinates (Xc, Yc) according to the information indicating the preset positional relationship. It is performed so as to perform coordinate conversion of Im1.
  • the position of the imaging marker Im10 on the corrected imaging coordinates (Xc, Yc) is shown in the figure because the imaging coordinates (Xi, Yi) could not be corrected by the automatic correction processing (S10). It deviates from the position of the projection marker G10 of 8.
  • the projection control device 250 refers to the image data D1 for projection of the memory 240 as the correction calculation unit 256, and the imaging marker Im10 on the imaging coordinates (Xc, Yc) and the projection coordinates (Xp, Yp).
  • the amount of deviation indicating the positional deviation between the above projection markers G10 is calculated (S13).
  • the projection control device 250 performs image analysis on the captured image Im1 and determines the position of a specific portion of the imaging marker Im10 corresponding to the reference position of the projection marker G10, that is, the marker position on the imaging coordinates (Xc, Yc). To detect.
  • the projection control device 250 refers to the image data D1 for projection as the image superimposing unit 254, generates a superposed image Im2 in which the projection marker G10 is superimposed on the captured image Im1, and communicates with, for example, the display control device 150.
  • the superimposed image Im2 is displayed on the monitor 160 via the monitor 160 (S14).
  • a display example of the monitor 160 in step S14 is shown in FIG.
  • the display example of FIG. 10 exemplifies the superimposed image Im2 corresponding to the captured image Im1 of FIG. 9B.
  • the projection control device 250 displays the calculated deviation amount ( ⁇ X, ⁇ Y) together with the superimposed image Im2 in step S14, for example, as in this display example.
  • the display as shown in FIG. 10 is performed to make the user 140 confirm whether or not to further adjust the position of the correction.
  • the user 140 can appropriately input a user operation for performing position adjustment from the operation unit 170.
  • the projection control device 250 determines whether or not the user operation input from the operation unit 170 has input the position adjustment operation (S15).
  • the projection control device 250 updates the information set in the position correction unit 253 and the position correction unit 251 according to the input user operation (S16), and steps.
  • the processing after S12 is executed again.
  • the imaging coordinates (Xc, Yc) of the captured image Im1 are further corrected (S12), and the display of the superimposed image Im2 and the like is updated (S14).
  • the desired correction is obtained by repeating, for example, the position adjustment operation, the user 140 inputs an operation for completing the position adjustment mode to the operation unit 170.
  • the projection control device 250 determines that the input user operation is not the position adjustment operation (NO in S15), and saves the adjustment result (S17). At this time, the projection control device 250 records various information regarding the position correction unit 253 for the visible light image and the position correction unit 251 for the invisible light image in the memory 240. After that, the process according to this flowchart ends.
  • the position adjustment is performed with reference to the projection marker G10, such that the correction of the imaging coordinates (Xc, Yc) is repeated while continuing to project the constant marker image G1 from the projector 220.
  • the user 140 can easily reach a desired correction state by performing a position adjustment operation so that the image pickup marker Im10 is brought closer to the projection marker G10 on the superimposed image Im2 of the monitor 160.
  • steps S10 to S16 the settings are similarly updated between the position correction unit 253 for the visible light image and the position correction unit 251 for the invisible light image, and both positions are sequentially updated.
  • the adjustment result for the correction units 251,253 can be the same (S17).
  • the projection control device 250 updates the setting of one position correction unit 253 in steps S10 to S16 so that both position correction units 251, 253 are finally the same in step S17.
  • the other position correction unit 251 may be set.
  • FIG. 11 is a flowchart illustrating the automatic correction process in the position adjustment mode.
  • the projection control device 250 uses, for example, a marker image for automatic correction instead of the marker image G1 in FIG. 8 to perform the same process as in steps S11 to S13 described above (S21 to S23). ).
  • FIG. 12 illustrates the image data D1a of the marker image G1a for automatic correction.
  • Yp) (xa ⁇ xb, ya ⁇ yb) includes marker points P1 to P4.
  • the size of each marker point P1 to P4 is appropriately set for one or more pixels.
  • the positions of the marker points P1 to P4 are examples of reference positions in the projected coordinates (Xp, Yp).
  • the projection marker G10a of this example is set to have a lower brightness than the region other than the projection marker G10a in the marker image G1a.
  • the entire region other than the projection marker G10a is set to white (that is, the highest brightness), and the projection marker G10a is set to black (that is, the lowest brightness).
  • the projection control device 250 first projects the marker image G1a for automatic correction as described above onto the projector 220 (S21). Next, the projection control device 250 acquires the captured image of the marker image G1a at the imaging coordinates (Xi, Yi) at the time of imaging by the camera 210 (S22), and the amount of deviation of the marker position on the imaging coordinates (Xi, Yi). Is calculated (S23).
  • the projection control device 250 sets initial correction information in the position correction unit 253 and the position correction unit 251 based on the calculated deviation amount (S24).
  • step S24 the projection control device 250 sets the deviation amount of the image pickup marker Im10 in the image pickup coordinates (Xi, Yi) corrected by the position correction unit 253 and the position correction unit 251 to be equal to or less than the upper limit value indicating the margin of error.
  • the correction information for the imaging coordinates (Xi, Yi) is calculated.
  • the correction information is defined by, for example, various parameters representing coordinate transformations for imaging coordinates (Xi, Yi), and includes, for example, translation, rotation, and enlargement / reduction parameters.
  • step S10 After the initial setting (S24) of the position correction unit 253 and the position correction unit 251, the projection control device 250 ends step S10 in FIG. 7 and proceeds to step S11.
  • the imaging coordinates (Xi, Yi) are automatically corrected with reference to the projection marker G10a projected from the projector 220, and the position adjustment in the surgery support system 100 can be facilitated. it can.
  • the above automatic correction process may be executed while displaying the superimposed image on the monitor 160, as in step S14 of FIG.
  • the projection control device 250 may display the deviation amount on the monitor 160 as in FIG.
  • the projection control device 250 may set the position correction unit 253 and the position correction unit 251 while gradually changing various parameters of the correction information.
  • the projection marker G10 has the same marker points P0 to P4 as in FIG. 12, as well as a guide line G11 for rotation adjustment and a guide line for scaling adjustment. Further includes G12.
  • the guide lines G11 and G12 are set to the same color as the marker points P0 to P4, for example.
  • the guide line G11 for rotation adjustment is provided radially from the central marker point P0.
  • the guide line G12 for scaling adjustment is provided in a rectangular shape having four marker points P1 to P4 as four corners.
  • the guide lines G11 and G12 are provided at predetermined intervals from the marker points P0 to P4, respectively.
  • the predetermined interval is set in a range larger than the upper limit of the amount of deviation assumed between the reference position of the projection marker G10 and the marker position of the imaging marker Im10, for example, in consideration of the margin of error in the automatic correction process (S10). ..
  • the projection control device 250 sets, for example, the color of the projection marker G10 to a color different from that at the time of projection (S11) (see FIG. 10). ).
  • the projection marker G10 is set to black in step S11 and light blue in step S14.
  • the imaging marker Im10 appears black in the superimposed image Im2, making it easy to distinguish between the projection marker G10 and the imaging marker Im10.
  • the display method for distinguishing the markers G10 and Im10 in the superimposed image Im2 is not limited to the above, and for example, a line type change or a blinking display can be adopted.
  • FIGS. 13A to 13D show display examples in the first to fourth position adjustment operations.
  • FIGS. 13A to 13D show display examples in the first to fourth position adjustment operations.
  • an example of the position adjusting method performed by repeating the processes of steps S12 to S15 by the user operation will be described.
  • FIG. 13A illustrates a state in which the first position adjustment operation is performed after the superimposed display shown in FIG.
  • first as the first position adjustment operation, the alignment in the X and Y directions is performed.
  • the imaging marker Im10 in the X and Y directions so that the marker positions of the corresponding imaging markers Im10 match with reference to the central marker point P0 of the projection marker G10, X and Y can be easily performed.
  • the direction can be aligned.
  • the imaging marker Im10 is rotated around the matched marker point P0.
  • Such adjustment can be easily performed by paying attention to the guide line G11 for rotation adjustment radially from the marker point P0.
  • the third and fourth position adjustments are performed so as to enlarge or reduce the imaging marker Im10 while fixing the position of the marker point P0.
  • Such position adjustment can be easily performed by paying attention to the guide line G12 for scaling adjustment arranged so as to surround the marker point P0.
  • the enlargement / reduction adjustment may be performed separately in the X direction and the Y direction. Further, both the X and Y directions may be enlarged or reduced at the same time to match the guide lines in one direction, and then the remaining directions may be finely adjusted.
  • the position adjustment method as described above is not limited to the user operation, and may be automatically executed by the projection control device 250 or the like.
  • the above position adjustment method may be applied to the automatic correction process (S10).
  • the deviation amount ( ⁇ X, ⁇ Y) may be displayed as in FIG.
  • the projection control device 250 sequentially calculates the deviation amount (S13) and updates the displayed deviation amount.
  • the projection control device 250 may change the display mode according to the magnitude of the deviation amount. For example, if the deviation amount is equal to or more than a predetermined threshold value, it may be displayed in red, and if it is less than the threshold value, it may be displayed in green.
  • FIG. 14 is a diagram illustrating an interpolation method of the projected image G3 in the normal mode of the surgical support system 100.
  • the position correction unit 251 of the projection control device 250 converts the coordinates from the imaging coordinates (Xi, Yi) to the projection coordinates (Xp, Yp) according to the setting of the position adjustment mode as described above. By doing so, the position of each part in the captured image Im1 is corrected so as to shift as a whole on the projected image G3. At this time, as shown in FIG. 14, a blank region G30 that does not correspond to the captured image Im1 may occur in the entire region of the projected coordinates (Xp, Yp).
  • the image generation unit 252 of the present embodiment generates the projected image G3 so as to set the blank area G30 as described above to white and interpolate.
  • the projected light emitted according to the blank region G30 in the projected image G3 at the time of projection from the projector 220 can be utilized as illumination for the surgical field 135 and the like.
  • the image generation unit 252 also sets the region other than the portion specified to correspond to the affected portion 130 by binarization or the like in the projected image G3 to be white. This makes it easier to secure lighting that utilizes the projected light.
  • an intermediate gradation that can be used as illumination light having a brightness higher than that of the region corresponding to the captured image may be appropriately used.
  • the surgery support system 100 of the present embodiment includes a camera 210 which is an example of an imaging unit, a projector 220 which is an example of a projection unit, and a projection control device 250 which is an example of a control unit. ..
  • the camera 210 images a subject such as the affected area 130 to generate a first captured image (S2).
  • the projector 220 projects a projected image G320 corresponding to the first captured image onto the subject (S4).
  • the projection control device 250 has a normal mode and a position adjustment mode.
  • the normal mode is an example of a first operation mode that generates a projected image according to the first captured image.
  • the position adjustment mode is an example of a second operation mode for adjusting the positional relationship in which the position in the first captured image and the position in the projected image are associated with each other.
  • the projection control device 250 causes the projector 220 to project a marker image G1 including a projection marker G10, which is an example of a marker indicating a reference in a positional relationship (S11).
  • the projection control device 250 acquires the captured image Im1 (second captured image) of the projected marker image G1 from the camera 210 (S12).
  • the projection control device 250 adjusts the above positional relationship based on the projection marker G10 in the marker image G1 and the marker in the captured image Im1, that is, the imaging marker Im10 (S15, S16).
  • the marker image G1 projected from the projector 220 is used as a reference, and the positional relationship between the projected image G320 and the captured image in the normal mode can be easily adjusted. ..
  • the projection control device 250 operates as the position correction unit 251 and the image generation unit 252 in the normal mode, and refers to the positional relationship adjusted in the position adjustment mode with the position in the first captured image.
  • the projected image G320 is generated so as to correspond to the position in the projected image G320 (S3).
  • the surgery support system 100 further includes a monitor 160, which is an example of a display unit that displays an image.
  • the projection control device 250 controls the monitor 160 so that the projection marker G10 in the marker image G1 is superimposed on the captured image Im1 and displayed (S14).
  • the user 140 can confirm the positional deviation of the imaging marker Im10 with respect to the projection marker G10 on the monitor 160, and can facilitate the position adjustment.
  • the surgery support system 100 further includes an operation unit 170 for inputting a user operation.
  • the projection control device 250 adjusts the positional relationship according to the user operation input from the operation unit 170 (S15, S16). According to the system 100, the user 140 can easily perform position adjustment so as to obtain a desired positional relationship.
  • the projection control device 250 calculates the amount of deviation between the position of the projection marker G10 in the marker image G1 and the position of the imaging marker Im10 in the captured image Im1 in the position adjustment mode (S13).
  • the system 100 may display the calculated deviation amount on the monitor 160, or may use it for automatic correction.
  • the camera 210 captures an invisible light image, which is an example of the first captured image, based on infrared light, which is an example of the first light having the first wavelength band. Further, the camera 210 is based on visible light which is an example of the second light having a second wavelength band different from the first wavelength band, and the captured image Im1 is used as a visible light image which is an example of the second captured image. To image.
  • the surgery support system 100 can realize various types of support by using the first and second images captured by the camera 210.
  • the subject is a living body such as a patient, and includes an affected portion 130 that fluoresces in the first wavelength band.
  • the surgery support system 100 further includes an excitation light source 230, which is an example of a light source unit.
  • the excitation light source 230 irradiates the excitation light 300 that excites fluorescence emission.
  • the affected part 130 that emits fluorescence can be visualized by the projection image G320 to realize support for surgery and the like.
  • the projection marker G10 has a lower brightness than the region other than the projection marker G10 in the marker image G1.
  • the projection marker G10 is set to black. Such a projection marker G10 makes it easy to confirm the projected position and facilitates position adjustment.
  • the brightness of the position not corresponding to the captured image such as the blank area G30 is higher than the brightness of the position corresponding to the captured image.
  • the projection control device 250 sets the brightness of the blank region G30 to white. With such a projected image G3, the projected light in the blank region G30 can be utilized for illumination.
  • the first embodiment has been described as an example of the technology disclosed in the present application.
  • the technique in the present disclosure is not limited to this, and can be applied to embodiments in which changes, substitutions, additions, omissions, etc. are made as appropriate. It is also possible to combine the components described in the first embodiment to form a new embodiment. Therefore, other embodiments will be illustrated below.
  • the automatic correction process (S10 in FIG. 7) is performed during the operation of the position adjustment mode has been described, but the automatic correction process may be omitted.
  • the operation of the position adjustment mode in the present embodiment may omit the acceptance of the user operation after step S15, or may be completed by executing the automatic correction process.
  • the position adjustment mode may be executed by using a non-visible light image instead of the visible light image.
  • the first modification will be described with reference to FIG.
  • FIG. 15 is a functional block diagram showing a modification 1 of the position adjustment mode in the surgery support system 100.
  • the fluorescence chart 410 is adopted as the subject.
  • the fluorescence chart 410 is made of a material that totally reflects the wavelength component of 830 nm that can be contained in, for example, white projected light.
  • the deviation amount is calculated for the correction result of the position correction unit 251 for the invisible light image instead of the position correction unit 253 for the visible light image.
  • the position adjustment mode of the present modification as described above and the position adjustment mode of the first embodiment may be used in combination.
  • the difference of the correction information can be set between the position correction unit 251 for the invisible light image and the position correction unit 253 for the visible light image.
  • the difference in the correction information makes it possible to deal with chromatic aberration of magnification between the visible light image and the invisible light image.
  • the difference in the correction information may be set by the position adjustment mode of this modification, for example, at the time of shipment from the factory.
  • the projection control device 250 acquires a third captured image obtained by capturing the marker image projected from the projector 220 by the camera 210 based on the first light such as invisible light in the position adjustment mode. You may. Based on the acquired third captured image, the projection control device 250 has a positional relationship of associating a position in the invisible light image (first captured image) with a position in the projected image, and a visible light image (second captured image). The difference between the position in the image) and the positional relationship in which the position in the marker image is associated may be set.
  • marker images G1 and G1a are illustrated, but the marker images G1 and G1a are not limited to the above examples, and various forms can be adopted. Such a modification 2 will be described with reference to FIG.
  • FIG. 16 illustrates image data D2 for projection of the marker image G2 according to the second modification.
  • the marker image G2 of this modification includes a plurality of marker points P20 arranged in a grid pattern as the projection marker G20. For example, 20 ⁇ 20 marker points P20 are arranged at predetermined intervals over the entire projected coordinates (Xp, Yp).
  • the form of the projection marker G20 is not particularly limited to the above, and for example, the number of marker points P20 other than the above may be set, and markers of various forms are arranged in place of or in addition to the marker points P20. May be done.
  • the projection marker G20 of this modification for example, when the projected image (or the captured image) is distorted, the projection marker G20 and the corresponding imaging marker can be compared in the same manner as in the above embodiment. , The amount of distortion can be measured. At this time, the amount of strain can be measured at various points in the projected coordinates (Xp, Yp) where the marker points P20 are arranged.
  • infrared light has been described as an example of invisible light, but the invisible light is not limited to infrared light and may be ultraviolet light. Further, the invisible light is not necessarily limited to light having a wavelength band in the invisible region, and may include, for example, weak fluorescence in the red region that emits light based on excitation light in the blue region. At this time, the visible light to be the target of the projected image and the visible imaging may be green light or the like.
  • the imaging unit, the projection unit, and the control unit of the present embodiment have the first light used for imaging in the normal mode as an infrared region and the second light used for imaging in the position adjustment mode as an ultraviolet region. May be configured.
  • the imaging unit, the projection unit, and the control unit of the present embodiment have the first light used for imaging in the normal mode as an infrared region and the second light used for imaging in the position adjustment mode as an ultraviolet region. May be configured.
  • the projection system in the present disclosure is not limited to this.
  • the projection system in the present disclosure may be applied. it can.
  • a fluorescent material is applied to an object whose state change cannot be visually confirmed, kneaded or poured, and the camera 210 is used. It may be the target of imaging.
  • the projection system in the present disclosure can be applied when working on a subject whose state change is difficult to visually confirm, such as a medical application, a construction site, a mining site, a construction site, or a factory for processing materials.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Surgery (AREA)
  • Public Health (AREA)
  • Animal Behavior & Ethology (AREA)
  • Veterinary Medicine (AREA)
  • Molecular Biology (AREA)
  • Theoretical Computer Science (AREA)
  • Geometry (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Pathology (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Hardware Design (AREA)
  • Biophysics (AREA)
  • Projection Apparatus (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Transforming Electric Information Into Light Information (AREA)
PCT/JP2020/017625 2019-06-25 2020-04-24 投影システム Ceased WO2020261741A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
JP2021527419A JP7281632B2 (ja) 2019-06-25 2020-04-24 投影システム
EP20830848.6A EP3968628B1 (en) 2019-06-25 2020-04-24 Projection system
US17/554,501 US20220109814A1 (en) 2019-06-25 2021-12-17 Projection system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2019-117375 2019-06-25
JP2019117375 2019-06-25

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/554,501 Continuation US20220109814A1 (en) 2019-06-25 2021-12-17 Projection system

Publications (1)

Publication Number Publication Date
WO2020261741A1 true WO2020261741A1 (ja) 2020-12-30

Family

ID=74060542

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/017625 Ceased WO2020261741A1 (ja) 2019-06-25 2020-04-24 投影システム

Country Status (4)

Country Link
US (1) US20220109814A1 (https=)
EP (1) EP3968628B1 (https=)
JP (1) JP7281632B2 (https=)
WO (1) WO2020261741A1 (https=)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2022108089A (ja) * 2021-01-12 2022-07-25 ｉ－ＰＲＯ株式会社 投影装置
WO2022163835A1 (ja) * 2021-01-29 2022-08-04 パナソニックＩｐマネジメント株式会社 投写制御方法、および、投写制御装置
WO2022253336A1 (zh) * 2021-06-03 2022-12-08 青岛海信激光显示股份有限公司 激光投影设备及投影图像的校正方法
JP2024514565A (ja) * 2021-04-09 2024-04-02 ユニバーサル シティ スタジオズ リミテッド ライアビリティ カンパニー アニメフィギュアの動的プロジェクションマッピングのためのシステム及び方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080004533A1 (en) 2006-06-30 2008-01-03 General Electric Company Optical imaging systems and methods
JP2011155412A (ja) * 2010-01-26 2011-08-11 Panasonic Electric Works Co Ltd 投影システムおよび投影システムにおける歪み修正方法
JP2016149618A (ja) * 2015-02-12 2016-08-18 株式会社フローベル 画像投影システム、プロジェクタ、およびプログラム
JP2017158042A (ja) * 2016-03-02 2017-09-07 パナソニック株式会社 投影装置

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
ATE523145T1 (de) * 2005-07-01 2011-09-15 Luminetx Corp Projektion von unteroberflächenstrukturen auf eine objektoberfläche
WO2008101129A1 (en) * 2007-02-14 2008-08-21 Luminetx Corporation System and method for projection of subsurface structure onto an object's surface
RU2655018C2 (ru) * 2013-03-19 2018-05-23 Конинклейке Филипс Н.В. Система гиперспектральной визуализации в видимом свете, способ записи гиперспектрального изображения и отображения гиперспектрального изображения в видимом свете
WO2015072047A1 (ja) * 2013-11-14 2015-05-21 パナソニックＩｐマネジメント株式会社 投影システム
JP2016224172A (ja) * 2015-05-28 2016-12-28 株式会社リコー 投影システム、画像処理装置、校正方法およびプログラム
JP2017147638A (ja) * 2016-02-18 2017-08-24 国立大学法人電気通信大学 映像投影システム、映像処理装置、映像処理プログラムおよび映像処理方法
JP6846661B2 (ja) * 2016-04-13 2021-03-24 パナソニックＩｐマネジメント株式会社 ３次元的な投影対象のための投影方法及び装置
US9930307B1 (en) * 2016-09-28 2018-03-27 Intel Corporation Projector image alignment
EP3701901B1 (en) * 2017-11-27 2022-03-02 Panasonic Corporation Projection device

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080004533A1 (en) 2006-06-30 2008-01-03 General Electric Company Optical imaging systems and methods
JP2011155412A (ja) * 2010-01-26 2011-08-11 Panasonic Electric Works Co Ltd 投影システムおよび投影システムにおける歪み修正方法
JP2016149618A (ja) * 2015-02-12 2016-08-18 株式会社フローベル 画像投影システム、プロジェクタ、およびプログラム
JP2017158042A (ja) * 2016-03-02 2017-09-07 パナソニック株式会社 投影装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3968628A4

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2022108089A (ja) * 2021-01-12 2022-07-25 ｉ－ＰＲＯ株式会社 投影装置
JP7568277B2 (ja) 2021-01-12 2024-10-16 ｉ－ＰＲＯ株式会社 投影装置
WO2022163835A1 (ja) * 2021-01-29 2022-08-04 パナソニックＩｐマネジメント株式会社 投写制御方法、および、投写制御装置
US12549695B2 (en) 2021-01-29 2026-02-10 Panasonic Projector & Display Corporation Projection control method and projection control device
JP2024514565A (ja) * 2021-04-09 2024-04-02 ユニバーサル シティ スタジオズ リミテッド ライアビリティ カンパニー アニメフィギュアの動的プロジェクションマッピングのためのシステム及び方法
WO2022253336A1 (zh) * 2021-06-03 2022-12-08 青岛海信激光显示股份有限公司 激光投影设备及投影图像的校正方法

Also Published As

Publication number Publication date
EP3968628B1 (en) 2023-11-22
JP7281632B2 (ja) 2023-05-26
JPWO2020261741A1 (https=) 2020-12-30
EP3968628A4 (en) 2022-10-12
US20220109814A1 (en) 2022-04-07
EP3968628A1 (en) 2022-03-16

Similar Documents

Publication Publication Date Title
JP7281632B2 (ja) 投影システム
US11273002B2 (en) Display system
JP6701669B2 (ja) 画像投射システム、プロジェクター、及び、画像投射システムの制御方法
US8297757B2 (en) Projector and projector control method
JPWO2018051679A1 (ja) 計測支援装置、内視鏡システム、内視鏡システムのプロセッサ、及び計測支援方法
CN112752079A (zh) 显示系统的控制方法和控制装置
JP2007060118A (ja) 投影装置及び投影制御方法
CN112752080A (zh) 显示系统的控制方法和控制装置
JP2010032842A (ja) プロジェクタおよびプロジェクタの制御方法
JP2003234983A (ja) プロジェクタ
JP6921987B2 (ja) 投影装置
JP2020134882A (ja) 投射制御装置、画像投射装置および投射制御方法
JP2010206671A (ja) プロジェクターおよびプロジェクターシステム
JP5383076B2 (ja) 眼科装置
JP2009049862A (ja) 投射画像の形状歪補正支援システム、投射画像の形状歪補正支援方法、及びプロジェクタ
JP6085382B1 (ja) 投影装置及び投影システム
JP6883256B2 (ja) 投影装置
JP6111356B1 (ja) 投影装置
JP2005099478A (ja) スペクトル解析機能をもつプロジェクタ
TWI577331B (zh) You can self-correct the method of marking visible light with visible light
JP6866915B2 (ja) 画像投射システム、及び画像投射システムの制御方法
JP3716839B2 (ja) プロジェクタの画像調整装置、および画像表示装置の画像調整方法
JP2015186080A (ja) プロジェクター、及び、プロジェクターの制御方法
JP2025007625A (ja) プロジェクターの設定方法、プロジェクター及びプログラム
JP2022077773A (ja) 投射制御装置、画像投射システム、投射制御装置の作動方法、およびプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20830848

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021527419

Country of ref document: JP

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 2020830848

Country of ref document: EP

Effective date: 20211208

NENP Non-entry into the national phase

Ref country code: DE