Classifications

• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/00002—Operational features of endoscopes
  • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
  • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
  • A61B1/000095—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope for image enhancement
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/00002—Operational features of endoscopes
  • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
  • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
  • A61B1/000096—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope using artificial intelligence
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/00002—Operational features of endoscopes
  • A61B1/00043—Operational features of endoscopes provided with output arrangements
  • A61B1/00045—Display arrangement
  • A61B1/0005—Display arrangement combining images e.g. side-by-side, superimposed or tiled
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/00163—Optical arrangements
  • A61B1/00188—Optical arrangements with focusing or zooming features
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/012—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor characterised by internal passages or accessories therefor
  • A61B1/018—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor characterised by internal passages or accessories therefor for receiving instruments
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
  • A61B1/05—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances characterised by the image sensor, e.g. camera, being in the distal end portion
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/06—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements
  • A61B1/0661—Endoscope light sources
  • A61B1/0676—Endoscope light sources at distal tip of an endoscope
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/06—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements
  • A61B1/0661—Endoscope light sources
  • A61B1/0684—Endoscope light sources using light emitting diodes [LED]
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
  • A61B1/00163—Optical arrangements
  • A61B1/00194—Optical arrangements adapted for three-dimensional imaging

Definitions

• the disclosure in this application relates to a surgical support system and a surgical support device.
• arthroscopic surgeries such as laparoscopic surgery and thoracoscopic surgery are rapidly becoming popular in place of conventional surgeries performed under direct vision such as laparotomy and thoracotomy.
• Arthroscopic surgery has various advantages in terms of cosmesis and minimal invasiveness.
• Patent Document 1 requires position markers to be provided on a plurality of trocars, a position sensor to detect the position markers, and images obtained based on the estimated position of the camera to be synthesized. Therefore, there is a problem that image composition becomes complicated.
• the purpose of the disclosure in this application is to provide a surgical support system and a surgical support device that can secure a better field of view during arthroscopic surgery.
• the disclosure in this application relates to a surgical support system and a surgical support device shown below.
• the imaging section is Multiple surgical aids are provided, Each imaging unit captures images of the inside of the body cavity, When the treatment instrument is inserted into the body cavity, an arbitrary imaging unit captures images of the inside of the body cavity including the treatment instrument,
• the image processing section is Combine images obtained from each imaging unit to generate a composite image, performing a blockage removal process that removes at least a portion of the image of the treatment instrument; Surgical support system.
• the treatment tools are a treatment section used for treatment within the body cavity; It has a non-treatment part that is continuous with the treatment part,
• the image processing unit at least a position indication image representing the position of the non-treated portion based on the shape of the non-treated portion; A genitalia image, which is an image of the shadow of the non-treated area, is combined to generate an occluded image.
• the image processing section is Generates a position indication image by machine learning the non-treatment part from the teacher image of the treatment instrument, The surgical support system according to (2) above.
• the position indication image is an image representing the outline of the non-treated area; The surgical support system according to (3) above.
• (6) Equipped with a depth estimation unit capable of estimating the depth to the point of interest in the body cavity;
• (7) comprising an image processing unit that processes images obtained from a plurality of imaging units of a surgical aid that assists in the use of a treatment instrument inserted into a body cavity; Combine images obtained from each imaging unit to generate a composite image, performing a blockage removal process that removes at least a portion of the image of the treatment instrument; Surgical support equipment.
• the surgical support system and surgical support device disclosed in this application can be suitably used for arthroscopic surgery.
• FIG. 1 is a configuration diagram of a surgical support system according to an embodiment.
• 2A is a schematic top view of the surgical aid 1
• FIG. 2B is a sectional view taken along the line XX' in FIG. 2A
• FIG. 2C is a diagram with the imaging section 2 removed from FIG. 2B.
• 2 is a block diagram schematically showing the hardware configuration of an image processing section 14.
• FIG. 3 is an explanatory diagram schematically showing the relationship between the surgical aid 1, the forceps 22, the organ 24, and images 32-1 to 32-3 taken by each imaging unit 2.
• FIG. 3 is an explanatory diagram schematically showing the relationship between photographed images 32-1 to 32-3 and a composite image 34.
• FIG. 3 is an explanatory diagram schematically showing an image of the forceps 22 after the shielding object removal process has been performed on the handle 22B. It is an explanatory view showing an image cut out from a simulation video of a surgery.
• FIG. 2 is an explanatory diagram illustrating the cause of out-of-focus occurrence in a composite image.
• (a) is an explanatory diagram showing an example of an image before calibration
• (b) is an explanatory diagram showing an example of an image after calibration.
• (a) is an explanatory diagram showing an example of a composite image before autofocus
• (b) is an explanatory diagram showing an example of a composite image after autofocus.
• the surgical support system 10 mainly includes a surgical aid 1, an image processing section 14, a display section 16, an operation section 18, and the like.
• the surgical aid 1 includes an imaging section 2 (FIGS. 2A to 2C) that assists in the use of a treatment instrument (here, forceps 22) inserted into the body cavity 20 and images the inside of the body cavity 20.
• the image processing unit 14 processes images obtained from the imaging unit 2.
• the display unit 16 displays image data subjected to image processing by the image processing unit 14.
• the operation unit 18 is used to input information necessary to make each device of the surgical support system 10 perform its functions.
• a plurality of imaging units 2 are provided in the surgical aid 1, and each imaging unit 2 captures an image of the inside of the body cavity, and when the treatment instrument is inserted into the body cavity, any imaging unit including the treatment instrument Images are taken to show the inside of the body cavity.
• the image processing unit 14 combines the images obtained from the respective imaging units 2 to generate a composite image, and performs an obstruction removal process to remove at least a portion of the image of the treatment instrument.
• Image processing for generating a composite image is image processing for generating an image with a wider field of view than the field of view obtained by one imaging unit 2 by combining moving images shot by a plurality of imaging units 2.
• the shielding object removal process is performed using parallax between the plurality of imaging units 2.
• the surgical aid 1 may be used as an insertion port for inserting a surgical treatment instrument into a body cavity 20.
• the surgical aid 1 is a trocar with a camera that also has a photographing function.
• the treatment instrument may be inserted into the opposing 20 through a hole other than the hole drilled for the surgical aid 1.
• the surgical auxiliary tool 1 photographs the treatment tool inserted from the other hole.
• FIG. 4 schematically shows the case in which the surgical aid 1 is inserted through a hole different from the hole for inserting the treatment tool.
• Treatment tools include various devices such as forceps 22 used in surgery. Generally, there are various types of forceps 22 depending on function, organ, or purpose, and any type may be used.
• the surgical aid 1 may be anything that assists in the use of a treatment instrument inserted into the body cavity 20; for example, the surgical aid 1 itself may be inserted into the body cavity. Further, the surgical aid 1 may not be inserted into a body cavity, but may be used by being placed in an incised part of the body.
• FIG. 2A shows a surgical aid 1 similar to that disclosed in one embodiment in the same application.
• 2A is a schematic top view of the surgical aid 1
• FIG. 2B is a sectional view taken along the line XX' in FIG. 2A
• FIG. 2C is a diagram from FIG. 2B with the imaging section 2 removed.
• the surgical aid 1 includes an imaging section 2, a holding section 3 that holds the imaging section 2, and a base material 4.
• the base material 4 includes the holding part 3.
• the base material 4 is formed into a tubular shape, and the imaging section 2 is attached to the base material 4 via the holding section 3 .
• the imaging section 2 and the holding section 3 are built into the base material 4, but the imaging section 2 (and/or the holding section 3) does not protrude from the outer periphery of the base material 4. It's okay.
• the imaging unit 2 is not particularly limited as long as it can image the inside of the body cavity. Examples include a CCD image sensor, a CMOS image sensor, Foveon X3, and an organic thin film image sensor. Note that there is no particular restriction on the imaging range of the imaging section 2. It is also possible to image the inside of the body cavity with a single imaging unit 2 using a wide-angle camera, but in that case, the edges of the image may become blurred. Further, there is a possibility that some parts (shaded parts) cannot be imaged due to surgical instruments, organs, etc. On the other hand, since the surgical aid 1 includes three or more imaging units 2, a field of view can be ensured even when using the above-mentioned sensors that are generally commercially available, and the number of shadow parts can be reduced.
• the number (N) of the imaging units 2 is an integer of 3 or more, for example, 4 or more, 5 or more, 6 or more.
• the upper limit of the number (N) of imaging units 2 may be determined while taking into consideration cost, convenience of image processing (processing speed), etc. For example, 20 or less, 15 or less, 10 or less, 8 or less, etc. Can be mentioned.
• the holding section 3 is provided on the base material 4 to hold the imaging section 2.
• the holding part 3 is formed to penetrate the base material 4, but if the holding part 3 can hold the imaging part 2 at a predetermined angle, the shape and location of the holding part 3 can be adjusted. There are no particular restrictions.
• Each imaging unit 2 is arranged so as to face the outside of the base material 4 at an angle of 0 degrees or more and 10 degrees or less.
• the surgical aid 1 is equipped with an endoscope and a light source.
• a chip LED is used as the light source, and is connected to an external power source.
• the surgical aid 1 can be provided with a sealing mechanism that prevents air leakage when inserting and removing the treatment instrument (forceps 22 in this embodiment), an air supply mechanism that sends air into the abdominal cavity, and the like.
• the imaging unit 2 has a zoom function and an autofocus function.
• the zoom function may be based on optical zoom or digital zoom. The autofocus function will be described later.
• the surgical aid 1 is connected to the image processing section 14.
• Image data acquired by the imaging section 2 is transmitted to the image processing section 14.
• the connection method between the imaging section 2 and the image processing section 14 may be wired or wireless.
• the imaging section 2 performs imaging including the treatment instrument, and the treatment instrument has a treatment section used for treatment within the body cavity 20 and a non-treatment section continuous with the treatment section.
• the treatment tool is the forceps 22
• the tip side pinching part 22A corresponds to the treatment part
• the handle 22B continuous to the pinching part 22A corresponds to the non-treatment part.
• the sandwich portion 22A is formed using a material such as a stainless steel alloy, and the handle 22B is coated with an electrically insulating material.
• the sandwich portion 22A may also be referred to as a “functional portion” or the like.
• the image processing unit 14 can be configured by installing a program (software) in a computer device to perform each function of the surgical support system 10.
• the image processing unit 14 constitutes a surgical support device in the surgical support system 10, but the entire surgical support system 10 can also be referred to as a surgical support device. In that case, the image processing unit 14 constitutes a part of the surgical support device.
• the computer device includes a control section 62, a storage section 64, and a communication section 66.
• the control unit 62 includes one or more processors and their peripheral circuits.
• the control unit 62 controls the overall operation of the image processing unit 14, and is, for example, a CPU (Central Processing Unit).
• the control unit 62 executes processing based on programs (computer programs such as driver programs, operating system programs, and application programs) stored in the storage unit 64. Further, the control unit 62 can execute multiple programs in parallel.
• the control unit 62 includes an obstruction removal unit 72, an autofocus unit 74, a depth estimation unit 76, and a 3D measurement unit 78.
• the shielding object removal unit 72 executes a shielding object removal function through a shielding object removal process (FIGS. 4 to 8), which will be described later.
• the autofocus unit 74 executes an autofocus function (FIGS. 9 to 11).
• the depth estimation section 76 executes a depth estimation function
• the 3D measurement section 78 executes a 3D measurement function.
• Each of these units included in the control unit 62 is a functional module implemented by a computer program executed on a processor included in the control unit 62.
• Each of these units included in the control unit 62 may be implemented in the image processing unit 14 as an independent integrated circuit, microprocessor, or firmware.
• the storage unit 64 is used to store information necessary for the control unit 62 to execute each function.
• the storage unit 64 includes, for example, at least one of a semiconductor memory, a magnetic disk device, and an optical disk device.
• the storage unit 64 stores driver programs, operating system programs, application programs (such as control programs for realizing the functions of the image processing unit 14), data, etc. used in processing by the control unit 62.
• the storage unit 64 stores, as a driver program, a communication device driver program for controlling the communication unit 66, which will be described later.
• the computer program may be installed in the storage unit 64 from a computer-readable portable recording medium such as a CD-ROM or DVD-ROM using a known setup program or the like. Further, the computer program may be a program downloaded from the cloud via a public communication line such as the Internet communication line.
• the communication unit 66 is configured to perform wired communication according to a communication method such as Ethernet (registered trademark) or wireless communication according to a communication method such as Wi-Fi (registered trademark) or Wi-Fi Aware (registered trademark). It has an interface circuit and performs wired or wireless communication with a communication section (not shown) included in the surgical aid 1, display section 16, operation section 18, or other external equipment (not shown). established and performs the transmission of information directly.
• a communication method such as Ethernet (registered trademark) or wireless communication according to a communication method such as Wi-Fi (registered trademark) or Wi-Fi Aware (registered trademark). It has an interface circuit and performs wired or wireless communication with a communication section (not shown) included in the surgical aid 1, display section 16, operation section 18, or other external equipment (not shown). established and performs the transmission of information directly.
• the interface circuit included in the communication unit 66 may perform short-range wireless communication according to a communication method such as Bluetooth (registered trademark) or communication using 920 MHz band specific low power wireless.
• the communication unit 66 is not limited to one for performing wireless communication, and may be one for transmitting various signals by, for example, infrared communication. Further, the communication unit 66 may be a communication interface for connecting to a USB (Universal Serial Bus) or the like, a wired or wireless LAN (Local Area Network) communication interface, or the like.
• the image processing unit 14 uses at least a position indication image representing the position of the non-treated area based on the shape of the non-treated area (here, the handle 22B) and a genital area image that is an image of the shadow of the non-treated area. to generate an occluded image.
• the shielding object removed image is an image that can represent the position and area of the non-treated area.
• the shielding object removed image is generated by replacing the image of the non-treated area (here, the image of the area shielded by the handle 22B) with an image obtained from another imaging unit 2.
• FIGS. 4 to 6 a shielding object removal process for removing the entire shielding object (also referred to as "erasing") will first be explained based on FIGS. 4 to 6. Thereafter, a description will be given of a shielding object removal process in which a shielding object is removed, leaving only a part of the shielding object, based on FIGS. 7 and 8.
• FIG. 4 schematically shows the process of generating a composite image from images acquired by the three imaging units 2 and the process of removing an obstruction from the composite image.
• the surgical aid 1 is shown in the center.
• the internal organs 24 in the body cavity 20 are simplified by the shapes of arrows, and the forceps 22 are simplified by the shapes of round bars.
• FIG. 4 does not show a situation in which the surgical aid 1 is used as an insertion port for inserting the forceps 22 into the body cavity 20; A situation in which it is inserted into a body cavity 20 through a hole is schematically shown.
• the left side of the forceps 22 corresponds to the distal end side (the side of the pinching part 22A), and the right side of the forceps 22 corresponds to the proximal end side (the side of the handle 22B).
• the forceps 22 and the handle 22B are the shields. Since the forceps 22 are located between the imaging units 2-1 to 2-3 and the organ 24, they serve as a shield that shields the organ 24 and the surgical site.
• FIG. 4 a branch number is added to each of the plurality of (here, three) imaging units 2, and the codes of the imaging units 2 are “2-1,” “2-2,” and “2-3.” It becomes. Furthermore, in FIG. 4, the images 32-1, 32-2, and 32-3 are attached to each image (also referred to as a "photographed image") taken by the imaging units 2-1 to 2-3. There is. In FIG. 4, circled numbers "1" to "3" are shown in the photographed images 32-1, 32-2, and 32-3. These numbers are shown in FIG. 4 to clearly distinguish between photographed images 32-1, 32-2, and 32-3, and the circled numbers are not superimposed on the actual images. do not have.
• FIG. 4 at least a portion of the organ 24 is shown in the captured images 32-1 to 32-3 of the respective imaging units 2-1 to 2-3.
• the photographed images 32-1 to 32-3 some of the photographed images 32-2 and 32-3 show the forceps 22, but other photographed images 32-1 do not show the forceps 22. Not shown.
• reference numerals 26 and 28 indicate surrounding objects of the organ 24, and these surrounding objects 26 and 28 are simplified by asterisks.
• the photographed image 32-1 shows one peripheral object 26, and the photographed image 32-3 shows the other peripheral object 28.
• the surrounding objects 26 and 28 can be referred to as "organs adjacent to" the organ 24 targeted for resection, or "organs around" the organ 24 targeted for resection.
• the photographed images 32-1 to 32-3 are combined by the image processing unit 14 as schematically shown in FIG.
• the photographed images 32-1 to 32-3 before combination are shown side by side, and in the lower part of FIG. 5, the photographed images 32-1 to 32-3 after combination are shown.
• the photographed image after combination will be referred to as a "composite image”, and the composite image will be designated by the reference numeral 34.
• the composite image 34 is generated by combining the photographed images 32-1 to 32-3 in an arrangement and inclination based on the positional relationship of the imaging units 2-1 to 2-3.
• the organ 24 and surrounding objects 26 and 28 are displayed so as to match their actual positional relationship.
• Each of the photographed images 32-1 to 32-3 is a moving image, and the composite image 34 is also displayed as a moving image.
• the image processing unit 14 is also capable of recording the composite image 34 or cutting out the composite image 34 to obtain image data of a still image.
• the forceps 22 are shown in the captured images 32-2 and 32-3.
• the organ 24 and surrounding objects 26 and 28 are shown in the composite image 34 in the lower part of FIG. 5, but the forceps 22 is not shown.
• the images of the forceps 22 that appeared in the captured images 32-2 and 32-3 are removed.
• the image of the forceps 22 is automatically removed when the composite image 34 is generated.
• the timing of image removal is arbitrary; for example, the image may be removed by an operator inputting an instruction to the image processing section 14 via the operation section 18.
• the image processing unit 14 may be provided with a voice recognition function, so that when an instruction is detected by voice, the image is removed.
• FIGS. 6(a) to 6(d) schematically show the procedure for image removal.
• the forceps 22 are shown in a simplified manner using a round bar.
• FIG. 6(b) the image of the forceps 22 has been removed.
• FIG. 6B the area 36 where the image of the forceps 22 was displayed in FIG. This is an object-removed image 38.
• a private part image 40 is superimposed on this shielding object removed image 38, as shown in FIG. 6(c).
• the private part image 40 is generated to compensate for the obstruction removal area 36. The generation of the shield removal area 36 will be described later.
• a contour portion 42 appears in a linear shape around the genital area image 40.
• the outline portion 42 becomes a position indicating image of the forceps 22.
• the position indication image is an image that indicates the area of the treatment instrument (here, the forceps 22) as the position of the treatment instrument to the operator or the like who views the display unit 16.
• the outline portion 42 appears because a slight difference in size or positional shift between the obstruction removed image 38 and the private part image 40 is expressed as a difference.
• the image processing unit 14 performs machine learning on the treatment instrument (also referred to as "teacher data”) from a teacher image (also referred to as "teacher data”) of the treatment instrument (here, the forceps 22) to generate a position indicating image (image of the contour portion 42).
• a large number (for example, several thousand) of image data regarding the forceps 22 is accumulated in advance.
• storage of image data for machine learning is performed using the storage unit 64 (FIG. 3) of the image processing unit 14; (including a storage unit).
• the training data is obtained by photographing the forceps 22 in various assumed directions and angles. Further, for each image of the teacher data, an image is detected in which the pinching portion 22A and the handle 22B are distinguished. In the image processing unit 14, the areas of the forceps 22, the pinching portion 22A, and the handle 22B are distinguished using the image detection results.
• the clamping portion 22A and handle 22B of the forceps 22 are distinguished based on the shape and color determination results.
• the color of the sandwich portion 22A is generally silver (also referred to as "silver color”), which is the color of the stainless steel alloy material.
• the handle 22B is generally covered with a cover (not shown), and the cover is made of electrically insulating synthetic resin or the like.
• the color of the cover is generally a color such as black or brown that can be distinguished from stainless steel.
• the image of the forceps 22 can be removed not only from the entire forceps 22 but also from a part of the forceps 22 ( Here, it becomes possible to perform the process for the pattern 22B).
• the image processing unit 14 when removing the entire image of the forceps 22, the image processing unit 14 generates an image of the entire forceps 22 in the composite image 34 and an image of the entire forceps 22 in the teacher data. Compare sequentially. The image processing unit 14 recognizes the entire area of the forceps 22. Furthermore, the image processing unit 14 removes the image of the forceps 22 from the composite image 34.
• the removed area is the part that will be in the shadow of the forceps 22.
• the private part images 40 are synthesized (FIGS. 6(c) and (d)).
• the data of the genital area image 40 (vulva image data) is created by selectively using the captured images 32-1 to 32-3 of the imaging units 2-1 to 2-3.
• As the genital region image 40 an image of a portion corresponding to an area inside the outline of the forceps 22 is used from a photographed image in which the forceps 22 are not shown (the photographed image 32-1 in the examples of FIGS. 4 and 5).
• the genital area image 40 is combined with the removed portion of the composite image 30, and a composite image 30 with the forceps 22 removed is generated (FIG. 5).
• FIG. 5 illustration of the private part image 40 and the contour part 42 is omitted.
• the outline portion 42 shown in FIG. 6(d) is not, for example, a line image superimposed on the outline part of the genital area image 40. Moreover, the forceps 22 are not stationary in most cases, but are moving. Therefore, the outline portion 42 is not always displayed stably.
• FIGS. 7 and 8 show a state in which only the image of the handle 22B of the forceps 22 has been removed.
• the shielding object removal process is performed only on the handle 22B, and the outline portion 42 (position indication image) is displayed only on the handle 22B.
• the shielding object to be removed only changes from the entire forceps 22 to the handle 22B, and the shielding object removal process is performed in almost the same way as for the entire forceps 22. is possible.
• FIG. 8 shows images obtained by cutting out a simulation video of a surgery performed by the surgery support system 10 of this embodiment at a certain timing.
• a model of the organ 24 is used in the simulation, and the model of the organ 24 is placed within the model of the rib 46.
• What is shown in FIG. 8 is a composite image 34, in which the forceps 22 are shown in front of the organ 24.
• the pinching portion 22A remains and the handle 22B is removed.
• the outline portion 42 of the handle 22B surrounds the genital area image 40.
• a video is generated by using a composite image as in the example of FIG. 8 as one frame, and the image processing unit 14 sequentially outputs composite images to the display unit 16, for example, every few milliseconds to tens of milliseconds. .
• the operator can perform surgery on the organ 24 and the like while viewing the moving image.
• the entire or part of the pattern 22B may be displayed, for example, for more than 1 second. There is no situation in which the image is displayed continuously for a period of time, but only for a moment, so there is no problem with the treatment in this respect either.
• the process of removing the entire forceps 22 (FIGS. 4 to 6) and the process of removing only the handle 22B (FIGS. 7 and 8) can be arbitrarily selected and executed. Further, it is possible to enable execution of only one of these processes. For example, it is possible to eliminate the function of removing the entire forceps 22 and provide only the function of removing only the handle 22B.
• Surgery using the surgical aid 1 is often performed in a narrow or dark area, so it is difficult to generate an image with an obstruction such as the entire forceps 22 (or handle 22B) removed. It is effective in Furthermore, if the position of the forceps 22 cannot be determined at all, the forceps 22 may come into contact with an unexpected position. Therefore, it is also effective in surgery to be able to determine the position of the forceps 22 to some extent.
• the clamping section 22A the state of the treatment section (here, the clamping section 22A) that performs the treatment, it is possible to more accurately manipulate the treatment instrument (here, the forceps 22). Therefore, it is also effective in surgery to leave the treatment section (in this case, the pinching section 22A).
• a function may be provided that allows the contour portion 42 to be removed depending on the situation. By doing so, it becomes possible to remove the shielding object in a manner that better meets the needs of the operator.
• the surgical aid 1 is normally directed toward the surgical target area. For this reason, during treatment using the pinching section 22A, it is considered that an image of the organ 24 is obtained by at least one of the imaging sections 2-1 to 2-3, and problems in the treatment are unlikely to occur.
• the surgical support system 10 of this embodiment is equipped with an autofocus function.
• the autofocus function is executed by an autofocus section 74 (FIG. 3) that can individually adjust the focal lengths of the imaging sections 2-1 to 2-3.
• the autofocus function attempts to eliminate focus-related deviations.
• the surgical aid 1 and the observation target (also referred to as the "subject"), such as the organ 24, etc. Since the distance changes during surgery, the image may become out of focus (so-called out-of-focus may occur).
• the focus position during calibration is the center 44B of three locations: front 44A, center 44B, and back 44C shown on the left side of FIG.
• the position of the organ 24 etc. moves to the front 44A or the back 44C due to a change in the position of the surgical aid 1
• a shift in focus occurs as shown in the upper or lower row on the right side of FIG.
• the composite image 34 is blurred.
• the autofocus function may use optical zoom or digital zoom.
• the autofocus function it is possible to use one that fixes the positions of the imaging units 2-1 to 2-3 and adjusts the focus by changing the relative distances of the imaging units 2-1 to 2-3 through image processing. It is.
• the relative distances related to the imaging units 2-1 to 2-3 can be changed by changing the three-dimensional positions of the imaging units 2-1 to 2-3 recognized by the image processing unit 14.
• the three-dimensional positions of the imaging units 2-1 to 2-3 are relative positions with one of the imaging units 2-1 to 2-3 as a base point.
• the relative position between the imaging units 2-1 to 2-3 is such that, for example, a plurality of calibration charts (for example, 30 images) as shown in the composite image 34 of FIGS. 10(a) and 10(b) are degree) has been photographed and estimated in advance.
• a plurality of calibration charts for example, 30 images
• the corners of a rectangle in the chart are detected, and each imaging unit 2-1 to 2-3 estimates a transformation (projective transformation) matrix that matches the positions of the corresponding corner points (corner points). It will be done as follows.
• the calibration chart is held by a holder (not shown), and the position and posture (direction) of the surgical aid 1 to which each of the imaging units 2-1 to 2-3 is attached is changed.
• the positions of the imaging units 2-1 to 2-3 may be fixed, and the positions and postures (directions) of the holders in the calibration chart may be changed.
• the change (image blur) when the distance between the imaging units 2-1 to 2-3 and the subject becomes closer or farther is the same as when the relative distance between the imaging units 2-1 to 2-3 is increased or decreased. Change and appearance become the same. Therefore, by moving the image in parallel, the edge strength in the composite image is monitored while changing the relative distance between each of the imaging units 2-1 to 2-3. The position where the edge strength in this composite image is maximum is determined, and the blur in the image is eliminated.
• FIG. 11(a) shows an example of a composite image before autofocus
• FIG. 11(b) shows an example of a composite image after autofocus.
• What is shown in FIGS. 11(a) and 11(b) are conductive cables placed randomly on the desk, and are not images taken during surgery. However, it can be seen that the blurred composite image in FIG. 11(a) becomes clear as shown in FIG. 11(b) by the autofocus function.
• the base material 4 of the surgical aid 1 is made of a flexible material, the mutual positional relationship of the imaging units 2-1 to 2-3 is likely to change during the surgery. In this case, the accuracy of the composite image 34 is likely to decrease, so the autofocus function is even more effective.
• the autofocus function may change the positional relationship of the imaging units 2-1 to 2-3 independently.
• the surgical support system 10 of this embodiment can be equipped with a depth estimation function.
• the depth estimation function is executed by the depth estimation unit 76 (FIG. 3) that can estimate the depth to the point of interest in the body cavity 20. Can estimate distance to objects. By operating the forceps 22 and the like while checking the estimated distance, it is possible to more reliably prevent a situation in which the forceps 22 and the like unexpectedly interfere with the organ 24 and the like.
• the depth estimation method can be employed as the depth estimation method.
• a method of depth estimation for example, there is a method using a geometric solution regarding cameras (here, the imaging units 2-1 to 2-3).
• the depth to the point of interest can be estimated on condition that the relative positions of the plurality of cameras and the position of the point of interest (part to be observed) in the subject in each camera image are known.
• Cycle-GAN is one of the methods of performing style conversion using GAN (Generative Adversarial Network) using AI (Artificial Intelligence).
• GAN Geneative Adversarial Network
• AI Artificial Intelligence
• a generator generates an image similar to training data, and a discriminator determines whether the image is training data or an image generated by the generator. Learning is performed by repeating this process.
• Style conversion is a method of converting the external characteristics of data.
• a method of clicking on a point of interest (part to be observed) on the panoramic vision and autofocusing on that point of interest can also be adopted as a depth estimation method.
• the surgical support system 10 By combining a depth estimation function that employs depth estimation methods such as these with a function of obstructing object removal processing, it is possible to further improve the safety of surgery. Further, for example, if a blood vessel with a predetermined thickness or more approaches within a predetermined distance as a result of depth estimation, the surgical support system 10 issues an alarm (alert) on the display unit 16, speaker (not shown), etc. It is also possible to call the surgeon's attention.
• the surgical support system 10 of this embodiment can be equipped with a 3D (three-dimensional) measurement function.
• the 3D measurement function aims to convert the visible area into 3D, and uses depth information at each point in the visible area.
• depth information depth information obtained by depth estimation can be used.
• 3D measurement enables three-dimensional surface measurement of the organ 24 and the like. For example, it is possible to determine the Euclidean distance between two specified points and measure the shortest distance along the organ surface. Furthermore, by combining such distance measurements, it is also possible to measure the surface area of the organ 24 and the like. 3D measurement functions such as these are particularly useful in surgical resections and the like.
• the surgical aid and surgical support system disclosed in this application can secure a wide field of view during arthroscopic surgery. Therefore, it is useful for the medical device manufacturing industry.

Landscapes

• Health & Medical Sciences (AREA)
• Life Sciences & Earth Sciences (AREA)
• Surgery (AREA)
• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Medical Informatics (AREA)
• Optics & Photonics (AREA)
• Biomedical Technology (AREA)
• Public Health (AREA)
• General Health & Medical Sciences (AREA)
• Heart & Thoracic Surgery (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Molecular Biology (AREA)
• Animal Behavior & Ethology (AREA)
• Veterinary Medicine (AREA)
• Pathology (AREA)
• Biophysics (AREA)
• Radiology & Medical Imaging (AREA)
• Signal Processing (AREA)
• Theoretical Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Artificial Intelligence (AREA)
• Evolutionary Computation (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Microelectronics & Electronic Packaging (AREA)
• Business, Economics & Management (AREA)
• General Business, Economics & Management (AREA)
• Epidemiology (AREA)
• Primary Health Care (AREA)
• Ophthalmology & Optometry (AREA)
• Image Processing (AREA)
• Endoscopes (AREA)

Priority Applications (2)

Applications Claiming Priority (2)

Publications (1)

Family

ID=88419686

Family Applications (1)

Country Status (3)

Cited By (1)

Citations (1)

• 2023
  • 2023-03-08 JP JP2024516127A patent/JPWO2023203908A1/ja active Pending
  • 2023-03-08 WO PCT/JP2023/008852 patent/WO2023203908A1/ja not_active Ceased
  • 2023-03-08 US US18/857,810 patent/US20250268454A1/en active Pending

Patent Citations (1)

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events