WO2022254836A1 - 情報処理装置、情報処理システム及び情報処理方法 - Google Patents

情報処理装置、情報処理システム及び情報処理方法 Download PDF

Info

Publication number
WO2022254836A1
WO2022254836A1 PCT/JP2022/008495 JP2022008495W WO2022254836A1 WO 2022254836 A1 WO2022254836 A1 WO 2022254836A1 JP 2022008495 W JP2022008495 W JP 2022008495W WO 2022254836 A1 WO2022254836 A1 WO 2022254836A1
Authority
WO
WIPO (PCT)
Prior art keywords
event
information processing
light
orientation
light emission
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/008495
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
悟士 尾崎
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Priority to US18/561,270 priority Critical patent/US20240246241A1/en
Priority to JP2023525399A priority patent/JPWO2022254836A1/ja
Publication of WO2022254836A1 publication Critical patent/WO2022254836A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1656—Program controls characterised by programming, planning systems for manipulators
    • B25J9/1664—Program controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/30—Surgical robots
    • A61B34/32—Surgical robots operating autonomously
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1694—Program controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • B25J9/1697—Vision controlled systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10141—Special mode during image acquisition
    • G06T2207/10152—Varying illumination
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing

Definitions

  • the present disclosure relates to an information processing device, an information processing system, and an information processing method.
  • surgical instruments such as endoscopes and electric scalpels (hereinafter referred to as surgical instruments) using robot arms.
  • an image sensor that acquires image data at a predetermined frame rate of, for example, about 60 fps (frames per second) is used. , which has made it difficult to autonomously control the robot arm at higher speeds.
  • the present disclosure proposes an information processing device, an information processing system, and an information processing method capable of controlling a robot arm at higher speed.
  • an information processing apparatus provides an event-based EVS (event-based A position/orientation recognition unit that recognizes the position and orientation of the surgical instrument based on event data input from a Vision Sensor, and supports the medical device equipped with the EVS based on the recognized position and orientation of the surgical instrument.
  • a control instruction unit that generates control information for controlling the robot arm.
  • FIG. 1 is a schematic diagram showing a schematic configuration example of an endoscope system according to a first embodiment of the present disclosure
  • FIG. 2 is a block diagram showing the functional configuration of the endoscope system shown in FIG. 1
  • FIG. 1 is a schematic diagram showing a configuration example of an endoscope according to a first embodiment of the present disclosure
  • FIG. 1 is a schematic diagram showing a schematic configuration example of a surgical instrument according to a first embodiment of the present disclosure
  • FIG. FIG. 5 is a schematic diagram showing a schematic configuration example of a surgical instrument according to a modification of the first embodiment of the present disclosure
  • FIG. 5 is a schematic diagram showing a schematic configuration example of a surgical instrument according to another modified example of the first embodiment of the present disclosure
  • 1 is a block diagram showing a schematic configuration example of an image sensor according to a first embodiment of the present disclosure
  • FIG. 1 is a block diagram showing a schematic configuration example of an EVS according to a first embodiment of the present disclosure
  • FIG. FIG. 4 is a waveform diagram showing an example of a light emission pattern for each surgical tool according to the first embodiment of the present disclosure
  • FIG. 4 is a diagram for explaining a search pixel range according to the first embodiment of the present disclosure
  • FIG. FIG. 4 is a diagram for explaining a pixel region in which an event is detected according to the first embodiment of the present disclosure
  • FIG. 4 is a flow chart showing an operation example of the light emitting device according to the first embodiment of the present disclosure
  • 4 is a flowchart showing an operation example of the information processing device according to the first embodiment of the present disclosure
  • 4 is a flow chart showing an operation example of the endoscope control device 40 according to the first embodiment of the present disclosure
  • FIG. 11 is a schematic diagram showing a schematic configuration example of an endoscope system according to a second embodiment of the present disclosure
  • 16 is a block diagram showing the functional configuration of the endoscope system shown in FIG. 15
  • FIG. FIG. 11 is a schematic diagram showing a schematic configuration example of an endoscope system according to a third embodiment of the present disclosure
  • 18 is a block diagram showing the functional configuration of the endoscope system shown in FIG. 17;
  • FIG. 11 is a waveform diagram showing an example of light emission patterns for each surgical instrument according to a modified example of the third embodiment of the present disclosure
  • 1 is a hardware configuration diagram showing an example of a computer that implements functions of an information processing apparatus according to the present disclosure
  • RGB sensors image sensors
  • image sensors generally used in autonomous control of robot arms are configured to acquire image data at a predetermined frame rate, for example, about 60 fps (frames per second). Therefore, there is a limit to the recognition speed for autonomously controlling the robot arm, which makes it difficult to autonomously control the robot arm at a higher speed.
  • high-speed cameras capable of capturing images at a high frame rate have been developed, but because the high-speed cameras themselves are large, they are difficult to mount inside an endoscope that is supported by a robot arm. Met.
  • the following embodiments propose an information processing device, an information processing system, and an information processing method that make it possible to improve the control speed of the robot arm. Further, in some or all of the following embodiments, an information processing device, an information processing system, and an information processing method capable of suppressing deterioration in control accuracy of a robot arm are proposed.
  • an information processing apparatus an information processing system, and an information processing method according to a first embodiment will be described in detail with reference to the drawings.
  • an operation system hereinafter referred to as an endoscope system
  • an endoscope system in which an endoscope is autonomously controlled by a robot arm
  • the technology according to the present disclosure is not limited to endoscope systems. It is possible to apply to various medical systems such as systems.
  • FIG. 1 is a schematic diagram showing a schematic configuration example of an endoscope system according to the first embodiment.
  • FIG. 2 is a block diagram showing the functional configuration of the endoscope system shown in FIG. 1;
  • the endoscope system 1 includes an imaging device 10, an information processing device 20, a display device 30, an endoscope control device 40, a light emitting device 50, and an endoscope 60. , a robot arm 70 and a surgical tool 80 .
  • the imaging device 10 includes, for example, an image sensor 11 and an EVS (Event-based Vision Sensor) 12, and is arranged within a camera head 61 (see FIG. 3) of an endoscope 60. As shown in FIG.
  • the image sensor 11 acquires color image data (hereinafter referred to as RGB image data) at a predetermined frame rate (eg, 60 fps), for example.
  • RGB image data color image data
  • the acquired RGB image data is input to the display device 30 .
  • the EVS 12 detects an event for each pixel based on, for example, a change in luminance of incident light, and generates event data indicating the content of the detected event.
  • the generated event data is sequentially output to the information processing device 20 as an event stream.
  • the information processing device 20 generates image data (hereinafter referred to as event image data) based on events detected for each pixel.
  • the frame rate at which the information processing device 20 generates event image data may be, for example, the same as or higher than the frame rate of the image sensor 11 (eg, 1000 fps).
  • the information processing device 20 is configured by, for example, a personal computer or the like, and includes a light source recognition section 21 , a position/orientation recognition section 22 , and a control instruction section 23 .
  • the light source recognition unit 21 identifies the light emission pattern of the light emitting unit 51 included in the event image data, for example, by performing recognition processing on the event image data based on the event data input from the EVS 12 of the imaging device 10. .
  • the position and orientation recognition unit 22 recognizes the position and orientation of the surgical instrument 80 to which the light emission unit 51 is attached from the event image data for each light emission pattern of the light emission unit 51 specified by the light source recognition unit 21 .
  • Recognition of the position and orientation of the surgical tool 80 is not limited to recognition processing such as pattern matching, and machine learning such as DNN (Deep Neural Network), CNN (Convolutional Neural Network), and RNN (Recurrent Neural Network) is used. may be In that case, the event image data may be input to the learned learning model, and the position and orientation of the surgical tool 80 may be output as the estimation result.
  • the control instruction unit 23 calculates the position and orientation of the endoscope 60 for capturing the image to be displayed on the display device 30 based on the position and orientation of the surgical instrument 80 specified by the position and orientation recognition unit 22 . Then, the control instruction unit 23 generates control information for setting the position and orientation of the endoscope 60 to the calculated position and orientation, and outputs the generated control information to the endoscope control device 40 .
  • the display device 30 is configured by, for example, a display, and displays images acquired by the image sensor 11 to surgical staff such as a surgeon, a scopist, and a nurse.
  • the endoscope control device 40 includes an information processing device such as an MPU (Micro Processor Unit), for example, and includes a control section 41 and a drive section 42 .
  • MPU Micro Processor Unit
  • the control unit 41 generates a drive signal for driving the robot arm 70 based on the control information input from the information processing device 20 and inputs the generated drive signal to the drive unit 42 .
  • the drive unit 42 controls the position and orientation of the endoscope 60 by driving the robot arm 70 according to the drive signal input from the control unit 41 .
  • the endoscope control device 40 controls the optical system in the endoscope 60 based on the control information input from the information processing device 20, thereby increasing the magnification (zoom-in) of the image data acquired by the imaging device 10. / zoom out) may be controlled.
  • the light emitting device 50 includes a light emitting section 51 and a light emission control section 52 .
  • the light emitting part 51 is composed of, for example, a light emitting element such as an LED (Light Emitting Diode), and is provided in a surgical tool 80 such as an electric surgical knife.
  • a light emitting element such as an LED (Light Emitting Diode)
  • the light emitting unit 51 outputs light having a wavelength outside the visible light range (for example, infrared light or near-infrared light). good.
  • the light emitting section 51 may output visible light.
  • the light emission control unit 52 blinks the light emission unit 51 according to a preset light emission cycle.
  • the light emission control section 52 may be configured separately from the surgical instrument 80 and may be connected to the light emitting section 51 provided on the surgical instrument 80 by wire or wirelessly. may be
  • FIG. 3 is a schematic diagram showing a configuration example of an endoscope according to this embodiment.
  • the endoscope 60 includes, for example, a camera head 61, a junction section 63, a scope 64, and a joint section 62.
  • the endoscope 60 includes, for example, a camera head 61, a junction section 63, a scope 64, and a joint section 62.
  • the endoscope 60 includes, for example, a camera head 61, a junction section 63, a scope 64, and a joint section 62.
  • the scope 64 has a structure in which an optical fiber is passed through a cylindrical cylinder made of metal such as stainless steel, and a lens is provided at the tip. A rear end of the scope 64 is fixed to the joint portion 62 via a junction portion 63 .
  • An optical fiber cable 65 for guiding the irradiation light output from the light emitting section 51 is inserted from the side into the junction section 63 .
  • the optical fiber cable 65 passes through the scope 64 from the junction 63 and is guided to the tip of the scope 64 . Therefore, the irradiation light output from the light emitting unit 51 is output from the tip of the scope 64 via the optical fiber cable 65 .
  • the joint part 62 is configured to be detachable from the camera head 61, for example.
  • the light incident on the tip of the scope 64 propagates through the optical fiber inside the scope 64 (however, the optical fiber is different from the optical fiber cable 65 ) and is guided to the inside of the camera head 61 .
  • a spectroscope 13 is provided inside the camera head 61, in addition to the image sensor 11 and the EVS 12 that constitute the imaging device 10.
  • the spectroscope 13 uses an optical element such as a prism or a dichroic mirror that separates incident light according to its wavelength. you can On the other hand, when the light emitting unit 51 outputs light of wavelengths other than the visible light range, a half mirror or the like may be used for the spectroscope 13 .
  • the image sensor 11 and the EVS 12 are arranged, for example, so that planes including their light receiving surfaces are substantially perpendicular.
  • the light passing through the spectroscope 13 is incident on the image sensor 11 and the light reflected by the spectroscope 13 is incident on the EVS 12 . That is, in this embodiment, the image sensor 11 and the EVS 12 share the same optical axis.
  • the coordinate system of the RGB image data acquired by the image sensor 11 and the coordinate system of the event image data acquired by the EVS 12 can be aligned. good.
  • the image data acquired by the image sensor 11 and the EVS 12 are output to the outside via the transmission cable 66 provided in the camera head 61.
  • the RGB image data acquired by the image sensor 11 may be directly input to the display device 30 and displayed, or may be input to the information processing device 20 and subjected to predetermined processing before being displayed. It may be input to and displayed on device 30 .
  • FIG. 4 is a schematic diagram showing a schematic configuration example of the surgical instrument according to the present embodiment.
  • a scalpel is exemplified as the surgical tool.
  • the surgical instrument 80 includes, for example, a shaft portion 81 that is held by a surgeon's hand, and a treatment portion 82 that is provided at the tip of the shaft portion 81 and performs treatment such as cutting an affected area.
  • the light-emitting portion 51 is provided at a specific position on the shaft portion 81 of the surgical instrument 80 as described above.
  • the surgical instrument 80 is a scalpel, it may be arranged at a position (in FIG. 4, the connecting portion between the treatment section 82 and the shaft section 81) where the position and orientation of the treatment section 82 to be opened and closed can be identified.
  • the number of light emitting units 51 provided in each surgical instrument 80 is not limited to one, and may be two or more, for example, as illustrated in FIG.
  • the arrangement of the light emitting units 51 may be determined so that the type, position and orientation of the surgical instrument 80 can be identified from the arrangement of the light emitting units 51 detected as an event in the EVS 12 . For example, as shown in FIG.
  • the present invention is not limited to this.
  • the shape of the light-emitting portion 51 such that the direction can be specified, it is possible to determine which side is the distal end side where the treatment portion 82 is arranged. It may be configured to be identifiable from the event image data.
  • FIG. 7 is a block diagram showing a schematic configuration example of the image sensor according to the first embodiment.
  • a CMOS (Complementary Metal-Oxide-Semiconductor) type image sensor is exemplified, but it is not limited to this, and a CCD (Charge-Coupled Device) type or the like can acquire color or monochrome image data.
  • CMOS Complementary Metal-Oxide-Semiconductor
  • CCD Charge-Coupled Device
  • the CMOS image sensor may be an image sensor manufactured by applying or partially using a CMOS process.
  • the image sensor 11 has, for example, a stack structure in which a semiconductor chip in which a pixel array section 111 is formed and a semiconductor chip in which a peripheral circuit is formed are stacked.
  • Peripheral circuits may include, for example, a vertical drive circuit 112, a column processing circuit 113, a horizontal drive circuit 114, and a system controller 115.
  • the image sensor 11 further includes a signal processing section 118 and a data storage section 119 .
  • the signal processing unit 118 and the data storage unit 119 may be provided on the same semiconductor chip as the peripheral circuit, or may be provided on a separate semiconductor chip.
  • the pixel array section 111 has a configuration in which the pixels 110 each having a photoelectric conversion element that generates and accumulates an electric charge according to the amount of received light are arranged in a two-dimensional lattice in rows and columns, that is, in rows and columns.
  • the row direction refers to the arrangement direction of pixels in a pixel row (horizontal direction in the drawing)
  • the column direction refers to the arrangement direction of pixels in a pixel column (vertical direction in the drawing).
  • pixel drive lines LD are wired along the row direction for each pixel row and vertical signal lines VSL are wired along the column direction for each pixel column with respect to the matrix-like pixel array.
  • the pixel drive line LD transmits a drive signal for driving when reading a signal from a pixel.
  • the pixel drive lines LD are shown as wirings one by one, but the number is not limited to one each.
  • One end of the pixel drive line LD is connected to an output terminal corresponding to each row of the vertical drive circuit 112 .
  • the vertical drive circuit 112 is composed of a shift register, an address decoder, etc., and drives each pixel of the pixel array section 111 simultaneously or in units of rows. That is, the vertical drive circuit 112 constitutes a drive section that controls the operation of each pixel in the pixel array section 111 together with a system control section 115 that controls the vertical drive circuit 112 .
  • the vertical drive circuit 112 generally has two scanning systems, a readout scanning system and a discharge scanning system, although the specific configuration thereof is not shown.
  • the readout scanning system sequentially selectively scans the pixels 110 of the pixel array section 111 in units of rows in order to read out signals from the pixels 110 .
  • a signal read out from the pixel 110 is an analog signal.
  • the sweep-scanning system performs sweep-scanning ahead of the read-out scanning by the exposure time for the read-out rows to be read-scanned by the read-out scanning system.
  • a so-called electronic shutter operation is performed by sweeping out (resetting) the unnecessary charges in this sweeping scanning system.
  • the electronic shutter operation means an operation of discarding the charge of the photoelectric conversion element and newly starting exposure (starting charge accumulation).
  • the signal read out by the readout operation by the readout scanning system corresponds to the amount of light received after the immediately preceding readout operation or the electronic shutter operation.
  • a period from the readout timing of the previous readout operation or the sweep timing of the electronic shutter operation to the readout timing of the current readout operation is a charge accumulation period (also referred to as an exposure period) in the pixel 110 .
  • a signal output from each pixel 110 in a pixel row selectively scanned by the vertical driving circuit 112 is input to the column processing circuit 113 through each vertical signal line VSL for each pixel column.
  • the column processing circuit 113 performs predetermined signal processing on a signal output from each pixel of the selected row through the vertical signal line VSL for each pixel column of the pixel array section 111, and temporarily stores the pixel signal after the signal processing. to be retained.
  • the column processing circuit 113 performs at least noise removal processing, such as CDS (Correlated Double Sampling) processing and DDS (Double Data Sampling) processing, as signal processing.
  • CDS Correlated Double Sampling
  • DDS Double Data Sampling
  • the CDS processing removes pixel-specific fixed pattern noise such as reset noise and variations in threshold values of amplification transistors in pixels.
  • the column processing circuit 113 also has an AD (analog-digital) conversion function, for example, and converts analog pixel signals read from the photoelectric conversion elements into digital signals and outputs the digital signals.
  • AD analog-digital
  • the horizontal driving circuit 114 is composed of shift registers, address decoders, etc., and sequentially selects readout circuits (hereinafter referred to as pixel circuits) corresponding to the pixel columns of the column processing circuit 113 .
  • pixel circuits readout circuits
  • the system control unit 115 is composed of a timing generator that generates various timing signals. and other drive control.
  • the signal processing unit 118 has at least an arithmetic processing function, and performs various signal processing such as arithmetic processing on pixel signals output from the column processing circuit 113 .
  • the data storage unit 119 temporarily stores data necessary for signal processing in the signal processing unit 118 .
  • RGB image data output from the signal processing unit 118 may be directly input to the display device 30 and displayed as described above, or may be input to the information processing device 20 and subjected to predetermined processing. After being applied, it may be input to the display device 30 and displayed.
  • FIG. 8 is a block diagram showing a schematic configuration example of the EVS according to this embodiment.
  • the EVS 12 includes a pixel array section 121 , an X arbiter 122 and a Y arbiter 123 , an event signal processing circuit 124 , a system control circuit 125 and an output interface (I/F) 126 .
  • I/F output interface
  • the pixel array section 121 has a configuration in which a plurality of event pixels 120 each detecting an event based on a change in brightness of incident light are arranged in a two-dimensional lattice.
  • the row direction also referred to as row direction
  • the column direction also referred to as column direction
  • the arrangement of pixels in pixel columns It refers to the direction (vertical direction in the drawing).
  • Each event pixel 120 includes a photoelectric conversion element that generates a charge according to the luminance of incident light, and requests reading from itself when a change in luminance of incident light is detected based on the photocurrent that flows from the photoelectric conversion element. request to the X arbiter 122 and the Y arbiter 123, and according to the arbitration by the X arbiter 122 and the Y arbiter 123, an event signal indicating that an event has been detected is output.
  • Each event pixel 120 detects the presence or absence of an event depending on whether or not the photocurrent corresponding to the luminance of incident light has changed by exceeding a predetermined threshold. For example, each event pixel 120 detects as an event that the change in brightness exceeds a predetermined threshold (positive event) or falls below it (negative event).
  • the event pixel 120 When the event pixel 120 detects an event, it outputs a request to the X arbiter 122 and the Y arbiter 123 to request permission to output an event signal representing the occurrence of the event. Then, the event pixel 120 outputs an event signal to the event signal processing circuit 124 when receiving a response indicating permission to output the event signal from each of the X arbiter 122 and the Y arbiter 123 .
  • the X arbiter 122 and the Y arbiter 123 arbitrate requests requesting the output of event signals supplied from the plurality of event pixels 120 respectively, and respond based on the arbitration results (permission/non-permission of event signal output), and , sends a reset signal for resetting event detection to the event pixel 120 that output the request.
  • the event signal processing circuit 124 performs predetermined signal processing on the event signal input from the event pixel 120 to generate and output event data.
  • the change in the photocurrent generated by the event pixel 120 can also be regarded as the change in the amount of light (luminance change) incident on the photoelectric conversion portion of the event pixel 120 . Therefore, an event can also be said to be a light amount change (brightness change) of the event pixel 120 exceeding a predetermined threshold.
  • the event data representing the occurrence of an event includes at least position information such as coordinates representing the position of the event pixel 120 where the light intensity change as the event has occurred.
  • the event data can include the polarity of the change in the amount of light in addition to the positional information.
  • the event data is the relative time when the event occurred. It can be said that it implicitly includes time information representing
  • the event signal processing circuit 124 includes time information, such as a time stamp, that indicates the relative time when the event occurred, in the event data before the interval between the event data is no longer maintained as it was when the event occurred. good too.
  • the system control circuit 125 is composed of a timing generator for generating various timing signals, and controls the X arbiter 122, Y arbiter 123, event signal processing circuit 124, etc. based on the various timings generated by the timing generator. drive control.
  • the output I/F 126 sequentially outputs the event data output in units of rows from the event signal processing circuit 124 to the information processing device 20 as an event stream.
  • the information processing device 20 for example, the light source recognition unit 21 or the event data processing unit (not shown)
  • control of the robot arm 70 is executed based on the event image data acquired by the EVS 12 . Therefore, in this embodiment, not only the position and orientation of each surgical tool 80 but also the individual surgical tool 80 and its type can be specified from the event image data.
  • the light emitting part 51 blinks in a light emitting pattern unique to the .
  • FIG. 9 is a waveform diagram showing an example of a light emitting pattern for each surgical tool according to the present embodiment.
  • the surgical tools A to C may be the same type of surgical tool 80 or may be different types of surgical tools 80 .
  • the light emitting unit 51 attached to the surgical tool A emits light (hereinafter referred to as "surgical tool A") in a light emitting pattern (hereinafter referred to as “light emitting pattern A”) in which lighting and extinguishing are repeated at a period A, for example. (also referred to as “light emission”).
  • the light-emitting unit 51 attached to the surgical tool B emits light in a light-emitting pattern (hereinafter referred to as “light-emitting pattern B”) in which lighting and extinguishing are repeated at a cycle B different from the cycle A (hereinafter referred to as “light emission of the surgical tool B”). ) to do.
  • the light emitting unit 51 attached to the surgical instrument C emits light (hereinafter referred to as “surgical instrument C (also referred to as “light emission”).
  • surgical instrument C also referred to as “light emission”
  • the periods A to C are preferably periods that are not in a multiple or divisor relationship with each other.
  • the event pixels 120 on which the images of the light-emitting units 51 of the surgical tools A to C in the EVS 12 are formed are illuminated at the timing when the surgical tools A-C are turned on, that is, at the timing when the drive signal supplied to the light-emitting units 51 rises.
  • a positive event is detected to generate event data (hereinafter also referred to as positive event data)
  • Negative events are detected to generate event data (hereinafter also referred to as negative event data).
  • the light source recognition unit 21 in the information processing device 20 identifies the generation cycle of the positive event or the negative event from the event image data input at a predetermined frame rate, so that each event generation region is determined by the surgical tools A to C. It is possible to specify which of the The position and orientation of each of the surgical instruments A to C can be specified, for example, from the position and orientation of the endoscope 60 and the position and shape or distribution of the occurrence area of each event in the event image data.
  • each surgical tool 80 or its type can be specified by the light emission pattern, for example, by taking the difference (luminance difference) between frames of the RGB image data acquired by the image sensor 11.
  • the RGB image data By using the event image data instead of , not only is it possible to specify the individual surgical instruments 80 or their types at a higher frame rate, but there is no need to calculate differences between frames, so individual of the surgical instrument 80 or its type. This means that even if the frame rate of the event image data is the same as the frame rate of the RGB image data, it is possible to identify each surgical tool 80 or its type at a higher speed. .
  • the EVS 12 can capture luminance changes at extremely high speed (eg, 1000 fps). Therefore, even if the position of the surgical instrument 80 moves, it is expected that the movement distance between frames is small. Therefore, in the present embodiment, as shown in FIG. 10, the event is narrowed down to events occurring in a region (hereinafter referred to as a search pixel range R2) in the vicinity of the pixel region R1 where the event is detected, and recognition processing is executed for the next and subsequent frames. This reduces the weight and speed of the recognition process and suppresses erroneous recognition.
  • the search pixel range R2 is based on the pixel region R1 where the event is detected, and the movement of the angle of view of the imaging device 10 is calculated using the optical flow of the entire event image data and the movement information of the robot arm 70. may be set as Further, when the angle of view has moved significantly, the setting of the search pixel range R2 may be canceled and the entire event image data may be searched.
  • the pixel region R1 in which the event is detected normally has a certain degree of spread. Therefore, for example, the barycentric coordinates of the pixel region R1 may be used as the reference position when setting the search pixel range R2.
  • the range may be set and adjusted in consideration of the shape and size of the pixel region R1 in addition to the barycentric coordinates obtained in the current frame. For example, when the light emitting unit 51 is close to the tip of the scope 64 of the endoscope 60, the image of the light emitting unit 51 in the event image data becomes large and many pixels detect the event. Therefore, the size of the search pixel range R2 may be adjusted according to the distance between the tip of the scope 64 and the light emitting section 51 .
  • a stereoscopic triangulation method using two or more event image data consecutive in time series. may be used.
  • a stereoscopic triangulation method using two or more event image data acquired from each viewpoint may be used.
  • the distance between the tip of the scope 64 and the light emitting section 51 may be measured using a distance measuring sensor such as a ToF (Time of Flight) sensor.
  • the position of the light emitting section 51 in the three-dimensional space may be specified.
  • the position of the light emitting unit 51 in the three-dimensional space it is possible to use a triangulation method, a distance measuring sensor, or the like, similarly to the distance measurement.
  • a triangulation method e.g., a triangulation method, a distance measuring sensor, or the like.
  • FIG. 12 is a flowchart showing an operation example of the light emitting device according to this embodiment.
  • the light emission control unit 52 of the light emitting device 50 sets a light emission pattern for each surgical tool 80 or its type (step S101).
  • the light emission control unit 52 recognizes the surgical tool 80 and automatically sets the light emission pattern assigned in advance to the recognized surgical tool 80.
  • surgical staff, mechanical staff, or the like may manually set the surgical instruments 80 used in the surgery.
  • the light emission control section 52 After setting the light emission pattern for each surgical instrument 80 in this way, the light emission control section 52 next inputs a drive signal to each surgical instrument 80 to cause the light emitting section 51 of each surgical instrument 80 to emit light. Light is emitted in a pattern (step S102).
  • the light emission control unit 52 determines whether or not to end this operation based on the end of surgery or the stoppage of use of the surgical instrument 80 (step S103). exit. On the other hand, if this operation is not finished (NO in step S103), the light emission control unit 52 returns to step S102 and continues the subsequent operations.
  • FIG. 13 is a flowchart showing an operation example of the information processing apparatus according to this embodiment. Note that this description focuses on the operation of a control unit (for example, corresponding to the CPU 1100 in FIG. 20 described later) that controls each unit of the information processing apparatus 20 .
  • a control unit for example, corresponding to the CPU 1100 in FIG. 20 described later
  • the control unit of the information processing device 20 stores event data that is irregularly input from the EVS 12 in a memory (not shown) (for example, a memory that will be described later). 20) is started (step S111).
  • the event data to be accumulated may be only positive event data, only negative event data, or both positive event data and negative event data. Further, each event data may include a time stamp (time), position information (coordinates), and polarity of light amount change.
  • control unit determines whether or not the frame period of the current frame has ended (step S112), and when the frame period of the current frame ends (YES in step S112), the events accumulated during the current frame period Event image data is generated from the data (step S113).
  • control unit inputs the generated event image data to the light source recognition unit 21, thereby recognizing the light emission pattern of each pixel in the event image data in which an event has occurred (step S114). For example, in the example shown in FIG. 9, emission patterns A to C are recognized. Note that the recognition of the light emission pattern in this step may use not only the event image data of the current frame but also the event image data of one or more previous frames stored in a memory (not shown) or the like.
  • control unit selects an unselected one from among the light emission patterns recognized in step S114 (step S115). For example, in the example shown in FIG. 9, one unselected light emission pattern is selected from the light emission patterns A to C recognized in step S114.
  • control unit extracts the event data accumulated for each of the pixels included in the search pixel range R2 set in the initial settings or in the process for the previous frame (step S116).
  • control unit extracts a pixel group that matches the light emission pattern selected in step S115 from the pixels included in the search pixel range R2 (step S117), and extracts the extracted pixel group.
  • barycentric coordinates (light source position) are calculated (step S118).
  • step S118 based on the barycentric coordinates calculated in step S118, the shape and size of the pixel group extracted in step S117, the optical flow of the entire event image data, the movement information of the robot arm 70, and the like, the control unit Then, the search pixel range R2 is updated (step S119).
  • control unit determines whether or not the processes of steps S115 to S119 have been executed for all the light emission patterns recognized in step S114 (step S120), and unselected light emission patterns remain. If so (NO in step S120), the process returns to step S115, and the subsequent processing is executed for the unselected light emission pattern.
  • step S120 the control unit controls the barycentric coordinates calculated in step S118 (and, if necessary, the pixel group extracted in step S117).
  • the position and orientation of the surgical instrument 80 are specified by inputting the shape and size thereof into the position and orientation recognition unit 22 (step S121).
  • control unit places the treatment section 82 of the surgical instrument 80 in a position and orientation that allows the surgeon, etc., to easily view the treatment section 82 within the angle of view.
  • Control information for controlling the endoscope 60 is generated (step S122).
  • the generated control information is input to the control section 41 of the endoscope control device 40 . Note that the processing of steps S115 to S122 may be performed by the position/orientation recognition unit 22. FIG.
  • control unit determines whether or not to end this operation based on the end of the surgery or the stoppage of use of the surgical instrument 80 (step S123). do. On the other hand, if this operation is not finished (NO in step S123), the control unit returns to step S112 and continues the subsequent operations.
  • FIG. 14 is a flowchart showing an operation example of the endoscope control device 40 according to this embodiment.
  • the control unit 41 of the endoscope control device 40 acquires control information from the information processing device 20 after activation (step S131)
  • the endoscope 60 is controlled based on the acquired control information. to a desired position and attitude (step S132).
  • the generated drive signal is supplied to the drive section 42 .
  • the robot arm 70 is driven, and the position and posture of the endoscope 60 are controlled to desired positions and postures.
  • control unit 41 determines whether or not to end this operation based on the end of the surgery or the stoppage of use of the surgical instrument 80 (step S133). finish. On the other hand, if this operation is not finished (NO in step S133), the control unit returns to step S131 and continues the subsequent operations.
  • the event image data obtained by the EVS 12 is used instead of the image data obtained by the image sensor 11 to recognize the position and orientation of the surgical instrument 80. Therefore, recognition processing can be performed at a frame rate higher than that of a general image sensor. Thereby, it becomes possible to improve the control speed of the robot arm.
  • the light-emitting unit 51 attached to each surgical instrument 80 blinks in a unique light-emitting pattern, and the blinking is detected by the EVS 12. Therefore, the type of each surgical instrument 80 can be identified at high speed with a small amount of processing. It becomes possible to specify its position and orientation. Thereby, it becomes possible to suppress the deterioration of the control accuracy of the robot arm.
  • the event image data acquired by the EVS 12 is used to recognize the type of surgical instrument 80 and its position and orientation.
  • the RGB image data acquired by the image sensor 11 may be used to recognize the type of surgical instrument 80 and its position and orientation.
  • the endoscope system 2 has the same configuration as the endoscope system 1 described in the first embodiment with reference to FIGS.
  • RGB image data acquired by the sensor 11 is input to the position/orientation recognition unit 22 of the information processing device 20 .
  • the position and orientation recognition unit 22 recognizes the position and orientation of the surgical tool 80 to which the light emission unit 51 is attached from the event image data and the RGB image data for each light emission pattern of the light emission unit 51 specified by the light source recognition unit 21 .
  • Recognition of the position and orientation of the surgical tool 80 is not limited to recognition processing such as pattern matching, and machine learning such as DNN, CNN, and RNN may be used, for example.
  • the position and orientation of the surgical instrument 80 recognized from the event image data and the position and orientation of the surgical instrument 80 recognized from the RGB image data. may be integrated to recognize the position and orientation of the surgical instrument 80 recognized from the final image data.
  • event image data and RGB image data may be input to a learned learning model, and the position and orientation of the surgical tool 80 may be output as the estimation result.
  • the frame rate of the EVS 12 is higher than the frame rate of the image sensor 11, it is not always possible to combine event image data and RGB image data for all frames. In that case, the event image data and the RGB image data may be used together only for frames in which the event image data and the RGB image data can be used together.
  • the light emission of the light emitting unit 51 by the light emitting device 50 and the frame data generation of the event image data by the information processing device 20 are executed independently.
  • the information processing device 20 controls the light emission of the light emitting unit 51 by the light emitting device 50 as an example.
  • FIG. 17 is a schematic diagram showing a schematic configuration example of an endoscope system according to the third embodiment.
  • 18 is a block diagram showing the functional configuration of the endoscope system shown in FIG. 17.
  • FIG. 17 is a schematic diagram showing a schematic configuration example of an endoscope system according to the third embodiment.
  • 18 is a block diagram showing the functional configuration of the endoscope system shown in FIG. 17.
  • the endoscope system 3 according to the present embodiment has a configuration similar to that of the endoscope system 2 described in the second embodiment with reference to FIGS.
  • the processing device 20 further includes a light emission instruction section 24 .
  • the present example is based on the endoscope system 2 according to the second embodiment, it is not limited to this.
  • the endoscope system 1 according to the first embodiment may be used as a base. good.
  • the light emission instruction unit 24 outputs light emission control information for causing the light emission unit 51 to emit light in synchronization with the generation timing of the frame data of the event image data by the information processing device 20 to the light emission control unit 52 of the light emission device 50 .
  • the light emission control section 52 generates a driving signal for causing the light emitting section 51 to emit light based on the input light emission control information, and inputs the generated driving signal to the light emitting section 51 .
  • the light emitting unit 51 is controlled to emit light in synchronization with the generation timing of the frame data of the event image data by the information processing device 20 .
  • the information processing apparatus 20 can cause the light emitting unit 51 of each surgical instrument 80 to emit light when necessary. Therefore, it is possible to recognize the type of surgical instrument 80 and its position and orientation more accurately. For example, when it is desired to display an image focusing on a specific surgical tool 80 such as an electric scalpel on the display device 30, the specific surgical tool 80 is selectively caused to emit light so that the specific surgical tool can be displayed more quickly and accurately. Since the position and posture of the robot arm 70 can be recognized, the reliability of the control of the endoscope 60 by the robot arm 70 can be enhanced.
  • each pixel 110 of the image sensor 11 may be supplied with a reset signal for resetting the charge accumulated in each pixel 110 by photoelectric conversion at the timing when the blinking period ends, that is, immediately before the start of the exposure period. .
  • the blinking of the light emitting unit 51 gives RGB image data. It is possible to minimize the impact.
  • FIG. 20 is a hardware configuration diagram showing an example of a computer 1000 that implements the functions of the information processing device 20, the endoscope control device 40, and the light emitting device 50.
  • the computer 1000 has a CPU 1100 , a RAM 1200 , a ROM (Read Only Memory) 1300 , a HDD (Hard Disk Drive) 1400 , a communication interface 1500 and an input/output interface 1600 . Each part of computer 1000 is connected by bus 1050 .
  • the CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400 and controls each section. For example, the CPU 1100 loads programs stored in the ROM 1300 or HDD 1400 into the RAM 1200 and executes processes corresponding to various programs.
  • the ROM 1300 stores a boot program such as BIOS (Basic Input Output System) executed by the CPU 1100 when the computer 1000 is started, and programs dependent on the hardware of the computer 1000.
  • BIOS Basic Input Output System
  • the HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs.
  • HDD 1400 is a recording medium that records a program for realizing each operation according to the present disclosure, which is an example of program data 1450 .
  • a communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (for example, the Internet).
  • the CPU 1100 receives data from another device via the communication interface 1500, and transmits data generated by the CPU 1100 to another device.
  • the input/output interface 1600 includes the I/F section 18 described above, and is an interface for connecting the input/output device 1650 and the computer 1000 .
  • the CPU 1100 receives data from input devices such as a keyboard and mouse via the input/output interface 1600 .
  • the CPU 1100 transmits data to an output device such as a display, a speaker, or a printer via the input/output interface 1600 .
  • the input/output interface 1600 may function as a media interface for reading a program or the like recorded on a predetermined recording medium.
  • Media include, for example, optical recording media such as DVD (Digital Versatile Disc) and PD (Phase change rewritable disk), magneto-optical recording media such as MO (Magneto-Optical disk), tape media, magnetic recording media, semiconductor memories, etc. is.
  • optical recording media such as DVD (Digital Versatile Disc) and PD (Phase change rewritable disk)
  • magneto-optical recording media such as MO (Magneto-Optical disk)
  • tape media magnetic recording media
  • magnetic recording media semiconductor memories, etc. is.
  • the CPU 1100 of the computer 1000 executes a program loaded on the RAM 1200 to , the functions of the information processing device 20, the endoscope control device 40, and the light emitting device 50 are realized.
  • the HDD 1400 also stores programs and the like according to the present disclosure.
  • CPU 1100 reads and executes program data 1450 from HDD 1400 , as another example, these programs may be obtained from another device via external network 1550 .
  • the present technology can also take the following configuration.
  • a position and orientation recognition unit that a control instruction unit that generates control information for controlling a robot arm that supports the medical device with the EVS based on the recognized position and orientation of the surgical instrument;
  • Information processing device (2) The information processing apparatus according to (1), wherein the position and orientation recognition unit recognizes the position and orientation of the surgical instrument based on frame data generated from event data input within a predetermined frame period.
  • the position/orientation recognition unit cancels the setting of the search range and recognizes the position and orientation of the surgical instrument based on the entire frame data.
  • the information processing apparatus according to any one of 5).
  • the position and orientation recognition unit recognizes the position and orientation of the surgical instrument by machine learning using the event data as input.
  • the position and orientation recognition unit recognizes the position and orientation of the surgical instrument further based on image data input from an image sensor sharing an optical axis with the EVS;
  • the information processing device according to .
  • (9) further comprising a light emission instruction unit that generates a light emission control signal for controlling the light emission period of the light emitting unit;
  • the light from the light emitting unit is light in the visible light range
  • (10) further comprising a light source recognition unit that individually identifies the surgical tool or the type of the surgical tool based on the event detected by each of the pixels;
  • the information processing apparatus according to any one of (1) to (9), wherein the position and orientation recognition unit recognizes the position and orientation of each surgical tool.
  • (11) further comprising a light emission instruction unit that generates a light emission control signal for controlling the light emission pattern of the light emission unit;
  • the light source recognizing unit identifies a light emission pattern of the light emitting unit based on the event detected by each of the pixels, and individually identifies the surgical tool or the type of the surgical tool based on the specified light emission pattern.
  • the information processing apparatus according to (10).
  • (12) The information processing apparatus according to (11), wherein the light emission instruction unit generates the light emission control signal so that the light emission units provided in different surgical tools or different types of surgical tools emit light in different light emission patterns.
  • a light-emitting portion provided on the surgical tool; a medical device comprising an EVS comprising a plurality of pixels for detecting changes in luminance of light from the light emitting unit as an event; an information processing device that generates control information for controlling a robot arm that supports the medical device based on event data input from the EVS; a control device that controls the robot arm based on the control information;
  • An information processing system comprising (14) The information processing device is a position and orientation recognition unit that recognizes the position and orientation of the surgical instrument based on event data input from the EVS; a control instruction unit that generates control information for controlling a robot arm that supports the medical device with the EVS based on the recognized position and orientation of the surgical instrument; The information processing system according to (13) above.
  • Reference Signs List 1 2, 3 endoscope system 10 imaging device 11 image sensor 12 EVS 13 spectrometer 20 information processing device 21 light source recognition unit 22 position and orientation recognition unit 23 control instruction unit 24 light emission instruction unit 30 display device 40 endoscope control device 41 control unit 42 drive unit 50 light emitting device 51 light emitting unit 52 light emission control unit 60 Endoscope 61 Camera head 62 Joint part 63 Junction part 64 Scope 65 Optical fiber cable 66 Transmission cable 70 Robot arm 80 Operation tool 81 Shaft part 82 Treatment part

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Robotics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mechanical Engineering (AREA)
  • Surgery (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Medical Informatics (AREA)
  • Public Health (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Veterinary Medicine (AREA)
  • Biophysics (AREA)
  • Optics & Photonics (AREA)
  • Pathology (AREA)
  • Radiology & Medical Imaging (AREA)
  • Endoscopes (AREA)
PCT/JP2022/008495 2021-06-03 2022-03-01 情報処理装置、情報処理システム及び情報処理方法 Ceased WO2022254836A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US18/561,270 US20240246241A1 (en) 2021-06-03 2022-03-01 Information processing apparatus, information processing system, and information processing method
JP2023525399A JPWO2022254836A1 (https=) 2021-06-03 2022-03-01

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021-093796 2021-06-03
JP2021093796 2021-06-03

Publications (1)

Publication Number Publication Date
WO2022254836A1 true WO2022254836A1 (ja) 2022-12-08

Family

ID=84324154

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/008495 Ceased WO2022254836A1 (ja) 2021-06-03 2022-03-01 情報処理装置、情報処理システム及び情報処理方法

Country Status (3)

Country Link
US (1) US20240246241A1 (https=)
JP (1) JPWO2022254836A1 (https=)
WO (1) WO2022254836A1 (https=)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2015028780A (ja) * 2013-07-29 2015-02-12 三星電子株式会社Ｓａｍｓｕｎｇ Ｅｌｅｃｔｒｏｎｉｃｓ Ｃｏ．，Ｌｔｄ． イベント情報を含む映像を分析する装置及び方法並びにユーザ入力処理装置
WO2017145475A1 (ja) * 2016-02-24 2017-08-31 ソニー株式会社 医療用情報処理装置、情報処理方法、医療用情報処理システム
JP2017164007A (ja) * 2016-03-14 2017-09-21 ソニー株式会社 医療用画像処理装置、医療用画像処理方法、プログラム
JP2017530467A (ja) * 2014-09-16 2017-10-12 クゥアルコム・インコーポレイテッドＱｕａｌｃｏｍｍ Ｉｎｃｏｒｐｏｒａｔｅｄ 事象ベースのシステムとフレームベースの処理システムとをインターフェースすること
JP2017535999A (ja) * 2014-09-30 2017-11-30 クアルコム，インコーポレイテッド フレームベースとイベントベースのハイブリッド方式を使用するセンサアーキテクチャ
JP2020025263A (ja) * 2018-07-31 2020-02-13 ソニーセミコンダクタソリューションズ株式会社 積層型受光センサ及び電子機器

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5502502B2 (ja) * 2010-01-20 2014-05-28 株式会社東芝 磁気共鳴映像装置
JP5784845B2 (ja) * 2013-07-09 2015-09-24 オリンパス株式会社 撮像システム
US11244464B2 (en) * 2018-03-09 2022-02-08 Samsung Electronics Co., Ltd Method and apparatus for performing depth estimation of object
US12575891B2 (en) * 2020-11-20 2026-03-17 Auris Health, Inc. AI-based automatic tool presence and workflow/phase/activity recognition

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2015028780A (ja) * 2013-07-29 2015-02-12 三星電子株式会社Ｓａｍｓｕｎｇ Ｅｌｅｃｔｒｏｎｉｃｓ Ｃｏ．，Ｌｔｄ． イベント情報を含む映像を分析する装置及び方法並びにユーザ入力処理装置
JP2017530467A (ja) * 2014-09-16 2017-10-12 クゥアルコム・インコーポレイテッドＱｕａｌｃｏｍｍ Ｉｎｃｏｒｐｏｒａｔｅｄ 事象ベースのシステムとフレームベースの処理システムとをインターフェースすること
JP2017535999A (ja) * 2014-09-30 2017-11-30 クアルコム，インコーポレイテッド フレームベースとイベントベースのハイブリッド方式を使用するセンサアーキテクチャ
WO2017145475A1 (ja) * 2016-02-24 2017-08-31 ソニー株式会社 医療用情報処理装置、情報処理方法、医療用情報処理システム
JP2017164007A (ja) * 2016-03-14 2017-09-21 ソニー株式会社 医療用画像処理装置、医療用画像処理方法、プログラム
JP2020025263A (ja) * 2018-07-31 2020-02-13 ソニーセミコンダクタソリューションズ株式会社 積層型受光センサ及び電子機器

Also Published As

Publication number Publication date
JPWO2022254836A1 (https=) 2022-12-08
US20240246241A1 (en) 2024-07-25

Similar Documents

Publication Publication Date Title
US12226285B2 (en) Indicators that facilitate intraoral scanning
JP6722771B2 (ja) 二重相補パターン照明を用いる３ｄシーン再構成のシステムおよび方法
JP6666518B2 (ja) 計測支援装置、内視鏡システム、及びプロセッサ
JP2017195573A (ja) 撮像装置および電子機器
JP5537250B2 (ja) 内視鏡システム
JP6454489B2 (ja) 観察システム
US11490785B2 (en) Measurement support device, endoscope system, and processor measuring size of subject using measurement auxiliary light
JP7441897B2 (ja) 制御装置、内視鏡システムおよび制御装置の作動方法
US10176375B2 (en) High speed pupil detection system and method
JP7055624B2 (ja) 医療用撮像装置
US20230396746A1 (en) Image projection system and image projection method
US20190058819A1 (en) Endoscope apparatus
JP2024037129A (ja) 画像処理装置、画像処理方法および画像処理プログラム
WO2022254836A1 (ja) 情報処理装置、情報処理システム及び情報処理方法
US12495951B2 (en) Endoscope system and operation method of the same
US10779715B2 (en) Endoscope apparatus
JP4322693B2 (ja) 眼球運動測定装置
US12064092B2 (en) Control device, endoscope system, and control method
JP2020141907A (ja) 医療用光源装置
JP6370168B2 (ja) 照明撮像装置及びそれを備えた視線検出装置
US20220256125A1 (en) Control device, medical observation system, control method, and computer readable recording medium
WO2023276242A1 (ja) 医療用観察システム、情報処理装置及び情報処理方法
WO2022172733A1 (ja) 医療用観察装置、観察装置、観察方法及びアダプタ
WO2021161394A1 (ja) 画像処理装置、撮像システム、画像処理方法およびプログラム
US20210287634A1 (en) Medical image processing device, medical observation system, and method of operating medical image processing device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22815594

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023525399

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 18561270

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22815594

Country of ref document: EP

Kind code of ref document: A1