WO2023027095A1 - マニピュレーションシステムおよびマニピュレーション方法 - Google Patents

マニピュレーションシステムおよびマニピュレーション方法 Download PDF

Info

Publication number
WO2023027095A1
WO2023027095A1 PCT/JP2022/031821 JP2022031821W WO2023027095A1 WO 2023027095 A1 WO2023027095 A1 WO 2023027095A1 JP 2022031821 W JP2022031821 W JP 2022031821W WO 2023027095 A1 WO2023027095 A1 WO 2023027095A1
Authority
WO
WIPO (PCT)
Prior art keywords
sample
force
manipulator
user
control device
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/031821
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
忠義 青山
俊希 藤城
和哉 坂本
佑記 舟洞
澄和 齋藤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tokai National Higher Education and Research System NUC
Original Assignee
Tokai National Higher Education and Research System NUC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tokai National Higher Education and Research System NUC filed Critical Tokai National Higher Education and Research System NUC
Priority to CN202280062736.4A priority Critical patent/CN117940822A/zh
Priority to US18/687,203 priority patent/US20240391111A1/en
Priority to JP2023543946A priority patent/JP7847383B2/ja
Priority to EP22861383.2A priority patent/EP4379446A4/en
Publication of WO2023027095A1 publication Critical patent/WO2023027095A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J11/00—Manipulators not otherwise provided for
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J7/00—Micromanipulators
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/06—Control stands, e.g. consoles, switchboards
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
    • B25J13/085—Force or torque sensors
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J15/00—Gripping heads and other end effectors
    • B25J15/06—Gripping heads and other end effectors with vacuum or magnetic holding means
    • B25J15/0616—Gripping heads and other end effectors with vacuum or magnetic holding means with vacuum
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1679—Program controls characterised by the tasks executed
    • B25J9/1689—Teleoperation
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1694—Program controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • B25J9/1697—Vision controlled systems
    • C—CHEMISTRY; METALLURGY
    • C12—BIOCHEMISTRY; BEER; SPIRITS; WINE; VINEGAR; MICROBIOLOGY; ENZYMOLOGY; MUTATION OR GENETIC ENGINEERING
    • C12M—APPARATUS FOR ENZYMOLOGY OR MICROBIOLOGY; APPARATUS FOR CULTURING MICROORGANISMS FOR PRODUCING BIOMASS, FOR GROWING CELLS OR FOR OBTAINING FERMENTATION OR METABOLIC PRODUCTS, i.e. BIOREACTORS OR FERMENTERS
    • C12M33/00—Means for introduction, transport, positioning, extraction, harvesting, peeling or sampling of biological material in or from the apparatus
    • C—CHEMISTRY; METALLURGY
    • C12—BIOCHEMISTRY; BEER; SPIRITS; WINE; VINEGAR; MICROBIOLOGY; ENZYMOLOGY; MUTATION OR GENETIC ENGINEERING
    • C12M—APPARATUS FOR ENZYMOLOGY OR MICROBIOLOGY; APPARATUS FOR CULTURING MICROORGANISMS FOR PRODUCING BIOMASS, FOR GROWING CELLS OR FOR OBTAINING FERMENTATION OR METABOLIC PRODUCTS, i.e. BIOREACTORS OR FERMENTERS
    • C12M35/00—Means for application of stress for stimulating the growth of microorganisms or the generation of fermentation or metabolic products; Means for electroporation or cell fusion
    • C—CHEMISTRY; METALLURGY
    • C12—BIOCHEMISTRY; BEER; SPIRITS; WINE; VINEGAR; MICROBIOLOGY; ENZYMOLOGY; MUTATION OR GENETIC ENGINEERING
    • C12M—APPARATUS FOR ENZYMOLOGY OR MICROBIOLOGY; APPARATUS FOR CULTURING MICROORGANISMS FOR PRODUCING BIOMASS, FOR GROWING CELLS OR FOR OBTAINING FERMENTATION OR METABOLIC PRODUCTS, i.e. BIOREACTORS OR FERMENTERS
    • C12M41/00—Means for regulation, monitoring, measurement or control, e.g. flow regulation
    • C12M41/30—Means for regulation, monitoring, measurement or control, e.g. flow regulation of concentration
    • C12M41/36—Means for regulation, monitoring, measurement or control, e.g. flow regulation of concentration of biomass, e.g. colony counters or by turbidity measurements
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B21/00—Microscopes
    • G02B21/32—Micromanipulators structurally combined with microscopes
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Program-controlled manipulators
    • B25J9/16—Program controls
    • B25J9/1628—Program controls characterised by the control loop
    • B25J9/1633—Program controls characterised by the control loop compliant, force, torque control, e.g. combined with position control
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/40—Robotics, robotics mapping to robotics vision
    • G05B2219/40174—Robot teleoperation through internet
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/40—Robotics, robotics mapping to robotics vision
    • G05B2219/40184—Compliant teleoperation, operator controls motion, system controls contact, force

Definitions

  • the present disclosure relates to manipulation systems and manipulation methods.
  • a manipulation system for manipulating cells using a manipulator is known. Since cells are very small, the cells are manipulated while observing the positions of the cells and the manipulator under a microscope.
  • the manipulator is attached to a table that can be finely moved on three XYZ axes, and the user uses a joystick or the like to move the manipulator.
  • the present disclosure has been made in view of these problems, and one of its exemplary purposes is to provide a technique for improving operability when manipulating a sample using a manipulator.
  • a manipulation system includes a manipulator for manipulating a sample, a manipulator driving mechanism for moving the manipulator, an imaging device for imaging the sample through an objective lens, and an image captured by the imaging device, A control device that generates force information indicating the magnitude of the force sense to be presented to the user, and a force sensation that receives an input operation from the user for designating the position of the manipulator and conforms to the force information generated by the control device. and a haptic presentation device configured to present to the user.
  • Another aspect of the present disclosure is a manipulation method.
  • This method includes the steps of acquiring position information based on a user's input operation for designating the position of the manipulator from the haptic presentation device, and controlling the operation of a manipulator drive mechanism that moves the manipulator based on the acquired position information.
  • imaging a sample manipulated using a manipulator through an objective lens generating force information indicating the magnitude of the force sensation to be presented to the user based on the imaged image; generating force and controlling the operation of the force sense presentation device so that the force sense according to the information is presented to the user.
  • This manipulation system includes a manipulator for manipulating a sample, a manipulator drive mechanism for moving the manipulator, a pump for variably controlling the suction force and the ejection force at the tip of the manipulator, a holding member held by the user, and a user's operation.
  • a link mechanism for supporting the holding member so that the position of the holding member is variable according to the position of the holding member; a position sensor for detecting the position of the holding member;
  • An input operation device including a member and an angle sensor for detecting the rotation angle of the rotating member; and controlling the operation of the manipulator drive mechanism based on position information indicating the position of the holding member detected by the position sensor.
  • a control device for controlling the operation of the pump based on angle information indicating the rotation angle of the rotation member detected by the angle sensor.
  • Yet another aspect of the present disclosure is a manipulation method.
  • This method includes the steps of acquiring position information indicating the position of a holding member held by a user and supported by a link mechanism so that the position of the holding member is variable according to the user's operation; Acquiring angle information indicating a rotation angle of a rotating member that rotates with respect to a holding member by operation; and operating a manipulator drive mechanism that moves a manipulator for manipulating a sample based on the acquired position information. and controlling the operation of a pump that variably controls the suction force and ejection force at the tip of the manipulator based on the obtained angle information.
  • FIG. 1 is a diagram schematically showing the configuration of a manipulation system according to a first embodiment
  • FIG. FIG. 4 is a diagram schematically showing a working distance considering the influence of refractive index
  • FIG. 4 is a diagram schematically showing a method of specifying position coordinates of a sample in the z direction
  • 10 is a graph showing the correlation between average edge intensity approximated by Gaussian distribution and working distance.
  • FIG. 10 is a diagram schematically showing a measurement pattern of a manipulator when deriving coordinate transformation parameters
  • FIGS. 6A to 6J are diagrams showing captured images within the field of view of the objective lens and display examples of the three-dimensional display device.
  • 4 is a flow chart showing a three-dimensional position presentation method according to an embodiment
  • FIG. 4 is a flowchart showing a haptic presentation method according to an embodiment
  • FIG. 10 is a diagram schematically showing the configuration of a manipulation system according to a second embodiment
  • FIG. 11(a) and 11(b) are diagrams schematically showing the configuration of the second force sense presentation device.
  • FIG. 4 is a diagram schematically showing a method of determining contact between a sample and a manipulator
  • FIG. 4 is a diagram schematically showing how a sample is manipulated by a manipulator
  • FIG. 4 is a diagram schematically showing how a sample is manipulated by a manipulator
  • FIG. 4 is a diagram schematically showing how a sample is manipulated by a manipulator
  • FIG. 4 is a diagram schematically showing how a sample is manipulated by a manipulator
  • the present disclosure relates to manipulation systems.
  • the manipulation system includes an optical microscope for observing samples such as cells, and a manipulator for manipulating the samples.
  • a user manipulates the sample by moving the manipulator while observing the sample and the manipulator with an optical microscope.
  • the manipulator is position-controlled by a driving mechanism such as an actuator in order to enable fine manipulation in units of micrometers.
  • a driving mechanism such as an actuator in order to enable fine manipulation in units of micrometers.
  • the force applied to the cell by the manipulator is very small, on the order of 1 ⁇ N to 1 mN (0.1 mgf to 0.1 gf), and it is difficult to detect the force in real time using a sensor or the like.
  • the amount of change in at least one of the position and shape of the sample manipulated by the manipulator is specified in real time, and the force applied to the sample is estimated based on the specified amount of change. Furthermore, by amplifying the estimated force by a factor of 100 to 10,000 and feeding it back through a haptic device, the user can manipulate the sample while feeling the reaction force applied to the manipulator. According to the present disclosure, it is possible to present a haptic sensation as if a cell were being directly manipulated by hand, so that it is possible to improve the operability of the manipulation system. In particular, it can help beginners with little experience in cell manipulation to master the skill.
  • FIG. 1 is a diagram schematically showing the configuration of a manipulation system 10 according to the first embodiment.
  • Manipulation system 10 comprises an inverted microscope configuration.
  • the manipulation system 10 includes a stage 12, an illumination device 14, a manipulator 16, a folding mirror 18, an objective lens 20, a variable focus lens 22, an imaging device 24, a manipulator drive mechanism 26, and a lens drive mechanism 28. , a control device 30 , a display device 32 , an input device 34 , and a haptic presentation device 36 .
  • a first coordinate system is set with the optical axis A of the objective lens 20 on the stage 12 as a reference.
  • the direction in which the optical axis A of the objective lens 20 extends on the stage 12 is the z-direction, and the directions perpendicular to the optical axis A are the x-direction and the y-direction.
  • the direction in which the optical axis A of the objective lens 20 extends on the stage 12 coincides with the direction orthogonal to the support surface 12a of the stage 12.
  • FIG. The direction in which the optical axis A of the objective lens 20 extends on the stage 12 may deviate from the direction orthogonal to the support surface 12a of the stage 12.
  • the stage 12 has a support surface 12a for horizontally supporting the sample 40, and an opening 12b for allowing observation light 42 from the sample 40 to pass therethrough.
  • the sample 40 to be manipulated is not particularly limited, but cells of humans, animals, and the like can be manipulated.
  • the sample 40 is accommodated in a sample plate 46 made of a transparent material such as resin or glass, and the sample plate 46 is placed on the stage 12 .
  • the sample 40 is suspended in a liquid 48 such as water contained in a sample pan 46, for example.
  • the illumination device 14 is provided above the stage 12 and illuminates the sample 40 on the stage 12 .
  • the illumination device 14 projects illumination light 44 such as white light toward the sample 40 .
  • Illumination device 14 is configured to provide transmitted illumination.
  • the illumination device 14 may be capable of projecting visible illumination light 44 of a specific wavelength selected for fluorescence observation or the like.
  • the illumination device 14 projects, for example, illumination light 44 that makes the illuminance distribution on the stage 12 uniform.
  • the manipulator 16 is provided on the stage 12 and used for manipulating the sample 40 .
  • the manipulator 16 includes a holding pipette 16a and an injection pipette 16b.
  • the holding pipette 16a is used to fix the cell
  • the injection pipette 16b is used to perform cell manipulation such as gene introduction into the cell.
  • two manipulators are provided in the illustrated example, the number of manipulators may be one, or three or more.
  • the folding mirror 18 is provided directly below the opening 12b of the stage 12. Folding mirror 18 is arranged to reflect observation light 42 from sample 40 towards objective lens 20 .
  • the optical axis A of the objective lens 20 is folded back by the folding mirror 18, but the objective lens 20 may be arranged on the optical axis extending in the z direction without providing the folding mirror 18. good.
  • the objective lens 20 is arranged at a position where the observation light 42 from the folding mirror 18 is incident.
  • the objective lens 20 is arranged at a position away from the folding mirror 18 in the +x direction.
  • the objective lens 20 preferably has a relatively long working distance (WD).
  • WD working distance
  • Specifications such as magnification and working distance of the objective lens 20 are not particularly limited, but for example, an ultra-long working type objective lens having a working distance of 20 mm to 40 mm at a magnification of 10 times to 50 times can be used. .
  • the variable focus lens 22 is arranged at a position where the observation light 42 that has passed through the objective lens 20 is incident.
  • a variable focus lens 22 is positioned between the objective lens 20 and the imager 24 , eg adjacent or close to the objective lens 20 .
  • the variable focus lens 22 is configured such that its refractive power is variable within a predetermined range.
  • the variable focus lens 22 may be a convex lens having only positive refractive power, may be a concave lens having only negative refractive power, or may be configured to switch between positive and negative refractive power. .
  • the variable focus lens 22 is composed of, for example, a liquid lens, and is configured to change the focal length by deforming a flexible transparent film that seals the liquid lens.
  • the shape of the transparent membrane is controlled by changing the pressure applied to the transparent membrane.
  • the focal length of the variable focus lens 22 can be electrically controlled by using an electromagnetic actuator or a piezoelectric element.
  • the variable focus lens 22 is configured, for example, to vary the effective working distance of the combination of the objective lens 20 and the variable focus lens 22 within a range of about 2 mm.
  • the imaging device 24 captures the observation light 42 that has passed through the variable focus lens 22 and generates a captured image.
  • the imaging device 24 has an imaging lens 24a and an imaging element 24b.
  • the imaging lens 24a forms an image of the observation light 42 on the imaging element 24b.
  • the imaging device 24b is an image sensor such as a CMOS sensor, and is capable of generating captured images at a high frame rate.
  • the frame rate of the imaging device 24 is not particularly limited, but is preferably 100 frames per second or higher, more preferably 500 frames per second or higher.
  • the objective lens 20, the variable focus lens 22, and the imaging device 24 are arranged along the optical axis A extending in the x direction, and fixed to, for example, a lens barrel extending in the x direction.
  • An additional folding mirror (not shown) may be provided between the variable focus lens 22 and the imaging device 24, and the optical axis A may be further folded.
  • the manipulator driving mechanism 26 moves the manipulator 16 and makes the three-dimensional position of the manipulator 16 variable.
  • the manipulator drive mechanism 26 includes a first drive mechanism 26a and a second drive mechanism 26b.
  • the first drive mechanism 26a is configured to move the holding pipette 16a and vary the three-dimensional position of the holding pipette 16a.
  • the second drive mechanism 26b is configured to move the injection pipette 16b and vary the three-dimensional position of the injection pipette 16b.
  • the three-dimensional positions of holding pipette 16a and manipulator 16 are controllable independently of each other.
  • the lens drive mechanism 28 drives the variable focus lens 22 and changes the refractive power of the variable focus lens 22 .
  • the lens driving mechanism 28 changes the effective working distance of the combination of the objective lens 20 and the variable focus lens 22 by changing the refractive power of the variable focus lens 22 .
  • the effective working distance is the distance from the tip of the objective lens 20 to the focal position of the observation light 42, and from the tip of the objective lens 20 to the focal plane on which the captured image captured by the imaging device 24 is in focus. is the distance of
  • the control device 30 controls the overall operation of the manipulation system 10.
  • the control device 30 can be implemented in terms of hardware by elements such as a CPU and memory of a computer, or mechanical devices, and in terms of software, by a computer program or the like.
  • the control device 30 is configured by, for example, a general-purpose personal computer.
  • the display device 32 includes a three-dimensional display device 32a and a two-dimensional display device 32b.
  • the three-dimensional display device 32a stereoscopically displays the three-dimensional positions of the sample 40 and the manipulator 16.
  • the three-dimensional display device 32a is, for example, a hologram display such as Looking Glass, and is a display device that enables stereoscopic vision without using 3D glasses or the like.
  • a computer graphic (CG) image simulating the sample 40 and the manipulator 16 is displayed on the three-dimensional display device 32a.
  • the three-dimensional display device 32a may be a head-mounted virtual reality (VR) display device.
  • VR virtual reality
  • the two-dimensional display device 32b is a liquid crystal display or the like, and displays captured images captured by the imaging device 24 in real time.
  • the two-dimensional display device 32b may display the three-dimensional positions of the sample 40 and the manipulator 16, and a rendered image generated by perspectively projecting the sample 40 and the manipulator 16 mapped in the virtual space onto an arbitrary observation plane. may be displayed.
  • the input device 34 is a device for performing input operations to the control device 30 and operating the manipulator 16 .
  • a mouse, a keyboard, or the like can be used as input operation means for the control device 30 .
  • a joystick or the like can be used as an operating means for the manipulator 16 .
  • an input device 34 such as a joystick, the tip position of the manipulator 16 can be moved on the order of micrometers, and the sample 40 can be manipulated precisely.
  • the haptic presentation device 36 is an operating means for the manipulator 16 and is configured to receive an input operation from the user for designating the position of the manipulator 16 .
  • the force sense presentation device 36 has a multi-joint arm 36a, and is configured such that a tip 36b of the multi-joint arm 36a can move along three axes of XYZ.
  • the user performs an input operation for designating the three-dimensional position of the manipulator 16 by gripping and moving the tip 36b of the articulated arm 36a.
  • the force sense presentation device 36 has, for example, a sensor for specifying the position of the tip 36b of the articulated arm 36a, and transmits position information to the control device 30 based on the position of the tip 36b of the articulated arm 36a.
  • the haptic presentation device 36 is also haptic presentation means for presenting the user with the force applied to the manipulator 16 during cell manipulation.
  • the force sense presentation device 36 is configured to be capable of presenting a force sense along three axes of XYZ at the distal end 36b of the articulated arm 36a.
  • the force sense presentation device 36 has an actuator for applying a reaction force to the articulated arm 36 a and controls the operation of the actuator based on force information transmitted from the control device 30 .
  • an input device 34 such as a joystick is used to operate the holding pipette 16a
  • a haptic presentation device 36 is used to operate the injection pipette 16b.
  • the force sense presentation device 36 may be used to operate the holding pipette 16a.
  • the three-dimensional positions of sample 40 and manipulator 16 are identified in real time, and the identified three-dimensional positions are used to estimate the force applied between sample 40 and manipulator 16 . Therefore, first, a method for identifying the three-dimensional positions of the sample 40 and the manipulator 16 in real time will be described.
  • the control device 30 Based on the captured image captured by the imaging device 24, the control device 30 specifies the three-dimensional position of the sample 40 included in the captured image.
  • the control device 30 identifies the three-dimensional position of the sample 40 using the first coordinate system based on the optical axis A of the objective lens 20 .
  • the control device 30 identifies the sample 40 included in the captured image using image recognition technology, and identifies the x-direction and y-direction position coordinates of the sample 40 from the center position of the sample 40 in the captured image.
  • the working distance WD is the focal length f1 of the objective lens 20, the focal length f2 of the variable focus lens 22, the distance d between the objective lens 20 and the variable focus lens 22, and the refraction of the optical path from the objective lens 20 to the sample 40. It can be calculated based on the rate distribution.
  • the synthetic focal length f0 corresponds to the working distance when the optical path from the objective lens 20 to the sample 40 is air and the refractive index is approximately one.
  • the sample dish 46 and the liquid 48 are present in the optical path from the objective lens 20 to the sample 40 . Therefore, the actual working distance WD deviates from the composite focal length f 0 due to the refractive index of the sample dish 46 and the liquid 48 .
  • FIG. 2 is a diagram schematically showing the working distance WD considering the influence of the refractive index.
  • the observation light 42 entering the objective lens 20 is refracted due to the presence of the sample dish 46 and the liquid 48 in the optical path from the objective lens 20 to the sample 40 .
  • the actual working distance WD is longer than the combined focal length f 0 of objective lens 20 and variable focus lens 22 without sample dish 46 and liquid 48 present.
  • the actual working distance WD is the combined focal length f 0 , the refractive index n 0 of air, the refractive index n 1 of the sample dish 46, the refractive index n 2 of the liquid 48, the distance a from the objective lens 20 to the sample dish 46, the sample Using the thickness b of the plate 46 and the effective radius r when the objective lens 20 is assumed to be an ideal plano-convex lens, it can be expressed by the following equation (1). Equation (1) can be derived based on geometric relationships based on Snell's law.
  • the synthetic focal length f 0 21.059 mm
  • the refractive index n 1 of the polystyrene (PS) sample dish 46 1.592
  • distance a 19.135 mm
  • thickness b 1.0 mm
  • effective radius r 5.0 mm
  • the working distance WD 21.894 mm.
  • the difference between the working distance WD and the combined focal length f0 is 0.835 mm, which is very large compared to the depth of field of the objective lens 20 (approximately 0.03 mm) and the cell size (approximately 0.1 mm). to big. Therefore, by correcting the working distance WD in consideration of the influence of the refractive index distribution from the objective lens 20 to the sample 40, the position coordinates of the sample 40 in the z direction can be specified accurately.
  • the control device 30 controls the lens driving mechanism 28 to change the working distance WD.
  • the control device 30 specifies the position coordinates of the sample 40 in the z direction based on a plurality of captured images captured while changing the working distance WD. Specifically, the average edge strength F of the sample 40 included in each of the plurality of captured images is calculated, and the correlation between the working distance WD and the average edge strength F is approximated by a Gaussian distribution. Identify the position coordinates in the z direction.
  • the average edge strength F is obtained by calculating the edge strength f(x, y) of each pixel of the captured image and averaging the edge strength f(x, y) of all pixels in the region including the sample 40 .
  • the average edge strength F indicates the contrast of the sample 40 included in the captured image, and the larger the average edge strength F, the more focused the sample 40 is captured with high contrast.
  • FIG. 3 is a diagram schematically showing a method of specifying the position coordinates of the sample 40 in the z direction.
  • the working distance WD of the objective lens 20 is set to a first distance z 1 , a second distance z 2 and a third distance z 3 (z 1 ⁇ z 2 ⁇ z 3 ).
  • Captured images 52a, 52b, and 52c obtained by capturing focal planes 50a, 50b, and 50c located at .
  • the central coordinate z of the sample 40 is located between the first distance z1 and the second distance z2 , and the sample 40 intersects the second focal plane 50b.
  • the sample 40 in focus is included in the second captured image 52b obtained by capturing the second focal plane 50b.
  • the first captured image 52a obtained by capturing the first focal plane 50a away from the sample 40 includes the sample 40 slightly out of focus.
  • a third captured image 52c obtained by capturing a third focal plane 50c further away from the sample 40 includes the sample 40 out of focus.
  • the interval ⁇ z among the first distance z 1 , the second distance z 2 and the third distance z 3 is set to a value about 1 to 2 times the size of the sample 40 . For example, if the size of the sample 40 is about 100 ⁇ m, ⁇ z is about 100 ⁇ m to 200 ⁇ m.
  • the control device 30 obtains a plurality of captured images 52a to 52c captured while changing the working distance WD, and calculates the average edge strengths F 1 and F 2 of the regions 54a to 54c including the sample 40 in each of the captured images 52a to 52c. , F3 .
  • the magnitude relationship of the average edge strengths of the captured images 52a to 52c is F 3 ⁇ F 1 ⁇ F 2 .
  • the controller 30 approximates the correlation between the calculated average edge strengths F 1 , F 2 , F 3 and the distances z 1 , z 2 , z 3 to the respective focal planes 50a to 50c with a Gaussian distribution.
  • FIG. 4 is a graph showing the relationship between the average edge strengths F 1 , F 2 , F 3 approximated by Gaussian distribution and the working distance WD.
  • the average edge intensities F 1 to F 3 can be approximated by a Gaussian distribution to specify the maximum value F max .
  • the maximum value F max corresponds to the best focused position of the sample 40
  • the working distance z 0 corresponding to the maximum value F max can be regarded as the z-coordinate of the center of the sample 40 .
  • the working distance z0 corresponding to the local maximum value Fmax is calculated using the average edge strengths F1 , F2 , and F3 and the distances z1 , z2 , and z3 to each focal plane by the following formula: It can be calculated using (2).
  • the control device 30 identifies the three-dimensional position of the manipulator 16 based on the operation of the manipulator drive mechanism 26.
  • the controller 30 first identifies the three-dimensional position of the manipulator 16 using a second coordinate system based on the drive shaft of the manipulator 16 .
  • the three-dimensional position of the manipulator 16 with respect to the second coordinate system is relatively specified by calculating the movement amount of the manipulator 16 in the three-dimensional direction from the rotation angle of the motor of the manipulator drive mechanism 26 and the drive amount of the actuator. can.
  • the second coordinate system is set for each of the holding pipette 16a and the injection pipette 16b, for example, and the three-dimensional positions of the tips 38a and 38b of the holding pipette 16a and the injection pipette 16b are specified in the respective second coordinate systems.
  • the second coordinate system may be a coordinate system common to holding pipette 16a and injection pipette 16b.
  • the control device 30 identifies the three-dimensional position of the manipulator 16 in the first coordinate system by performing coordinate conversion from the second coordinate system to the first coordinate system.
  • the first coordinate system is a coordinate system with the optical axis A of the objective lens 20 as a reference.
  • the position coordinates in the first coordinate system be P 1 (X 1 , Y 1 , Z 1 ) and the position coordinates in the second coordinate system be P 2 (X 2 , Y 2 , Z 2 ).
  • Coordinate conversion to the first coordinate system can be expressed by the following equation (3).
  • r ij is a transformation parameter indicating rotation from the second coordinate system to the first coordinate system
  • t i is the parallelism from the origin of the second coordinate system to the origin of the first coordinate system. It is a conversion parameter that indicates movement.
  • the coordinate transformation parameters r ij and t i can be derived from the correlation between the position coordinates of the tip of the manipulator 16 measured in the second coordinate system and the position coordinates of the tip of the manipulator 16 measured in the first coordinate system. Specifically, the tip of the manipulator 16 is arranged at a plurality of measurement positions of about 100 to 200 different from each other, and the position coordinates of the tip of the manipulator 16 at each measurement position are measured in the first coordinate system and the second coordinate system. do.
  • the coordinate transformation parameters r ij and t i can then be estimated by identifying the correlation of the position coordinates in the first coordinate system and the second coordinate system using the least squares method.
  • the position coordinates in the first coordinate system can be identified from the working distance WD of the objective lens 20 when imaging the tip of the manipulator 16 and the coordinates (u, v) of the pixel at the tip of the manipulator 16 in the captured image.
  • the relationship between the position coordinates P 1 (X 1 , Y 1 , Z 1 ) of the first coordinate system and the coordinates (u, v) of the pixels of the captured image can be expressed by the following equation (4) by perspective projection transformation. can.
  • Equation (4) (c x , c y ) are the center coordinates of the captured image, and s is an arbitrary constant. Simultaneous expression (3) and expression (4) yield the following expression (5).
  • Equation (5) based on the working distance WD of the objective lens 20, the pixel coordinates (u, v) in the captured image, and the position coordinates P 2 (X 2 , Y 2 , Z 2 ) in the second coordinate system can be used to estimate the coordinate transformation parameters r ij and t i .
  • FIG. 5 is a diagram schematically showing the measurement pattern of the manipulator 16 for deriving coordinate transformation parameters.
  • a plurality of measurement planes 60a, 60b, and 60c are set apart from each other by a distance ⁇ in the z direction in a first coordinate system based on the optical axis A of the objective lens 20, and a plurality of measurement positions 62a are set in each of the measurement planes 60a to 60c.
  • 62b and 62c are set in a grid pattern.
  • the number of measurement planes and the number of measurement positions are not limited to this.
  • the number of measurement planes may be four or more, and the number of measurement positions in each measurement plane may be more or less than fifty-four.
  • the manipulator 16 is moved so that the tip 38 of the manipulator 16 is positioned within the field of view of the objective lens 20 .
  • the working distance WD of the objective lens 20 is adjusted to bring the tip 38 of the manipulator 16 into focus.
  • the distance WD to the first measurement surface 60a is determined.
  • the manipulator 16 is moved to move the manipulator 16 at each of the first coordinate system and the second coordinate system at a plurality of measurement positions 62a within the first measurement plane 60a. Identify the position coordinates of the tip 38 .
  • the tip 38 of the manipulator 16 is moved to the second measurement plane 60b.
  • the tip 38 of the manipulator 16 is moved by ⁇ in the z direction, and the working distance is increased by ⁇ to WD+ ⁇ .
  • a distance ⁇ between the first measurement surface 60a and the second measurement surface 60b is set to a value larger than the depth of field of the objective lens 20, for example, about 5 to 10 times the depth of field.
  • the manipulator 16 is moved while the tip 38 of the manipulator 16 is in focus. Thereby, the position coordinates of the tip 38 of the manipulator 16 are specified in each of the first coordinate system and the second coordinate system at the plurality of measurement positions 62b in the second measurement plane 60b.
  • the manipulator 16 is moved by ⁇ in the z direction so that the tip 38 of the manipulator 16 is positioned on the third measurement plane 60c, and the working distance WD is set. is increased by .delta. to WD+2.delta.
  • the position coordinates of the tip 38 of the manipulator 16 are specified in each of the first coordinate system and the second coordinate system at a plurality of measurement positions 62c in the third measurement plane 60c.
  • the control device 30 After specifying the position coordinates at all the measurement positions 62a, 62b, 62c, the control device 30 determines the coordinate transformation parameters r ij and t i using the above equation (5). Controller 30 may separately measure tips 38a, 38b of holding pipette 16a and injection pipette 16b to determine coordinate transformation parameters r ij and t i . Note that once the coordinate transformation parameters r ij and t i are determined, they can be used continuously as long as the device configuration does not change. That is, controller 30 need not determine coordinate transformation parameters r ij and t i each time manipulation system 10 is used. The controller 30 stores previously determined coordinate transformation parameters r ij and t i , and uses the stored coordinate transformation parameters r ij and t i to determine the position of the tip 38 of the manipulator 16 . Coordinates may be transformed.
  • the coordinate transformation parameters r ij and t i accurately transform the relative movement of the manipulator 16 in the second coordinate system into the relative movement in the first coordinate system.
  • the coordinate points of both are shifted. I know it will go away. In other words, the accuracy of the absolute value of the position coordinates becomes low only with the coordinate transformation parameters r ij and t i .
  • the coordinate transformation parameters r ij and t i described above are determined using the property of perspective projection transformation that an object to be imaged appears larger the closer it is to the viewpoint and appears smaller the farther away it is from the viewpoint.
  • the depth of field of the camera is relatively large, so by shifting the position of the object in the depth direction within the depth of field of the camera, the apparent size of the object can be significantly different. can be done.
  • the apparent size of the object hardly changes. .
  • the absolute value of the z-coordinate of the tip 38 of the manipulator 16 is calculated based on the average edge strength F of the captured image of the tip 38 of the manipulator 16, similar to the method of specifying the position coordinate of the sample 40 in the z direction. and calibrate the absolute value (initial value or origin) of the position coordinates in the first coordinate system.
  • the controller 30 first positions the tip 38 of the manipulator 16 at an initial point (temporary origin) O set within the field of view of the objective lens 20 .
  • the control device 30 coordinates the position coordinates (X 20 , Y 20 , Z 20 ) of the initial point O in the second coordinate system to obtain the position coordinates (X 10 , Y 10 , Z 20 ) of the initial point O in the first coordinate system.
  • Z 10 is calculated.
  • the control device 30 images the tip 38 of the manipulator 16 located at the initial point O while changing the working distance WD to generate a plurality of captured images.
  • the width of change in the working distance WD be as small as possible, preferably less than the depth of field of the objective lens 20 (for example, about 1 ⁇ m).
  • the control device 30 calculates the average edge strength F of the tip 38 of the manipulator 16 included in each of the plurality of captured images, and specifies the captured image with the maximum average edge strength F.
  • the control device 30 sets the true value of the z-coordinate of the initial point O to the working distance WD max when the captured image with the maximum average edge strength F is captured.
  • Z 1 ′ Z 1 ⁇ Z 10 +WD max .
  • the controller 30 determines in real time the three-dimensional position of the sample 40 within the field of view of the objective lens 20 in the first coordinate system, and the three-dimensional position of the tip 38 of the manipulator 16 in the first coordinate system. Identify the original position in real time.
  • the three-dimensional position of the sample 40 can be identified by acquiring at least three captured images while changing the working distance WD.
  • the time to specify the three-dimensional position of the sample 40 is mainly constrained by the time required to drive the variable focus lens 22 to change the working distance WD.
  • the time required to change the working distance WD depends on the specifications of the variable focus lens 22 and the amount of change in the working distance WD, but for example, it is about 10 to 20 milliseconds.
  • the time required to acquire three captured images by changing the working distance WD is about 30 to 60 milliseconds, and the three-dimensional position of the sample 40 is measured at a cycle of 16 to 33 times per second. can be captured.
  • the three-dimensional position of the tip 38 of the manipulator 16 can be captured sequentially based on the motion of the manipulator drive mechanism 26 and can be identified within the time period for identifying the three-dimensional position of the sample 40 .
  • the control device 30 Based on the identified three-dimensional positions of the sample 40 and the manipulator 16, the control device 30 maps the three-dimensional positional relationship between the sample 40 and the manipulator 16 in the virtual space.
  • the range of the mapped virtual space is wider than the range of the field of view of the objective lens 20 .
  • the three-dimensional position of the sample 40 can only be determined within the field of view of the objective lens 20, while the three-dimensional position of the manipulator 16 can be determined both inside and outside the field of view of the objective lens 20.
  • FIG. Therefore, in virtual space, even if the tip 38 of the manipulator 16 is not located within the field of view of the objective lens 20, the position of the tip 38 of the manipulator 16 can always be mapped.
  • a three-dimensional arrangement relationship between the sample 40 and the manipulator 16 mapped in the virtual space is displayed on the three-dimensional display device 32a in real time.
  • the display cycle of the three-dimensional display device 32a is, for example, 40 milliseconds (25 frames per second).
  • 6(a) to (j) are diagrams showing an example of a captured image within the field of view of the objective lens 20 and a display example of the three-dimensional display device 32a.
  • the flow of manipulating the sample 40 with the pipette 16b is shown in chronological order.
  • 6A to 6E are captured images (that is, microscope images) within the field of view of the objective lens 20, and FIGS. 6F to 6J are images of FIGS.
  • a first object 70 imitating the sample 40, a second object 72 imitating the holding pipette 16a, and a third object 74 imitating the injection pipette 16b are stereoscopically displayed on the three-dimensional display device 32a.
  • FIGS. 6(f) and 6(g) show the state before the injection pipette 16b is brought close to the sample 40, and the injection pipette 16b is not shown in the captured image.
  • the three-dimensional position of the injection pipette 16b can be captured even outside the field of view of the objective lens 20.
  • FIG. Therefore, a third object 74 imitating the injection pipette 16b is also displayed on the three-dimensional display device 32a of FIGS. 6(f) and 6(g).
  • the injection pipette 16b can be operated so that the tip of the injection pipette 16b approaches the sample 40 while looking at the three-dimensional display device 32a.
  • the three-dimensional display device 32a can arbitrarily switch the viewpoint.
  • the display can be switched to either side view when viewed in any direction.
  • FIG. 6F is a top view display example when the sample 40 is viewed in the z direction
  • FIG. 6G is a side view display example when the sample 40 is viewed in the y direction.
  • FIGS. 6(c) and (d) show a state in which the injection pipette 16b is brought into contact with the sample 40 and operated. Looking at the captured image in FIG. However, looking at the three-dimensional display device 32a in FIG. 6(h), it can be seen that the injection pipette 16b is hidden under the sample 40 and the injection pipette 16b is not stuck in the sample 40. FIG. In addition, it can be seen that the sample 40 is lifted upward by the injection pipette 16b, and the sample 40 is about to be released from the holding pipette 16a.
  • FIG. 6(e) shows a state where the injection pipette 16b is stuck in the sample 40 and the sample 40 is separated from the holding pipette 16a.
  • FIG. 6(i) shows a state where the injection pipette 16b is stuck in the sample 40 and the sample 40 is separated from the holding pipette 16a.
  • FIG. 7 is a flow chart showing a three-dimensional position presentation method according to the embodiment.
  • the origin of the manipulator 16 is calibrated (S10). Calibration of the origin of the manipulator 16 is performed by imaging the tip 38 of the manipulator 16 at different working distances and specifying the working distance WD max at which the average edge strength F of the captured image is maximized. Subsequently, a captured image of the observation area is acquired (S12), and if the sample 40 is present in the captured image (Y of S14), a plurality of captured images are acquired by changing the working distance WD (S16).
  • the three-dimensional position of the sample 40 is identified based on the maximum value when the average edge strength F of the sample 40 included in the plurality of captured images is approximated by Gaussian distribution (S18). Subsequently, the three-dimensional position of the manipulator 16 in the second coordinate system is identified based on the movement of the manipulator 16, and the three-dimensional position of the manipulator 16 in the first coordinate system is determined by coordinate conversion from the second coordinate system to the first coordinate system. is specified (S20). Based on the identified three-dimensional positions of the sample 40 and the manipulator 16 in the first coordinate system, the sample 40 and the manipulator 16 are mapped in the virtual space (S22), and the three-dimensional position of the sample 40 and the manipulator 16 mapped in the virtual space is determined.
  • the layout relationship is displayed on the three-dimensional display device 32a or the like (S24). If there is no sample 40 in S14 (N of S14), the processing of S16 to S24 is skipped. When continuing to use the manipulation system 10 (N of S26), the processing of S12 to S24 is repeated. When the use of the manipulation system 10 ends (Y of S26), this flow ends.
  • the processing of S12 to S24 is repeated every 40 milliseconds, and the three-dimensional positions of the sample 40 and the manipulator 16 are updated and displayed at a cycle of 25 frames per second. Since this display cycle is substantially the same as the frame rate of a general moving image, the user viewing the three-dimensional display device 32a will not be able to tell whether the three-dimensional positions of the sample 40 and the manipulator 16 are immediately reflected without time lag. looks like. As a result, the sample 40 can be accurately manipulated by the manipulator 16 while looking at the three-dimensional display device 32a without feeling any stress caused by the shift between the actual position and the displayed position.
  • the three-dimensional display device 32a presents the positional relationship between the sample 40 and the manipulator 16 in a stereoscopically viewable state. You can check the arrangement relationship of In other words, the magnified sample 40 and the manipulator 16 can be displayed as if they existed three-dimensionally in front of one's eyes. As a result, the operability when precisely manipulating a minute sample 40 such as a cell can be improved, and the convenience of the manipulation system 10 can be improved.
  • FIGS. 8A to 8C are diagrams schematically showing how the sample 40 is manipulated by the manipulator 16.
  • FIG. FIG. 8(a) shows a holding operation of the sample 40
  • FIG. 8(b) shows a rotating operation of the sample 40
  • FIG. 8A to 8C correspond to captured images captured by the imaging device 24 and show the state of the sample 40 viewed in the z direction.
  • the projecting direction of the tip 38a of the holding pipette 16a is the +x direction.
  • FIG. 8(a) shows the operation of holding the sample 40 using the holding pipette 16a.
  • the sample 40 is fixed by bringing the tip 38a of the holding pipette 16a close to the sample 40 and sucking the sample 40 with the tip 38a of the holding pipette 16a.
  • the sample 40 is in contact with the tip 38a of the holding pipette 16a in the x direction, and a holding force Fa in the x direction ( ⁇ x direction) due to suction is applied to the sample 40.
  • sample 40 is a mature human or animal egg cell that has a central cytoplasm 40a, a zona pellucida 40b surrounding the cytoplasm 40a, and a polar body 40c partially between the cytoplasm 40a and the zona pellucida 40b. have.
  • FIG. 8(b) shows the operation of rotating the sample 40 using the injection pipette 16b.
  • the position of the polar body 40c is adjusted.
  • the polar body 40c is positioned in the y direction from the center 40d of the sample 40. The orientation of the sample 40 is adjusted to .
  • the injection pipette 16b is moved in a direction (eg, y direction) orthogonal to the projection direction of the tip 38b of the injection pipette 16b, not in the projection direction (eg, x direction).
  • a direction eg, y direction
  • the position and orientation of the sample 40 are shifted.
  • a force Fb in the +y direction is applied to the sample 40, and the center position of the sample 40 shifts in the +y direction from 40d0 to 40d1.
  • reaction force Fc caused by the holding force Fa is applied to the injection pipette 16b.
  • the direction of this reaction force Fc corresponds to the opposite direction of the displacement d from the center position 40d0 of the sample 40 before operation toward the center position 40d1 of the sample 40 after operation.
  • the magnitude of the reaction force Fc correlates with the magnitude of the displacement d (that is, displacement amount) of the sample 40 due to manipulation by the injection pipette 16b.
  • the reaction force Fc can be represented by the following formula (6), for example.
  • the function f is a function that indicates the correlation between the magnitude of the reaction force Fc and the magnitude of the displacement d of the sample 40 .
  • the function f may have the magnitude of the holding force Fa as a variable, and may be expressed as f(d, Fa). If it is assumed that the magnitude of the reaction force Fc and the magnitude of the displacement d of the sample 40 are proportional, Equation (6) can be expressed by Equation (7) below using a proportionality constant k.
  • the displacement amount d in the above formulas (6) and (7) can be specified in real time based on the change in the position of the center 40d of the sample 40.
  • the control device 30 identifies the displacement amount d at each moment by identifying the position of the center 40d of the sample 40 using the above-described three-dimensional position identification method based on the captured image.
  • the controller 30 estimates the force Fc applied between the sample 40 and the injection pipette 16b at each moment based on the identified displacement amount d.
  • FIG. 8(c) shows an operation of piercing the sample 40 with the injection pipette 16b.
  • a force Fd in the ⁇ x direction is applied to the sample 40 by pressing the tip 38b of the injection pipette 16b toward the sample 40 in the ⁇ x direction.
  • the sample 40 is sandwiched between the holding pipette 16a and the injection pipette 16b and is deformed so that the portion in contact with the tip 38b of the injection pipette 16b becomes concave in the -x direction.
  • a reaction force Fe is applied to the injection pipette 16b due to the elastic force that causes the sample 40 to return to its original shape.
  • the direction of this reaction force Fe is the opposite direction (+x direction) to the projecting direction ( ⁇ x direction) of the tip 38b of the injection pipette 16b.
  • the magnitude of the reaction force Fe correlates with the magnitude of deformation wd of the sample 40, and can be expressed by the following equation (8), for example.
  • E is the elastic modulus (Young's modulus) of the sample 40
  • v is the Poisson's ratio of the sample 40
  • h is the thickness of the clear zone.
  • ⁇ c/a, where a is the radius of the tip 38a of the holding pipette 16a and c is the radius of the tip 38b of the injection pipette 16b.
  • a vector e is the projecting direction of the injection pipette 16b.
  • the details of formula (8) are described in "Y. Sun, K. T. Wan, K. P. Roberts, J. C. Bischof and B. J. Nelson: “Mechanical property characterization of mouse zona pellucida”, IEEE Trans Nanobioscience, vol. 2, no. 4, pp. 279.286, 2003.”
  • the amount of deformation wd in the above formula (8) corresponds to the amount of depression of the sample 40 in the projecting direction (x direction) of the tip 38b of the injection pipette 16b. It corresponds to the distance in the projecting direction (x direction) to the tip 38b of 16b.
  • the deformation amount wd can be measured, for example, based on the captured image of the sample 40 .
  • the positions of the center 40 d and the edge 40 e of the sample 40 can be specified based on the captured image of the sample 40 .
  • the position of the tip 38b of the injection pipette 16b can be specified by a method using coordinate transformation in the three-dimensional position specifying method described above.
  • the control device 30 identifies the position of the center 40d of the sample 40 in real time using the above-described three-dimensional position identification method based on the captured image.
  • the control device 30 specifies the positions of the center 40d and the end 40e of the sample 40 in real time based on the captured image in which the sample 40 is most focused, and specifies the distance w1.
  • the control device 30 identifies the position of the tip 38b of the injection pipette 16b in real time by coordinate-transforming the three-dimensional position with respect to the drive shaft of the injection pipette 16b. Thereby, the control device 30 specifies the deformation amount wd at each instant.
  • the control device 30 estimates the force Fe applied between the sample 40 and the injection pipette 16b at each instant based on the identified deformation amount wd.
  • the control device 30 transmits force information based on the estimated forces Fc and Fe to the force sense presentation device 36 so that the force sense corresponding to the estimated forces Fc and Fe is presented to the user through the force sense presentation device 36. do. Since the magnitudes of the estimated forces Fc and Fe are as small as about 1 ⁇ N to 1 mN, if the magnitudes of the estimated forces Fc and Fe are transmitted as they are to the haptic presentation device 36, the user may is difficult to perceive. Therefore, in the present disclosure, the estimated force is amplified and presented to the user.
  • the control device 30 generates force information indicating the forces ⁇ Fc and ⁇ Fe obtained by multiplying the estimated forces Fc and Fe by a predetermined amplification factor ⁇ , and transmits the force information to the force sense presentation device 36 .
  • the amplification factor ⁇ is 100 times or more and 10000 times or less, for example, 1000 times or more and 5000 times or less.
  • the force sense presentation device 36 presents the amplified forces ⁇ Fc and ⁇ Fe by operating the actuators based on the force information acquired from the control device 30 . For example, when the amplification factor ⁇ is 1000 times, a force sensation of about 1 mN to 1 N is presented, so that the user can easily perceive the reaction force accompanying the manipulation of the manipulator 16 .
  • the amplification factor ⁇ may be a parameter that can be appropriately adjusted by the user.
  • FIG. 9 is a flow chart showing the haptic presentation method according to the embodiment.
  • the control device 30 acquires position information based on the user's input operation for designating the position of the manipulator 16 from the force sense presentation device 36 (S30). For example, position information corresponding to the position of the tip 36b of the articulated arm 36a is transmitted to the control device 30 by the user holding and moving the tip 36b of the articulated arm 36a. Subsequently, the control device 30 controls the operation of the manipulator driving mechanism 26 based on the acquired positional information to control the position of the manipulator 16 (S32).
  • the imaging device 24 images the sample 40 manipulated using the manipulator 16 through the objective lens 20, and the control device 30 acquires the image captured by the imaging device 24 (S34).
  • the control device 30 specifies the amount of change in the position or shape of the sample 40 based on the acquired image (S36), and adjusts the force applied between the sample 40 and the manipulator 16 based on the specified amount of change.
  • Estimate S38
  • the control device 30 specifies the amount of displacement d of the central position of the sample 40, and based on the specified amount of displacement d, the injection pipette 16b is subjected to Estimate the direction and magnitude of the reaction force Fc.
  • the control device 30 specifies the deformation amount wd of the sample 40, and based on the specified deformation amount wd, the direction of the reaction force Fe applied to the injection pipette 16b. and estimate the magnitude.
  • the control device 30 transmits force information corresponding to the estimated forces Fc and Fe to the force sense presentation device 36 so that the force sense is presented to the user through the force sense presentation device 36 (S40).
  • the control device 30 transmits force information indicating the forces ⁇ Fc and ⁇ Fe obtained by amplifying the estimated forces Fc and Fe with a predetermined amplification factor ⁇ to the force sense presentation device 36 so that the amplified forces are presented to the user.
  • the user can operate the manipulator 16 using the force sense presentation device 36 while feeling the reaction force applied to the manipulator 16 with the force sense presentation device 36 . Therefore, it is possible to operate based on both the sense of sight and touch, and the sample 40 can be operated while grasping what kind of force is applied to the sample 40 according to the positional relationship between the sample 40 and the manipulator 16 . As a result, it is possible to help master the skill for appropriately controlling the force applied to the cell.
  • the force information indicating the magnitude of the force sensation to be presented to the user is generated to generate force information between the sample 40 and the manipulator 16. It is possible to present a haptic sensation according to the force applied to the It is difficult to detect the force applied to the sample 40 by the manipulator 16 in real time using a sensor or the like. can improve.
  • a first aspect of the present disclosure includes a manipulator for manipulating a sample, a manipulator driving mechanism for moving the manipulator, an imaging device for capturing an image of the sample through an objective lens, and an image captured by the imaging device.
  • a controller for specifying a change amount of at least one of the position and shape of the sample and estimating a force applied between the sample and the manipulator based on the specified change amount; and specifying the position of the manipulator.
  • a force sense presentation device configured to receive an input operation from a user for and present to the user a force sense corresponding to the force estimated by the control device.
  • a second aspect of the present disclosure is that the control device specifies the amount of deformation of the sample in the projecting direction of the tip of the manipulator based on the image, and estimates the force based on the amount of deformation.
  • a manipulation system according to the first aspect characterized.
  • a third aspect of the present disclosure is that the control device specifies the amount of deformation based on a tip position of the manipulator, a center position of the sample, and an end position of the sample in the projecting direction.
  • a manipulation system according to the second aspect characterized in that:
  • a fourth aspect of the present disclosure is characterized in that the control device specifies the amount of displacement of the sample based on the image, and estimates the force based on the amount of displacement.
  • a manipulation system according to any one of the aspects of .
  • control device transmits to the haptic presentation device force information indicating a force obtained by amplifying the magnitude of the estimated force by 100 times or more, and the haptic presentation device: A manipulation system according to any one of the first to fourth aspects, wherein the amplified force is presented to the user.
  • a sixth aspect of the present disclosure is a step of acquiring position information based on a user's input operation for designating a position of a manipulator from a haptic presentation device, and moving the manipulator based on the acquired position information.
  • controlling the operation of a manipulator drive mechanism imaging a sample manipulated using the manipulator through an objective lens; and determining the amount of change in at least one of the position and shape of the sample based on the captured image.
  • a seventh aspect of the present disclosure includes a function of acquiring position information based on a user's input operation for designating a position of a manipulator from a haptic presentation device, and moving the manipulator based on the acquired position information.
  • a function of controlling the operation of a manipulator driving mechanism a function of acquiring an image of a sample manipulated using the manipulator through an objective lens; and at least one of position and shape of the sample based on the acquired image.
  • a program characterized by causing a computer to realize a function of controlling the operation of the haptic device.
  • FIG. 10 is a diagram schematically showing the configuration of a manipulation system 110 according to the second embodiment.
  • a second haptic presentation device for operating the holding pipette 16a is provided separately from the haptic presentation device 36 for operating the injection pipette 16b (also referred to as a first haptic presentation device 36). 80 is used, which is different from the above-described first embodiment.
  • the second embodiment will be described with a focus on the points of difference from the first embodiment, and the description of the common points will be omitted as appropriate.
  • the manipulation system 110 includes a stage 12, an illumination device 14, a manipulator 16, a folding mirror 18, an objective lens 20, a variable focus lens 22, an imaging device 24, a manipulator drive mechanism 26, and a lens drive mechanism 28. , a control device 30 , a display device 32 , an input device 34 , a first haptic device 36 , a pump 78 , and a second haptic device 80 .
  • the first haptic presentation device 36 is configured in the same manner as in the first embodiment described above.
  • the first haptic presentation device 36 is an input operation device for operating the injection pipette 16b.
  • the articulated arm 36a of the first haptic device 36 is a first link mechanism for supporting the tip 36b of the articulated arm 36a, for example, a serial link mechanism.
  • a tip 36b of the articulated arm 36a is a first holding member held by the user. Therefore, the first force sense presentation device 36 includes a first holding member held by the user and a first link mechanism that supports the first holding member so that the position of the first holding member is variable according to the user's operation.
  • the first force sense presentation device 36 further has an actuator 36c for driving the first link mechanism (multi-joint arm 36a) and applying a reaction force to the first holding member (tip 36b).
  • the first force sense presentation device 36 further has a position sensor 36d for detecting the orientation of the articulated arm 36a and detecting the position of the first holding member (tip 36b).
  • the first haptic presentation device 36 is operated by the user's right hand, for example.
  • the pump 78 is connected to the inside of the holding pipette 16a and generates a suction force or an ejection force at the tip 38a of the holding pipette 16a.
  • Pump 78 is, for example, an electric microinjector.
  • the pump 78 creates a suction force at the tip 38a of the holding pipette 16a by creating a negative pressure inside the holding pipette 16a.
  • the pump 78 generates a discharge force at the tip 38a of the holding pipette 16a by creating a positive pressure inside the holding pipette 16a.
  • the pump 78 variably controls the suction force and the discharge force based on commands sent from the control device 30 .
  • the second haptic presentation device 80 is an input operation device for operating the manipulator 16 and an input operation device for operating the holding pipette 16a.
  • the second haptic presentation device 80 is configured to receive an input operation for designating the three-dimensional position of the holding pipette 16a and an input operation for designating the suction force and ejection force at the tip of the holding pipette 16a.
  • the second haptic device 80 includes a second link mechanism 82 , a second holding member 84 and a rotating member 86 .
  • the second force sense presentation device 80 is operated by the user's left hand, for example.
  • the second link mechanism 82 supports the second holding member 84 so that the position of the second holding member 84 is variable according to the user's operation.
  • the second link mechanism 82 is configured such that the second holding member 84 attached to the tip of the second link mechanism 82 can move along the XYZ three axes.
  • the second link mechanism 82 is, for example, a parallel link mechanism in which three arms are connected in parallel.
  • the second link mechanism 82 includes, for example, an actuator 82a for driving each of the three arms to apply a reaction force to the second holding member 84, and a position sensor for detecting the orientation of each of the three arms.
  • a sensor 82b is provided.
  • the second holding member 84 is a member held by the user.
  • the second holding member 84 is held, for example, by the user's thumb and middle finger during use.
  • the user By holding the second holding member 84 and moving the second holding member 84, the user performs an input operation for designating the three-dimensional position of the holding pipette 16a.
  • the three-dimensional position of the second holding member 84 is detected by a position sensor 82b provided on the second link mechanism 82, for example.
  • the rotating member 86 is a member for receiving an input operation by a user's gripping motion.
  • the rotating member 86 is provided near the second holding member 84 and configured to be rotatable with respect to the second holding member 84 .
  • the user By rotating the rotating member 86, the user performs an input operation for designating the suction force and ejection force of the tip 38a of the holding pipette 16a.
  • the second haptic presentation device 80 for example, Omega 7 manufactured by Force Dimension can be used, and for example, the active gripper and haptic device disclosed in International Publication No. 2008/003416 can be used.
  • FIG. 11(a) and 11(b) are diagrams schematically showing the configuration of the second force sense presentation device 80.
  • FIG. 11(a) schematically shows the configuration of the second holding member 84 and the rotating member 86
  • FIG. 11(b) schematically shows an input operation by a gripping motion of the user holding the second holding member 84. shown.
  • the second holding member 84 has a shape that can be easily held by the user's fingers, for example, a cylindrical shape. Attached to the second holding member 84 is a connecting bar 84 a extending radially from the second holding member 84 . A rotating shaft 88 that rotatably supports a rotating member 86 is provided at the tip of the connection bar 84a.
  • the rotating member 86 has a linear member 86a, an arc member 86b, and a pad 86c.
  • the linear member 86a is connected to a rotating shaft 88 and is rotatable about the rotating shaft 88.
  • the circular arc member 86b extends from the end of the linear member 86a toward the pad 86c along an arc around the pivot shaft 88.
  • a pad 86c is provided at the end of the arc member 86b and contacts the user's index finger during use.
  • the second holding member 84 is provided with a pulley 90a for driving the rotating member 86, an actuator 90b for driving the pulley 90a, and an angle sensor 90c for detecting the rotation angle of the pulley 90.
  • the pulley 90a is configured to mesh with the outer circumference of the arc member 86b.
  • the pulley 90a rotates according to the amount of movement in the circumferential direction when the arc member 86b moves in the circumferential direction due to the rotation of the rotating member 86.
  • the actuator 90b gives a reaction force to the rotating member 86 and presents a force sensation to the user who operates the rotating member 86.
  • the angle sensor 90c detects the rotation angle ⁇ of the rotation member 86 with respect to the second holding member 84 by detecting the rotation angle of the pulley 90a.
  • the control device 30 acquires position information indicating the three-dimensional position of the second holding member 84 from the second haptic presentation device 80 .
  • the control device 30 controls the operation of the first drive mechanism 26a that moves the holding pipette 16a, based on the positional information acquired from the second haptic device 80.
  • the control device 30 acquires angle information indicating the rotation angle ⁇ of the rotation member 86 from the second haptic presentation device 80 .
  • the control device 30 controls the operation of the pump 78 that variably controls the suction force and discharge force at the tip 38a of the holding pipette 16a based on the angle information acquired from the second haptic device 80.
  • the angle information indicating the rotation angle ⁇ of the rotating member 86 is an example of operation information indicating an operation amount specifying the suction force and the discharge force of the pump 78 .
  • the control device 30 controls the tip 38a of the manipulator 16 so that neither the suction force nor the ejection force is generated. to That is, when the rotation angle ⁇ of the rotation member 86 is the predetermined initial value ⁇ 0 , the operation of the pump 78 is stopped.
  • the control device 30 operates the pump 78 so that a suction force is generated at the tip 38a of the manipulator 16. make it work.
  • the control device 30 may operate the pump 78 so that the suction force changes according to the rotation angle ⁇ of the rotating member 86, and the suction force increases as the rotation angle ⁇ of the rotating member 86 increases. You can make it bigger.
  • the control device 30 operates the pump 78 to generate a discharge force at the tip 38a of the manipulator 16. make it work.
  • the control device 30 may operate the pump 78 so that the discharge force changes according to the rotation angle ⁇ of the rotating member 86, and as the rotation angle ⁇ of the rotating member 86 decreases, the discharge force increases. You can make it bigger.
  • the control device 30 generates force information indicating the magnitude and direction of the force sense to be presented to the user through the second force sense presentation device 80 .
  • the control device 30 generates force information so that force feedback corresponding to the suction force or ejection force at the tip 38a of the holding pipette 16a is fed back.
  • the control device 30 sets the force to zero. to generate force information. Therefore, when neither the suction force nor the ejection force is generated, the haptic sensation is not presented to the user. Thus, the user can be notified through the sense of force whether or not the rotation angle ⁇ of the rotation member 86 has reached the predetermined initial value ⁇ 0 .
  • the controller 30 adjusts the magnitude of the force corresponding to the attractive force. Generates force information that indicates the strength.
  • the control device 30 generates force information in a direction opposite to the rotating direction of the rotating member 86 .
  • the controller 30 adjusts the force corresponding to the ejection force. Generates force information that indicates the strength.
  • the control device 30 generates force information in a direction opposite to the rotating direction of the rotating member 86 .
  • the control device 30 may generate force information in which the magnitude of the reaction force varies depending on whether the sample 40 is in contact with the tip 38a of the holding pipette 16a.
  • the reaction force when the sample 40 is in contact with the tip 38a of the holding pipette 16a is defined as the reaction force when the sample 40 is not in contact with the tip 38a of the holding pipette 16a. It may be larger than the force.
  • the reaction force increases at the moment when the sample 40 contacts the tip 38a of the holding pipette 16a, the contact of the sample 40 can be conveyed to the user through the sense of force.
  • the reaction force decreases at the moment when the sample 40 is separated from the tip 38a of the holding pipette 16a, it is possible to notify the user of the separation of the sample 40 through the sense of force.
  • FIG. 12 is a diagram schematically showing a method of determining contact between the sample 40 and the manipulator 16.
  • the control device 30 determines whether the sample 40 is in contact with the tip 38a of the holding pipette 16a.
  • the control device 30 makes contact determination based on the position of the sample 40 with respect to the position of the tip 38a of the holding pipette 16a.
  • the position of the sample 40 can be specified based on the captured image of the sample 40 .
  • the position of the tip 38a of the holding pipette 16a can be specified by the method using coordinate transformation described in the three-dimensional position specifying method according to the first embodiment.
  • the control device 30 determines contact depending on whether the end 40f of the sample 40 is included within the range of the vicinity area 92 of the tip 38a of the holding pipette 16a.
  • a neighboring region 92 indicated by a dashed line in FIG. 12 has, for example, a width wb in the projection direction (+x direction) of the holding pipette 16a and a width wb in the projection direction ( ⁇ x direction) with respect to the tip 38a of the holding pipette 16a.
  • the width wa and the width wc in directions perpendicular to the projecting direction ( ⁇ y directions) are set.
  • the control device 30 identifies the position coordinates (Cx, Cy) of the center 40d of the sample 40 and the radius rs of the sample 40 based on the captured image.
  • the control device 30 specifies the position coordinates of the tip 38a of the holding pipette 16a in the second coordinate system from the operation of the manipulator drive mechanism 26, and converts the coordinates from the second coordinate system to the first coordinate system to obtain the position coordinates of the tip 38a. Identify (Mx, My).
  • the control device 30 calculates the position coordinates (Cx ⁇ rs, Cy) of the end portion 40 f of the sample 40 from the position coordinates (Cx, Cy) of the center 40 d of the sample 40 and the radius rs of the sample 40 .
  • the control device 30 calculates the range of the neighboring region 92 from the position coordinates (Mx, My) of the tip 38a of the holding pipette 16a. The control device 30 determines that the sample 40 is in contact with the tip 38a of the holding pipette 16a when the end portion 40f of the sample 40 is within the vicinity region 92.
  • FIG. Specifically, when Mx-wa ⁇ Cx-rs ⁇ Mx+wb and My-wc ⁇ Cy ⁇ My+wc, it is determined that the sample 40 is in contact with the tip 38a of the holding pipette 16a.
  • the control device 30 determines that the sample 40 is not in contact with the tip 38a of the holding pipette 16a when the end portion 40f of the sample 40 is outside the range of the neighboring region 92 .
  • the value k 1 ⁇ multiplied by 1 coefficient k 1 is assumed to be the magnitude of the reaction force.
  • a value k 2 ⁇ obtained by multiplying 2 by a coefficient k 2 is defined as the magnitude of the reaction force.
  • the second coefficient k 2 is greater than the first coefficient k 1 (that is, k 2 >k 1 ).
  • the ratio k 2 /k 1 between the first coefficient k 1 and the second coefficient k 2 is, for example, 1.1 or more and 5 or less, preferably 1.5 or more and 3 or less, and is 2 as an example.
  • the first coefficient k1 and the second coefficient k2 can be set so that the maximum value of the reaction force applied to the rotating member 86 is 0.5 N or more and 5 N or less, preferably 1 N or more and 2 or less. .
  • the magnitude of the reaction force may be changed continuously. For example, if the suction force or the ejection force is constant, the magnitude of the reaction force may decrease as the distance ds from the tip 38a of the holding pipette 16a to the sample 40 increases.
  • the magnitude of the reaction force is f (ds) may be ⁇ .
  • FIG. 13-15 show a working area 94 for manipulating the sample 40.
  • the working area 94 is roughly divided into three areas 94a, 94b, 94c.
  • the first area 94a is an area in which the unprocessed sample 40 before cell manipulation is placed.
  • the second area 94b is an area where cell manipulation is performed on the sample 40 using the injection pipette 16b.
  • the third area 94c is an area in which the processed sample 40 after cell manipulation is placed.
  • the holding pipette 16a is moved to the first region 94a, and the tip 38a of the holding pipette 16a is brought close to the sample 40 in the first region 94a.
  • the user can move the tip 38a of the holding pipette 16a to the first region 94a by holding and moving the second holding member 84 of the second haptic device 80. As shown in FIG. 13, the holding pipette 16a is moved to the first region 94a, and the tip 38a of the holding pipette 16a is brought close to the sample 40 in the first region 94a.
  • the user can move the tip 38a of the holding pipette 16a to the first region 94a by holding and moving the second holding member 84 of the second haptic device 80. As shown in FIG.
  • a suction force is generated at the tip 38a of the holding pipette 16a to suck and hold the sample 40 at the tip 38a of the holding pipette 16a.
  • the user causes the tip 38a of the holding pipette 16a to generate a suction force by gripping the rotating member 86 so as to increase the rotation angle ⁇ .
  • the user can grasp the magnitude of the suction force from the reaction force applied to the rotating member 86 . Further, when the sample 40 comes into contact with the tip 38a of the holding pipette 16a and is held by the suction force, the reaction force applied to the rotating member 86 increases.
  • the user can confirm whether or not the tip 38a of the holding pipette 16a can hold the sample 40 by the change in the reaction force applied to the rotating member 86.
  • FIG. Due to the reaction force applied to the rotating member 86, the user can obtain a sense of force as if the sample 40 were directly grasped and held by the user's hand.
  • the holding pipette 16a is moved from the first area 94a to the second area 94b.
  • the user can move the sample 40 held at the tip 38a of the holding pipette 16a to the second area 94b by moving the second holding member 84 while maintaining the gripping action of gripping the rotating member 86.
  • FIG. It should be noted that the holding of the sample 40 cannot be maintained when the holding pipette 16a is moved, and the sample 40 may come off the tip 38a. In this case, it is determined that the sample 40 is not in contact with the sample 40 and the reaction force applied to the rotating member 86 is reduced. is removed.
  • cell manipulation is performed using the injection pipette 16b on the sample 40 held at the tip 38a of the holding pipette 16a.
  • the user can operate the holding pipette 16a with the left hand through the second force presentation device 80 while operating the injection pipette 16b with the right hand through the first force presentation device .
  • the user performs cell manipulation using the injection pipette 16b as shown in FIGS. 8(b) and (c).
  • the injection pipette 16b can be moved with the right hand while adjusting the suction force by changing the rotation angle ⁇ of the rotating member 86 with the left hand.
  • the perforating operation of FIG. 8(c) not only can the injection pipette 16b be moved, but also the operation of moving the holding pipette 16a to adjust the position of the sample 40 and the suction force of the sample 40 can be performed at the same time. .
  • the user can confirm whether the sample 40 can be held by the holding pipette 16a by the change in the reaction force applied to the rotating member 86 through the sense of force.
  • the holding pipette 16a is moved from the second area 94b to the third area 94c while holding the sample 40 for which cell manipulation has been completed.
  • the user can move the sample 40 held at the tip 38a of the holding pipette 16a to the third region 94c by moving the second holding member 84 while maintaining the gripping action of gripping the rotating member 86.
  • an ejection force is generated at the tip 38a of the holding pipette 16a.
  • the user can generate a discharge force at the tip 38a of the holding pipette 16a by performing a grasping operation with the index finger so that the rotation angle ⁇ of the rotating member 86 becomes small.
  • the user can confirm the magnitude of the ejection force by the reaction force applied to the rotating member 86 .
  • the sample 40 is removed from the tip 38a of the holding pipette 16a by the ejection force, and the holding is released. When the sample 40 is separated from the tip 38a of the holding pipette 16a and is out of contact with the tip 38a of the holding pipette 16a, the reaction force applied to the rotating member 86 is reduced. It can be confirmed that the sample 40 is removed from the .
  • the holding pipette 16a can be moved from the third area 94c to the first area 94a.
  • the manipulations of FIGS. 13 to 15 are repeatedly performed.
  • the operation of moving the holding pipette 16a and the operation of aspirating and ejecting the sample 40 with the holding pipette 16a can be realized with a single input operation device.
  • a joystick is used to move the holding pipette 16a as in the first embodiment, it is necessary to operate suction and discharge using an operation means other than the joystick. It takes time to switch. If the sample 40 comes off when the holding pipette 16a is moved while holding the sample 40, the operating means must be switched in order to hold the sample 40 again, which complicates the operation.
  • the suction force and ejection force at the tip 38a of the holding pipette 16a can be presented to the user as haptics. Since the user can adjust the magnitude of the suction force or the ejection force while feeling the force according to the magnitude of the suction force or the ejection force, the operability of suction and ejection can be improved.
  • the present disclosure by determining the contact of the sample 40 with the tip 38a of the holding pipette 16a, it is possible to present to the user as a force sense whether the sample 40 is held at the tip 38a of the holding pipette 16a. As a result, it becomes easier to confirm whether or not the sample 40 is held in the holding pipette 16a compared to the case of only visual observation, and the operability when holding the sample 40 or releasing the holding of the sample 40 is improved. can improve.
  • a gripping motion in a direction of gripping the rotating member 86 generates a suction force, and a reaction force corresponding to the suction force is applied to the rotating member 86 to hold the sample 40 . It is possible to present a force sensation as if you were directly grasping the In addition, by increasing the reaction force when the sample 40 is held by contact with the tip 38a of the holding pipette 16a, it is possible to present a force that mimics the moment when the sample 40 is directly touched by hand. Since the reaction force is reduced when the sample 40 is removed from the tip 38a of the holding pipette 16a, it is possible to present a force sensation simulating the moment the sample 40 is released. As a result, it is possible to present a haptic sensation with less sense of incongruity to the user, and to more effectively improve the operability of suction and ejection.
  • Example 1 the second force sense presentation device 80 was used as an input operation device, and the force sense through the rotating member 86 was not presented.
  • Example 2 the second force sense presentation device 80 is used as an input operation device, and a force sense is presented through the rotating member 86 .
  • Example 1 and Example 2 For each of Comparative Example, Example 1 and Example 2, the average value of the time required by 6 adults with no experience of cell manipulation was calculated. It was 116 seconds in the comparative example, 51 seconds in Example 1, and 42 seconds in Example 2. Welch's T-test p-values between Comparative Example and Example 1 and between Example 1 and Example 2 were both less than 0.01. From this, it was found that the operability can be significantly improved by using the second force sense presentation device 80 as an input operation device instead of the joystick. Further, it was found that, when the second force sense presentation device 80 is used as an input operation device, the operability can be significantly improved by giving a reaction force to the rotating member 86 to present a force sense.
  • FIG. 16 is a flow chart showing an example of a manipulation method according to the present disclosure.
  • the control device 30 acquires position information based on the user's input operation for designating the position of the manipulator 16 from at least one of the first haptic presentation device 36 and the second haptic presentation device 80 (S50).
  • the control device 30 controls the operation of the manipulator drive mechanism 26 based on the acquired position information to move the manipulator 16 (S52).
  • the imaging device 24 images the sample 40 manipulated using the manipulator 16 through the objective lens 20 (S54).
  • the control device 30 generates force information indicating the magnitude of the force sense to be presented to the user based on the image captured by the imaging device 24 (S56).
  • the control device 30 controls the operation of at least one of the first force sense presentation device 36 and the second force sense presentation device 80 so as to present the user with a sense of force according to the generated force information (S58).
  • FIG. 17 is a flow chart showing an example of a manipulation method according to the present disclosure.
  • the control device 30 acquires position information indicating the position of the second holding member 84 from the second force sense presentation device 80 (S70).
  • the control device 30 acquires angle information indicating the rotation angle of the rotation member 86 from the second force sense presentation device 80 (S72).
  • the control device 30 controls the operation of the manipulator drive mechanism 26 based on the acquired position information to move the manipulator 16 (S74).
  • the control device 30 controls the operation of the pump 78 based on the acquired angle information, and controls the suction force and ejection force at the tip of the manipulator 16 (S76).
  • the imaging device 24 images the sample 40 manipulated using the manipulator 16 through the objective lens 20 (S78).
  • the control device 30 determines whether or not the sample 40 and the manipulator 16 are in contact based on the image captured by the imaging device 24 (S80).
  • the control device 30 If the sample 40 and the manipulator 16 are in contact with each other (Y in S82), the control device 30 outputs force information indicating the magnitude of the force obtained by multiplying the rotation angle ( ⁇ 0 ) by the second coefficient k 2 . is generated (S84). If the sample 40 and the manipulator 16 are not in contact with each other (N in S82), the controller 30 outputs force information indicating the magnitude of the force obtained by multiplying the rotation angle ( ⁇ 0 ) by the first coefficient k 1 . is generated (S86).
  • the second coefficient k2 may be greater than the first coefficient k1 (that is, k2>k1).
  • the control device 30 controls the operation of the second force sense presentation device 80 so that the force sense according to the generated force information is presented to the user through the rotating member 86 (S88).
  • FIG. 18 is a diagram schematically showing the configuration of a manipulation system 210 according to the third embodiment.
  • the third embodiment differs from the above-described second embodiment in that it further uses a third force sense presentation device 100 for presenting to the user the sense of force received by the sample 40 due to the operation of the manipulator 16. differ.
  • the third embodiment will be described with a focus on differences from the second embodiment, and descriptions of common points will be omitted as appropriate.
  • the manipulation system 210 includes a stage 12, an illumination device 14, a manipulator 16, a folding mirror 18, an objective lens 20, a variable focus lens 22, an imaging device 24, a manipulator drive mechanism 26, and a lens drive mechanism 28. , a control device 30 , a display device 32 , an input device 34 , a first haptic presentation device 36 , a pump 78 , a second haptic presentation device 80 , and a third haptic presentation device 100 .
  • the third force sense presentation device 100 is worn on the user's body and configured to present a planar force sense to the user's body surface.
  • the third force sense presentation device 100 is, for example, attached to the user's forearm 108 and is configured to apply a tightening force to the forearm.
  • the third force sense presentation device 100 presents a force sense according to deformation of the sample 40 manipulated by the manipulator 16 .
  • the third haptic presentation device 100 simulates to the user the haptic that the sample 40 would be experiencing by manipulating the manipulator 16 .
  • the user can experience the change in the shape of the sample 40, which is the object to be operated, as a force tactile sensation that tightens the user. This makes it easier to grasp the extensibility of the sample 40 .
  • the extensibility of the sample 40 means that the injection pipette 16b penetrates the transparent band 40b of the sample 40 to perforate in the drilling operation of the sample 40 shown in FIG. It refers to the degree of deformation of 40.
  • the case where the extensibility of the sample 40 is high corresponds to the case where the transparent zone 40b is relatively flexible and the sample 40 is deformed so as to be greatly dented before perforation occurs. Conversely, when the extensibility of the sample 40 is low, it corresponds to when perforation occurs without the sample 40 deforming so much.
  • the degree of deformation of the sample 40 is presented to the user as a force sensation, thereby assisting in grasping the extensibility of the sample 40 .
  • FIG. 19 is a diagram schematically showing the configuration of the third force sense presentation device 100.
  • the third force sense presentation device 100 includes a fixing device 102, a plurality of actuators 104a, 104b, 104c, 104d, 104e, 104f, and 104g, and a drive control device .
  • the fixing device 102 is a member for fixing the plurality of actuators 104a to 104g to the user's body surface.
  • a securing device 102 When worn on the user's forearm 108, one example of a securing device 102 is a cylindrical stretch fabric arm cover.
  • the fixing device 102 may have a band or the like for wrapping and fixing the fixing device 102 around the user's body surface.
  • the plurality of actuators 104a to 104g are, for example, McKibben type artificial muscles, and have elastic members such as rubber tubes driven by pneumatic pressure.
  • a plurality of actuators 104a to 104g are attached to the fixing device 102 while being wound around the outer periphery of the fixing device 102 having a cylindrical shape.
  • a plurality of actuators 104a-104g apply a force that clamps the brace 102 by increasing air pressure.
  • the plurality of actuators 104a to 104g are arranged at intervals in the axial direction L of the cylindrical fixing device 102, and apply force to a plurality of different points in the axial direction L. In the illustrated example, seven actuators 104a to 104g are arranged at intervals of 30 mm in the axial direction L, but the number and intervals of the actuators are not particularly limited.
  • the drive control device 106 drives a plurality of actuators 104a-104g.
  • the drive control device 106 includes, for example, a compressor for generating compressed air, a tank for accumulating the compressed air, and a plurality of electropneumatic regulators for controlling air pressure supplied to each of the plurality of actuators 104a-104g.
  • the drive control device 106 variably controls the magnitude of force applied by the plurality of actuators 104a to 104g to the plurality of locations on the user's body surface by changing the air pressure supplied to each of the plurality of actuators 104a to 104g.
  • the drive control device 106 independently variably controls the magnitude of force applied by each of the plurality of actuators 104a to 104g. Thereby, it is possible to present a haptic sensation in which the magnitude of force can be different at each of a plurality of locations on the user's body surface.
  • the control device 30 generates a plurality of pieces of force information for controlling the operation of the third haptic presentation device 100 .
  • Controller 30 generates multiple pieces of force information for driving multiple actuators 104a-104g.
  • the control device 30 identifies deformation amounts of the sample 40 at a plurality of positions based on the captured image of the sample 40 . Based on the deformation amounts of the sample 40 at the specified positions, the control device 30 generates a plurality of pieces of force information corresponding to the deformation amounts at each of the plurality of positions.
  • FIG. 20 is a diagram schematically showing deformation amounts Da to Dg at a plurality of positions of the sample 40 manipulated by the manipulator 16.
  • FIG. FIG. 20 shows the punching operation to the sample 40, and the outer shape of the sample 40 before starting the punching operation is indicated by a broken line.
  • the sample 40 is deformed so as to be crushed in the x direction by applying a force F due to the drilling operation in the x direction by the injection pipette 16b.
  • the control device 30 specifies the deformation amounts Da, Db, Dc, Dd, De, Df, and Dg of the sample 40 at a plurality of different positions in the y direction orthogonal to the drilling direction ( ⁇ x direction). .
  • the control device 30 generates a plurality of pieces of force information corresponding to the specified deformation amounts Da to Dg.
  • the control device 30 generates, for example, a plurality of pieces of force information having force magnitudes proportional to the specified deformation amounts Da to Dg. In this case, the greater the specified deformation amounts Da to Dg, the greater the magnitude of force information generated.
  • a planar force sense as if the user's forearm 108 is being crushed in the same shape as the sample 40 is presented.
  • the user wears the third force sense presentation device 100 on the left arm of the user and operates the injection pipette 16b with the right hand of the user, the user operates the sample 40 with the right hand while feeling the degree of deformation of the sample 40 with the left arm. can do.
  • FIG. 21 is a diagram schematically showing a method of calculating the amount of deformation of the sample 40 using optical flow.
  • the controller 30 can calculate the deformation amounts Da to Dg of the sample 40 in real time by calculating the optical flow of the captured image.
  • the control device 30 divides the range 112 including the sample 40 to be drilled in the captured image into a plurality of minute regions 114, and calculates the velocity vector 116 in each minute region 114 using a known optical flow method such as the Lucas-Kanade method. Calculated using the method.
  • Velocity vector 116 indicates the direction and magnitude of image change between the captured image at time t ⁇ 1 and the captured image at time t.
  • the number of divisions in the y direction of the plurality of minute regions 114 is set to 7 according to the number of the plurality of actuators 104a to 104g.
  • the division number n in the x direction of the plurality of minute regions 114 is not particularly limited, but can be set to the same division number as in the y direction.
  • Velocity vector 116 has an x-direction component and a y-direction component.
  • the control device 30 calculates the x-direction deformation amount ⁇ D of the sample 40 at a specific y-direction position by adding up the x-components of the velocity vector 116 over the range of x1 to xn for each y-direction position. For example, at the third y-direction position yc from the top in FIG. is calculated.
  • the amount of deformation ⁇ Di represents the minute amount of deformation of the sample 40 from time t ⁇ 1 to time t.
  • the control device 30 calculates the amount of deformation ⁇ Di in the x direction at a plurality of y positions yi each time a captured image is acquired.
  • the control device 30 generates a plurality of pieces of force information corresponding to the deformation amounts Di based on the x-direction deformation amounts Di of the sample 40 at the plurality of y-positions yi.
  • the control device 30 may make the air pressure Pi equal to a predetermined upper limit value P0 when the deformation amount Di exceeds a predetermined upper limit value.
  • FIGS. 22A to 22C are diagrams schematically showing the operation of drilling the sample 40 by the manipulator 16.
  • FIG. 22A to 22C are diagrams schematically showing the operation of drilling the sample 40 by the manipulator 16.
  • FIG. 22(a) shows a state in which the sample 40 is held by the holding pipette 16a, and shows a state before the injection pipette 16b starts drilling.
  • FIG. The control device 30 acquires a captured image as shown in FIG. 22( a ) and starts calculating the optical flow of the sample 40 .
  • FIG. 22(b) shows a state in which the injection pipette 16b is piercing the sample 40, showing the state immediately before the sample 40 is pierced.
  • the sample 40 is deformed so as to be crushed in the x direction by applying the force F in the x direction by the tip 38b of the injection pipette 16b.
  • the x-direction size of the sample 40 decreases from w3 before the drilling operation to w4 during the drilling operation.
  • the third force sense presentation device 100 drives the plurality of actuators 104a to 104g based on the air pressure Pi calculated by the control device 30, and presents a planar force sense to the user.
  • FIG. 22(c) shows the state immediately after the injection pipette 16b perforates the sample 40, and the tip 38b of the injection pipette 16b has reached the inside of the sample 40.
  • FIG. 22(c) shows the state immediately after the injection pipette 16b perforates the sample 40, and the tip 38b of the injection pipette 16b has reached the inside of the sample 40.
  • FIG. 22(b) shows the state immediately after the injection pipette 16b perforates the sample 40, and the tip 38b of the injection pipette 16b has reached the inside of the sample 40.
  • FIG. 22(c) shows the state immediately after the injection pipette 16b perforates the sample 40, and the tip 38b of the injection pipette 16b has reached the inside of the sample 40.
  • FIG. 22(c) shows the state immediately after the injection pipette 16b perforates the sample 40, and the tip 38b of the injection pipette 16b has reached the inside of the sample 40.
  • the third force sense presentation device 100 presents the user with a planar force sense in which the magnitude of the force is smaller than in the state immediately before the perforation in FIG. 22(b). The user can perceive through the sense of force that the sample 40 has been perforated by experiencing a reduction in tightening during the drilling operation of the sample 40 through the third force sense presentation device 100 .
  • the control device 30 may evaluate the extensibility of the sample 40 based on the captured image.
  • w3-w4 corresponds to the deformation amount D of the sample 40 from before the drilling operation to just before the drilling.
  • the extensibility CE can also be said to be a deformation ratio, which is a ratio of the deformation amount D based on the size of the sample 40 .
  • the control device 30 can specify the sizes w3 and w4 of the sample 40 using the captured image.
  • the x-direction size w4 of the sample 40 immediately before perforation may be calculated from the coordinates of the tip 38a of the holding pipette 16a and the tip 38b of the injection pipette 16b as the distance between these coordinates.
  • the control device 30 Based on the captured image, the control device 30 detects reversal of the deformation direction of the sample 40 in a state in which a force F is applied from the injection pipette 16b to the sample 40 in the drilling direction (-x direction). can be identified.
  • the control device 30 specifies, for example, the timing at which the deformation amount ⁇ Di of the sample 40 in the x direction calculated based on the optical flow changes from the ⁇ x direction to the +x direction as the drilling moment.
  • the control device 30 may determine whether or not the extensibility is good based on whether or not the calculated extensibility CE exceeds a predetermined threshold, and output the determination result.
  • a predetermined threshold value for example, 0.7
  • the control device 30 may output a determination result that the extensibility is good.
  • the control device 30 may output a determination result that the extensibility is poor.
  • the control device 30 may output the calculated numerical value of the extensibility CE. In this case, since the compliance CE of the sample 40 is automatically evaluated by the control device 30, even a beginner with little experience in cell manipulation can appropriately assess the compliance CE of the sample 40 during the perforation operation. can.

Landscapes

  • Engineering & Computer Science (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Chemical & Material Sciences (AREA)
  • Wood Science & Technology (AREA)
  • Organic Chemistry (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Zoology (AREA)
  • Genetics & Genomics (AREA)
  • Biotechnology (AREA)
  • General Health & Medical Sciences (AREA)
  • Microbiology (AREA)
  • Biomedical Technology (AREA)
  • Biochemistry (AREA)
  • General Engineering & Computer Science (AREA)
  • Sustainable Development (AREA)
  • Human Computer Interaction (AREA)
  • Analytical Chemistry (AREA)
  • Physics & Mathematics (AREA)
  • Molecular Biology (AREA)
  • Cell Biology (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Microscoopes, Condenser (AREA)
PCT/JP2022/031821 2021-08-27 2022-08-24 マニピュレーションシステムおよびマニピュレーション方法 Ceased WO2023027095A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
CN202280062736.4A CN117940822A (zh) 2021-08-27 2022-08-24 操纵系统和操纵方法
US18/687,203 US20240391111A1 (en) 2021-08-27 2022-08-24 Manipulation system and manipulation method
JP2023543946A JP7847383B2 (ja) 2021-08-27 2022-08-24 マニピュレーションシステムおよびマニピュレーション方法
EP22861383.2A EP4379446A4 (en) 2021-08-27 2022-08-24 MANIPULATION SYSTEM AND MANIPULATION METHOD

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
JP2021139395 2021-08-27
JP2021-139395 2021-08-27
JP2022-081752 2022-05-18
JP2022081752 2022-05-18

Publications (1)

Publication Number Publication Date
WO2023027095A1 true WO2023027095A1 (ja) 2023-03-02

Family

ID=85322803

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/031821 Ceased WO2023027095A1 (ja) 2021-08-27 2022-08-24 マニピュレーションシステムおよびマニピュレーション方法

Country Status (4)

Country Link
US (1) US20240391111A1 (https=)
EP (1) EP4379446A4 (https=)
JP (1) JP7847383B2 (https=)
WO (1) WO2023027095A1 (https=)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024262549A1 (ja) 2023-06-20 2024-12-26 国立大学法人東海国立大学機構 マニピュレーションシステムおよびマニピュレーション方法
WO2025191857A1 (ja) * 2024-03-15 2025-09-18 国立大学法人東海国立大学機構 マニピュレーションシステムおよびマニピュレーション方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004255493A (ja) * 2003-02-25 2004-09-16 Ibaraki Univ 微細操作用マイクロマニピュレーション装置
WO2008003416A1 (en) 2006-07-03 2008-01-10 Force Dimension S.A.R.L. Active gripper for haptic devices
JP2013050665A (ja) * 2011-08-31 2013-03-14 National Univ Corp Shizuoka Univ 微小付着物剥離システムおよび微小付着物剥離方法
JP2020122898A (ja) * 2019-01-31 2020-08-13 日本精工株式会社 マニピュレーションシステム及びマニピュレーションシステムの駆動方法

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100475098B1 (ko) * 2002-11-12 2005-03-11 한국과학기술연구원 단일 셀 조작을 위한 바이오 자동 조작 장치
KR100945881B1 (ko) * 2008-01-23 2010-03-05 한국과학기술원 세포 조작용 햅틱장치
WO2012018136A1 (ja) * 2010-08-06 2012-02-09 日本精工株式会社 マニピュレータシステム及び微小操作対象物の操作方法
KR102397670B1 (ko) * 2014-03-28 2022-05-16 인튜어티브 서지컬 오퍼레이션즈 인코포레이티드 정량적 3차원 영상화에 기초한 햅틱 피드백을 갖는 수술 시스템
US10369045B2 (en) * 2014-07-29 2019-08-06 The Johns Hopkins University Micromanipulation systems and methods
US10377047B2 (en) * 2016-12-23 2019-08-13 Benchmade Knife Co., Inc. Magnetically actuated latch
WO2019222211A1 (en) * 2018-05-15 2019-11-21 Intuitive Surgical Operations, Inc. Method and apparatus for manipulating tissue
US20210208170A1 (en) * 2018-05-25 2021-07-08 Spovum Technologies Private Limited [In/In] A method for regulated manipulation of a biological sample and a system thereof
US10369701B1 (en) * 2018-10-30 2019-08-06 Mujin, Inc. Automated package registration systems, devices, and methods

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004255493A (ja) * 2003-02-25 2004-09-16 Ibaraki Univ 微細操作用マイクロマニピュレーション装置
WO2008003416A1 (en) 2006-07-03 2008-01-10 Force Dimension S.A.R.L. Active gripper for haptic devices
JP2013050665A (ja) * 2011-08-31 2013-03-14 National Univ Corp Shizuoka Univ 微小付着物剥離システムおよび微小付着物剥離方法
JP2020122898A (ja) * 2019-01-31 2020-08-13 日本精工株式会社 マニピュレーションシステム及びマニピュレーションシステムの駆動方法

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
See also references of EP4379446A4
Y. SUN, K. T. WAN, K. P. ROBERTS, J. C. BISCHOF,B. J. NELSON: "Mechanical property characterization of mouse zona pellucida", IEEE TRANS NANOBIOSCIENCE, vol. 2, no. 4, 2003, pages 279 - 286, XP011104204, DOI: 10.1109/TNB.2003.820273

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024262549A1 (ja) 2023-06-20 2024-12-26 国立大学法人東海国立大学機構 マニピュレーションシステムおよびマニピュレーション方法
WO2025191857A1 (ja) * 2024-03-15 2025-09-18 国立大学法人東海国立大学機構 マニピュレーションシステムおよびマニピュレーション方法

Also Published As

Publication number Publication date
US20240391111A1 (en) 2024-11-28
JPWO2023027095A1 (https=) 2023-03-02
JP7847383B2 (ja) 2026-04-17
EP4379446A4 (en) 2025-03-26
EP4379446A1 (en) 2024-06-05

Similar Documents

Publication Publication Date Title
CN103717169B (zh) 医疗用机械手及其控制方法
WO2023027095A1 (ja) マニピュレーションシステムおよびマニピュレーション方法
US9314309B2 (en) Surgical robot and method of controlling the same
US20120053728A1 (en) Object-learning robot and method
KR101407986B1 (ko) 3차원 텔레스트레이션을 제공하는 의료용 로봇 시스템
KR100483790B1 (ko) 마이크로 조립용 다자유도 원격조종 로봇 시스템
CN110200702B (zh) 集成眼球凝视跟踪用于立体观看器的医疗装置、系统和方法
Pitakwatchara et al. Force feedback augmentation modes in the laparoscopic minimal invasive telesurgical system
CN108601670A (zh) 图像处理装置和方法、手术系统和手术构件
WO2018159336A1 (ja) 医療用支持アームシステムおよび制御装置
JP2012529971A (ja) 低侵襲手術のための仮想測定ツール
CN115397353B (zh) 用于外科机器人系统的移动虚拟现实系统
Noonan et al. Gaze contingent articulated robot control for robot assisted minimally invasive surgery
Kazi Operator performance in surgical telemanipulation
Staub et al. Human-computer interfaces for interaction with surgical tools in robotic surgery
JP2024508820A (ja) 外科用ロボットシステムのカメラ組立品のオートフォーカスのためのシステムおよび方法
CN110461271A (zh) 用户可安装零件安装检测技术
US20100069941A1 (en) Systems and Methods For Sensing Hand Motion By Measuring Remote Displacement
WO2018216501A1 (ja) 制御装置、制御方法、および手術システム
KR100505145B1 (ko) 마이크로로봇 그리퍼 장치
JP2000052289A (ja) 微小操作のためのセンシング方式と操作方式およびそ の装置
CN117940822A (zh) 操纵系统和操纵方法
Hill et al. Telepresence interface with applications to microsurgery and surgical simulation
Chinbe et al. Finger-attachment device for the feedback of gripping and pulling force in a manipulating system for brain tumor resection
JP7584123B2 (ja) マニピュレーションシステムおよび三次元位置提示方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22861383

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023543946

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 18687203

Country of ref document: US

Ref document number: 2022861383

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2022861383

Country of ref document: EP

Effective date: 20240227

WWE Wipo information: entry into national phase

Ref document number: 202280062736.4

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE